What should I do? loosen the nut? I have it set down to the lowest height anyways....
You have gotten good advice.
Also; I don't remember if I've mentioned to you that you can most likely remove the chain tensioner completely. It is usually not needed.
If your drive chain doesn't rub on your chainstay, you should consider getting rid of it.
Lot's of folks use them, but I see them as a potential problem that I can do away with. Think of that thing rolling over into your spokes while you're at speed- pretty bad news.
Sizing your chain appropriately will allow for this.
If you must have one, the best type bridges the chainstay and seatstay.
Good luck!