The chainstay is the bar the tensioner is mounted to. Basically drill a hole through the bracket and into that bar and put a screw in there. That should keep it from rotating on the bar.
You could also try something like this:
Or if you've got the money, go with Pablo's kit and you'll get rid of the tensioner, the extra sprocket, and the stretchy chain.