I did that too. If you round off the corners and paint it, it's not so bad.
I have made 1x1 tensioners out of old rear derailers before. You can leave them intact, and run the chain normally, but short as possible, or cut the pulley cage down and use one pulley with a long bolt and locknut, best with some kind of big washer to keep the chain from ever possibly skipping off there. You can cut a spoke or piece of wire hanger to substitute for the cable to keep it in position, or just replace the limit screws with long allen bolts and a drop of loctite on them. It looks nice, and I never dropped a chain even on a mtn bike.
There are commercial products now like the one shown above, but you can make mine for $0-5 easily.
Originally Posted by SlowBalt
I just use the kit tenshioner on the pedal side.