I added a brace to tie both sides together better, I will add another one on the other side of the roller later. The problem I am having with working with aluminum is flex. If I had used steel I would not have to have braces everywhere, but with the tools at my disposal that would have made things quite a bit more difficult. Things are finally pretty rigid now, and the bike feels great.
I think that I will end up replacing a few bits with steel, though, like the U bracket that everything pivots on.