Every time I make steel stuff on my mill for a while and then need to make an aluminum part, I am always amazed at how much easier and nicer the aluminum cuts. With a sharp tool bit you can easily get a near-mirror-finish on a piece of aluminum in one pass and cutting with no lube; most steels won't cut nearly as nicely even with lube. They come out smooth, but don't look nearly the same.
...Of course sometimes you aren't free to choose, and other factors can basically require steel--but if you can use aluminum it is worth considering it for that reason alone.
With the lathe, it doesn't seem to make much difference if I'm cutting aluminum or steel for most kinds of operations.