To drill it out, I would use drill bit in a chuck mounted on the tailstock, with the shaft in a 4 jaw chuck, dialed in for greatest accuracy. 3 jaw will do but will prolly be a few thousands off center.
When you spin the shaft against a stationary drill, the drill automaticly follows dead center.
I would drill it undersize and use a boring tool sneak up on the desired diameter.
I would make a sleeve that is a slip fit, then cut a slot for a key, and use a wide keystock to traverse all the keyways.
You could prolly make one from bushing stock tubing, from Aircraft Spruce.
I also think that I have seen such bushings for sale somewhere, like Graingers or McMasters