i have a jackshaft on a scram I-moition 3 speed freewheel hub.I don't see why you can't use a coaster brake hub.You just won't be able to use that break cause your pedals won't pedal backwards.Hopefully, someone who knows will jump on here.
I'm using a Nuvinci with a SBP shifter kit, and I've read other posts from people here with a wide variety of other internal hubs and shifter kits. Doesn't seem like the presence of a coasterbrake in these hubs would change anything. But you would definitely lose your ability to actually brake using that coasterbrake - when you try to pedal backwards, your cranks will just freewheel, never engaging the brake.