The tube is just a spacer. Although essential, it is not critical in nature.
You should be able to shift it to one side or the other by prying it away against the inner race of the bearing and use the end that you expose to drift one bearing out a little to the point where the spacer becomes loose inside. You can then punch/drift the bearings out from opposing side as the spacer should then just flop about in there.
I assume you are replacing the bearings and damaging the ones in place makes no difference?
If you do harm the space, which I doubt, they are readily available for around $16.