From your description it appears that you intend to rely on contact fricton between your shaft ahd the bearing inner race to transfer the rotational loading between these two items. If that is true, it is going to be an very unreliable method of torque transfer. It also will have a very short life due surface wearing or galling and, will potentially damage the bearing due the substantial longitudinal loads during engagement and disengagement.