Instead of Set to Component, you can also just try Object for the manipulator orientation. Frequently switching between world and object is pretty common.