You can't keyframe a pivot point. However, you could play with a variety of constraint-based options (I'm thinking a parent constraint would work well here).

Or, just keyframe the animation. It's much faster that way.