no it is not a crazy workaround. it would be a hell of a mess if everytime you change the position of pivot the xyz coordinates would change. you can enable the handle, and snap the pivot to the handle w/o snapping the pivot to a vertex. point snapping works also for handles - but only if they are visible.