are you trying to get the specularity to be irregular, or to follow say a grain in wood for example?
If you are trying to do that then yeah you want to apply your map to spec colour.

if you want to adjust to spec roll off then apply your ramp to the roll off.

If you aren't sure what you are looking to do, then experiment and compare results from applying it to all three values on separate occasions.