You could use the multiply node found in utilities and multiply the bump with the spec map, that would leave zero bump where the spec is black and one where the spec is white.