You may already know this but you can turn off specularity for individual lights if that's what you're having problems with. Just uncheck the specularity and leave the diffuse checked for whichever lights you only want diffuse light for.