I don't know what to tell you on that. I have never had problems selecting the layer check boxes. I am not aware of any bug in Maya for any version where that happens. I don't use macs so I cannot comment on whether it's an OS issue. I'd report it to Autodesk as a bug and see what they say.