did you have dmap shadows on? It sounds like a problem I had with the shadown map not being high enough res. And that would explain why it went away with Mental Ray aswell.

Post a pic of it.

Alan