Part of the problem may be antialiasing . . . for a definition of that, go here - https://en.wikipedia.org/wiki/Anti-aliasing
I am quite sure Mental Ray has settings you can change to alleviate the jagged edges, both thru the antialiasing settings and the quality of render, but I am not that familiar with Mental Ray yet.
"The Sage as an Astronomer: If you still see the stars as something above you, you lack the eye of knowledge." Friedrich Nietzsche