When rendering in Mental Ray, use Box filtering. I find it gives the best results.

The jagged edges are probably from your anti aliasing settings. Take a look into them, as well as your sampling settings as well.