I think I know why this happens. Maya's renderer does not anti-alias the Z-Depth channel, which is used for the DOF blur. Maybe mental ray anti-aliases it and that's why you get better results. Anyway you could render your images without the DOF but saving the Z-Depth channel in the files. Then do the DOF in post but blur/anti-alias the z-depth a bit before. Here's what the z-depth looks like directly from maya's renderer.