As far as I am aware, you can't render Maya Paint Effects in Mental Ray. I had the same issue with my Wii WIP on the glow.
To get round it I converted the paint Effects to Ploygons, tweaked the shader slightly and rendered that way.

From what I know of Paint Effects... "... Paint Effects strokes are a post-render effect in the Maya software renderer, so they have to be recreated as geometry to be rendered with Mental Ray, ..."

They only other option I would suggest is compositing in Post-Production...