a few recommendations...

first: increase your virtual memory. when rendering larger files (in succession) i usually set my page file to about 1.5 times the amount of ram i have.

second: make sure you've got 'use all available processors' checked. not sure if there's a different method for enabling that for mental ray, i've never bothered cause i don't have a multiprocessor computer anymore.

third: maya usually renders faster when you don't have maya open. render from the command line, using the Render command (i do believe the capital R is necessary), but you have to make sure the file you're rendering is all one word - no spaces. set up all your render options, save your file, and close maya. besides, it'll free up some ram.