these are my settings, if these dont work for you try moving your user preferences out of the way (back them up) and let maya generate some new ones, then give it a go.

Normally settings are in...

This is due to the fact that maya's default settings for Mental Ray are 16bit which is not supported by the IFF file formate. Just change the file formate to an (EXR,TIF,or any format that supports...