Okay I got it figured out.

(1) for single frame renders

in maya 2009 under the render tab open render current frame options

uncheck auto memory limit
click on calculate

This selects the soft memory limit for a single process based on the memory in your system.

(2) for batch renders

In my case I left everything default (ie auto memory limit checked) and batch render works great. In fact it renders significantly faster then a current frame render.