haven't used xp64bit (heard it doesn't play well with maya) but is there a folder program files and program files (x86) that you can install to?
if there is make sure you install to program files

i think you can change the limit of memory you assign to a render in the render options
render > render current frame > []
and
render > batch render > []
in the box titled Memory

is that what you're looking for?
also, when i render i notice the ram doesn't shoot up but the processor speed does, perhaps your renders aren't memory hungry as you expect? *shrug*