It does sound like your machines running low on RAM.

What I would be tempted to do is render in layers using opacity masks so its easier to comp in together that way your rendering less geo each time.

Also render from the command line to free up system resources (or create a sceen for each layer with said masks then render using a .bat file) and see if that works.

Also since your rendering layers render in passes too that way you can fiddle to your hearts content in post to get it looking how you want.

Also to remember to render the alpha out (look in custom entities in the mental ray tab even thought it might be switched on in the render globals)