Can you still open it in Maya on your 64 bit system?
optimize your scene if you still can open it.
Of how many polys does the scene consist?
What are your rendersettings.
Batch render your scene don't use the renderwindow it will never output higher quality images it stays in 8 bit.

If you want people to help you offer some info