All your references would be loaded at render time anyway . How much memory do you have. You could allways break down the render into passes of certain objects.

E.g. Render half of the stuff on one render layer and the rest on another then merge them for the final comp.