Amusingly I can't find anything detailed (maybe I'm not looking hard enough), but it seems that Maxwell uses a raytracing algorithm called "Metropolis light transport" which basically does some basic raytracing and then determines which bits are visible before doing a more detailed raytracing of the visible bits.

I couldn't find much on mental ray as well, although it is a raytracer like Maxwell, but uses a scanline method to determine visible bits (which is better I have yet to find out) but it seems that mental ray is targeted towards massive multiprocessor systems, so your dual core box might not fall into that category, which could explain the performance hit.