Final Gather Question
 

I posted this before in Maya Basics but no answers, can anyone help?

I'm very new to final gather, this is my first try at HDRI, and I'm having some artifacts, mainly blotchiness, mostly noticeable in the shadows. At first I thought it was a problem with global illumination, because the blotchiness looks like it could be derived from photons, but it still happens with just final gather (in fact in this image, I'm not sure that global illumination has any effect, only having final gather on seems to produce the same result).

http://spanorak.com/350zFinalFG500wGI.jpg
500 Final Gather Rays
http://spanorak.com/350zFinalFG2000wGI.jpg
2000 Final Gather Rays
http://spanorak.com/350zFinalFG4000wGI.jpg
4000 Final Gather Rays

What is causing this? It does seem to get better as I up the number of rays, but 4000 is pretty high already and the render times are going up a LOT. Enough that I didn't want to go up to 8000 or more... Is there a setting I'm missing? Or is something else causing this? Anyone?