I'm assuming you're using an area light with the light shape activated. The high samples need boosting. The trick is to balance that with the render settings samples. On a side note, it does look like the normals on the gems could be hardened for a more chiseled look and the refraction traces in the render settings could be increased.