HDR images are used to ight the scene, I.e they give off light that then interacts with your objectsgiving it the look of it being in the environment.

What I think your not doing is swithcing on final gather