i don't really know what you mean by refraction pass tbh, but you could switch the alpha channel with which ever one you want. just do a quick ID pass.

what are you using to composite?