I understand what you are saying. But.... Why couldn't you make a mask in Aftereffects (or shake, Make it have a large Feather, and then add it to the frames (rgb) that you export from Maya, So, that should mask everything, including the reflections/refractions? Not sure, sorry I am a little confused...