I'm using 2012, I don't think my PC can handle Maya 2014.
As of Maya 2014, the mental ray dielectric material has been deprecated.
If you utilize a mia_material_x_passes material, it should work as it supports render layers. The glass presets should get you started.
I've tried compositing the specular's separate render (while I was using Blinn) but what I get is a black background with it, and if I overlay that on the hourglass that has transparent background and use the add operation, I'd still get a black background instead of a transparent one.
I just re-read my last post. It's supposed to be: "specular doesn't show in the pass's alpha, it's supposed to be layered on with an add mode in comp".
Having a 1.5 IoR on the hourglass in your scene would look strange because its a single sided surface, extrude it a bit for some thickness, especially if you want to get the proper look near the choke point where the glass is thicker.
I haven't used the mia_material_x_passes yet, I'll do that tomorrow along with what Gen suggested about extruding the glass.
Did you try what I mentioned previously? Apply the transparency of the diffuse to the specular layer, then add. That should retain the transparency.
It leaves me with this. The black still keeps holding on to the specular layer after using Add.
Yep just mask off the black background.
These are the raw renders for specular and diffuse with Blinn.
I believe you're applying the matte incorrectly - you shouldn't use a magic wand tool to select anything. What you need to do is mask out the background of the spec pass with the alpha channel provided in the diffuse pass. (The magic wand tool, generally, does not do this. It simply looks for similar colors) Unfortunately, I have no idea how to do this in Corel. In Photoshop, you can simply copy the alpha channel from one layer into another layer. If you can attach the raw images, I can give it a go.
If it's this, it was already maxed out. I don't know where else to look for increasing the ray limit.
It looks strange because your refraction ray limit is too low. Try 6 or higher.