found the problem. The depth file has a black RGB channel and contains the depth information in an additional channel. To get it out in photoshop, install the IFF reader plugin (look on google) and open the channels box, on mine it was Apha 2 (I had an alpha channel as well)

Also for reference mental ray does not appear to produce a depth map at all, at least not one that gets saved anywhere.