Hi dave. Thankyou. I have 'Z-Brush Character Creation, Advanced Digital Sculpting' book, by Scott Spencer. In the displacement chapter (chapter 9) he talks about the different types of maps, 16 bit, 32 bit floating point, multiple 32bit maps e.t.c. He reommends 32 bit floating point which I created and exported via the multi displacement 3 plugin for Z-Brush. Then, on the DVD included with the book there is a batch file which you can drag and drop your files onto to convert them. The displacement file was coming out of Z-Brush in TIFF format. What this file does is convert them to .map format via the command line, an automated process and made kind of sense to me as I have heard about mental ray's preference for this file format before...
So then in maya I set up the approximation editor and the shader. Now on the DVD with the book I mention there is a short tutorial on there covering this process, again by Scott Spencer. However it isnt clear what file format he exports out of Z-Brush but he doesnt convert it to a .map, its still a TIFF, not sure 16 or 32 bit. He also sets up his approximation editor a little differently. He creates one for displacement and one for subdivision. In the book he just uses Subdivision. Plus he uses slightly different settings. On the video he goes for length/distance/angle, in the book he goes with parametric. So there are all these factors which could be at play. Thats all relying on the fact that I created a successful displacement map in the first place. Again, I did this by following the books instruction so Im assuming I have. I have tried several variations of the above settings, different approximation settings and values.
This is the picture of the ztool I generated the map from. This is at level 6 or 5. I generated the map off of level 1. And it is the exported level 1 obj I am trying to put the exported map back on in maya.