Got it partially working, with white as the transparent background in the alpha (which surprised me, I thought it was the other way about?).
Still got a problem; though it is transparent, when I layer them, I can still see the background, but I can't see the layer under it (was going to have layers of hair for the brows and the head hair, this is just a very rough test of the technique).
I've tried setting mental ray's trace depth higher, but still I can't see the underneath layer's colour.
again, any help would be great.