On the layered shader, change the composite flag(or something like that) to layer texture instead of layered shader. Otherwise I think you'll run into some energy issues. Also you shouldn't have to connect anything to the surface shader transparency, just plug the mask texture into the layered shader transparency for that layer and let that do the work.