I think the problem is that there is displacement on the corners (sorry I wasn't clear enough) I think I will just blacken the edges on the displacement map. Fortunately I am doing all this in photoshop with lots of layers (one for each brick) so I can adjust things as I like there.