I didn't think about tinkering with the relationship editor to try and fix it.

I have two sets of UV map1, and a lightmap set I made that is identical to map1. When I changed the relationship to have the textures associated with the lightmap it actually worked, well it looked like it should.

How will this effect my exporting into fbx for unity... and any idea why that would have happened in the first place?