I would not approach this project by doing multiple extrudes. This can cause its own issues.
I would do it like:
- Create a poly cube to match the bottom most corner (bottom right) and get it to the size you want
- Use multiple extrudes along the base where the window uprights are and continue along until the end
- Create the uprights with an extrude until you get the desired height
- Delete those faces at the top of the uprights, they are not needed
- Duplicate the geometry to create the 1st floor and again for the 2nd and so on...
- On the last duplicate delete the upright geometry and use the append polygon tool to fill the gaps left
- Combine the bottom and first level geometry and merge the interconnecting verts, do this for each floor
- Once done you should have a complete front face, add any required details and texture
Now do as Steve (Gster) suggested and use a lattice to defrom the building.