Ok, it was pretty simple. Before you extrude, go to Polygons->Tool Options and select "Keep Faces Together"