Set it up as a soft body with a goal and allow particle collisions in the menu then select the particles and allow them to collide with the object (can't remember the menu its under, think it may be he particle menu)

You might have to set up springs on the softbody so the buliding returns to the orig shape