You are correct in assuming you need to go to face mode and just start deleting and then cleaning up the geo. Sort of treat it like you would if you were actually building a house - not everything...