Here is an idea, If you don't want to delete all the edges you can use them to your advantage. If you look at your castle reference images you will realize that most old castle walls are not perfectly straight.
Try moving some vertices in slightly to show a bit of irregularity in the building. It is not necessary, but it adds some believability.
Hope it helps.