Here is what I mean by spliting up the geometry in a way that a hard surface object might actually be manufactured in panels. It solves the problem of carrying unwnated geometry through your entire model and creates realistic panel lines.

1/4