No its a good start mate. I think by starting again is a waste of what is here. Edit the mesh by going around it and tidying it up. You can probably get rid of the geometry thats causing the bumps. Its a tricky object to do, you could have used booleans but you didnt and for me doing it this way is better. Booleans have their uses but not where Im from LOL.

Just keep tweaking it until its right, you may need to add geo or take some out depending on where you edit. You can always do the one side anyway and then dupe it over and fill the spaces manually to save doing both sides.

If you look at the wire frame image you can see where the 'dinks' are you just need to pull them about.

Jay