Looks like to me the problem smoothing your model was that there most likely were NOT any edges connecting the in circle of the d to the outer edge.

I quickly did a model just by using your template and using the create poly tool to outline the outer part of the d. then again on the inner circle of the d. I extruded both. Then I made an edge connecting the top rounded edge of the d to the bottom rounded edge of the d. Then I used the boolean difference tool to subtrace the extruded circle from the d. Now my d has a hole in it and the hole is connected to the outer edge. I cleaned up the verts and did a bevel. Then I added a few anchor edgeloops and smoothed. It only took me about 5 minutes to do this. I'm pretty fast at modeling but it's a pretty easy way to do it I think.

Edit: I didn't spend time making it perfect but for your model, you should make it all quads before smoothing so you have a nice clean mesh.