If you end up with a lot of "loose" vertices, ie ones without an edge connected, you can delete them all by going into vertex mode, selecting everything and hitting the "delete" key on your keyboard (note that this is NOT the same as the delete vertex function).

This method will usually remove any loose vertices. However sometimes it will screw up, so make sure to check your model after using it, and be ready to undo. Also, it shouldn´t really be necessary very often, since you should make a habit of deleting the vertices after deleting an edge..