While in vertex mode, simply pressing the delete key will only delete any wandering vertices that you may have selected. So the ones that are at edge intersections remain unchanged.
Edit Polygons > Delete Vertex will delete the vertices you've selected and any related edges. As far as i know this is the default setting, but hey, I'm new too.