The default setting for Anti-alaising is set low. In the render globals set it higher. That should smooth your edges.