I think I ran in to this often when dealing with paint effects.

If you're using Toon Outline from: (Rendering Menu)>Toon>Assign Outline>Add new Toon Outline, I think it means it's a Paint Effects outline.
If you want to tweak PFX anti-aliasing you should try Oversample. You should go to Render Settings>Maya Software>Paint Effects Rendering Options> ... and check Oversample checkbox. I'm not sure, but I think tweaking "MultiPixel Filtering" under Anti "Aliasing Quality" also affects the Toon Outline quality.

Hope that helps.

Bye

Mário.
www.mario-vieira.blogspot.com