im not sure about vector render settings but if you are using Maya 5 u can achieve the cartoon look by using the new RAMP shader and using the Maya software renderer.
Also here is a nice tutorial on creating a cartoon look using a surface shader.

https://www.vrvis.at/br1/cgr4/cartoonshader/