im not sure about vector render settings but if you are using Maya 5 u can achieve the cartoon look by using the new RAMP shader and using the Maya software renderer.
Also here is a nice tutorial on creating a cartoon look using a surface shader.
SimplyMaya specialises in Maya tutorials. We offer over 1,000 individual Maya training videos, ranging from basic Maya tutorials through to intermediate Maya tutorials. Our tutorials are created by instructors with industry experience and are designed to get you up and running in Maya quickly without making it seem like hard work.