I generally render iff or openEXR out of Maya depending on if I want 8-bit or 32-bit. I render out of AE with H.264 selected as the format. When you say choppy or stuttering, do you mean that it looks like your media player is having a hard time keeping up? Or that it is consistently juttery looking. What are your file size outputs?