Can't tell, what you are doing wrong. My maya did this a couple of times to me. I seems to be moody. But since I lost a whole night of rendering. I do not render to movie files anymore. You could just render a png, jpg or wahtever you want sequence, and use quick time or something similar to compile it to a movie.