i also had a similiar problem in both version 8 and 8.5. i looked for any unknown changes in the settings but couldn't find any. they were all fine. getting nearly frustrated, i finally resorted to loading the default settings for everything (that too for more than 4/5 times) including render globals, preferences and every thing i could. then only could Maya render proper .avi files. i hope this helps you. it did to me and has not bothered me again.