In your LAST image abouve, there is a tab named: "Maya Hardware", make sure you have your settings correct in that part before you begin the following steps:

I'm going to answer YOUR question, as well as reveal a fix/workaround for those of us that still have not upgraded our graphics cards yet, but STILL NEED to work with hardware objects and paricles.

Ok, since you said that when you render one frame using Hardware, it renders fine, that solves my first question to you:
1. Do you have a qualified Graphics card with Hardware Overlay capability?
Maya 5's new "render using Hardware" feature REQUIRES a qualified card now.

HOWEVER, I've found a workaround for this, AND found an answer for you in the process of experimentation:

I am an old fart, and as such, my video card doesn't qualify to take advantage of the hardware batch feature.

When I try, I get the following error message:
// Error: Graphics card capabilities are insufficient for rendering. Batch Render aborted. //

THIS DOES NOT mean that I/YOU CANNOT render hardware objects or particles!

YOU CAN! whoohoo!

If your card is old like mine, you will get this error.

Here is how to fix the issue, AND/OR here is probably why you are only getting a 10 frame render using the new Batch hardware render function:

Feel free to setup your batch hardware settings the way that you want.

THEN:

Instead of using that new feature, treat your situation JUST LIKE MAYA 4.5 and older!

1. Go up to WINDOW > RENDERING EDITORS > HARDWARE RENDER BUFFER.

2. Go to the RENDER > ATTRIBUTES menu, and REset all of your hardware settings like, frame start and end times, frame size and aspect ratio, shading, lighting etc.

3. Then, simply preform your hardware render from the Hardware Render Buffer like we always have! Easy.


Good luck!

PS, about the avi setting, I'm not sure about that one, I'm still looking