As for the first problem, I believe this actually has to do with the physical limitation of the vertex processor on your GPU....don't qoute me.

"Vertex shaders up to 1024 instructions with flow control"

straight from ati's website.

As for program performance.....check what's running during your Maya adventures....and maybe tweak some of the program options....you definitely have a decent rig.....so I would look there first