You probably have to lower the resolution, i've got an ati radeon x1950xtx 512 mb card and it only renders hardware to 512*400 (or something like this), if you can wait until maya 2008 comes out because in this version mental ray can render them