What resolution are the frames at? PFTrack will bring in the full resolution plates on import, so what you should do is scale down the frames afterwards, and tell Maya to use those. Also, if you play through your entire scene a few times, so that every frame has been displayed, it will cache those images to RAM, but again, it all depends on how big these images are. If they're 1920x1080, usually 960x540 is acceptable.
Hey thanks for the quick response.