So it has to be something with the codec or format of the RAW footage. It's MOV, but when I take the h.264 rendered clips I can scrub through the timeline and do real-time editing no problem. So it must be something to do with the raw footage causing problems. I will have to convert all footage to something more acceptable by Premiere in the future I guess.

Thanks again for helping me out. The CUDA thing worked great, I had to manually add it into the cuda file for Premiere every time I turned my computer off before, now that is permanently fixed via that link.