Hey man, I wouldn't pull the plug on this yourself. If you do the best you can, and they feel you're not quite suited for the work they're doing, well that's all you can do. However, if they do offer you a job, it means they think you have what it takes based on what they see.
The way I see it (and I might be missing something) you're not losing anything by going for this, besides maybe a bit of "wasted" effort and some pride, however if you do back out, who knows, you might have been doing better than you figured.

At the end of the day it's their call whether you're "in over your depth" and even if you are, they're not going to phone everyone else and say "don't hire this guy". They obviously have liked the work you've done. Does that make any sense?

Of course, on the other hand, if you don't want to do the kind of work they do, give it a try (if offered the chance) and make sure you don't like it, then you can always try for something else.