On the DVD they say that the fire effect was achieved with particle sprites, using clips of real fire.

I think what makes things so complex is the fact that the fire only emerges from the folds in the Balrog's skin, which is a pretty dynamical thing. Great effect!