On the DVD they say that the fire effect was achieved with particle sprites, using clips of real fire.
I think what makes things so complex is the fact that the fire only emerges from the folds in the Balrog's skin, which is a pretty dynamical thing. Great effect!
Subdivide and conquer!
Free your mind, and your ass will follow!