Ummm ...
Stability has to do with topology and how you do the wiring. There is not a direct relationship with how much gain you got. In fact, for some topology, G=X1 is perfectly stable solution. But as you already know some opamps are not stable at X1.
X1 gain means even if there is noise, you don't amplify it. This way, you contain the noise problem so it doesn't become problem. This is the easiest and most straightforward way to go at noise problem.
Of course, there are opamps that operate more stable at higher gain, but this does NOT mean the headphone amplifier should have that much gain. We have researched and discussed about multi-looping (to death) and it ought to work well. (Because opamps can operate at decently high gain, but overall gain can be more reasonable gain.)
Tomo
P.S. I understand the details regarding B22. I ought to try it out. But remember there are two types of pursuits of audio listening. Euphonics and Purity. Who knows you might love the "distortion" you get off of transformers and oil-caps (yumm).