I believe that the ideal ratio of amplifier output impedance to headphone impedance should be 1/8 of the impedance of the headphone you are using. For example, if the output impedance of your headphone amplifier is 20 ohms, then the ideal match would be a headphone with a 160 ohm impedance (or 8x that impedance).
It's amazing how an idea from a single person, maybe as much as a decade ago, can become distorted and turn into meaning things that they didn't in the first place.
Tyll Hertzens, who was very famous in the hobby, as he made the first commercial portable amplifier (under the HeadRoom brand -- now long defunct), then went on to measure headphones and write for Innerfidelity, which Stereophile shut down (as after Tyll retired, the person they selected to take over was worse than useless), suggested that a headphone amplifier, ideally, should have an output impedance
at least 1/8th or lower than the impedance of the headphones it was driving. This is connected with
damping factor.
Long story short, as a speaker or headphone driver is extended by voltage, it will return to its original position, in turn generating voltage that it will send back to the amplifier. To lessen distortion from this, the driver more or less "driving" the headphone amp, the amplifier should have an impedance that is considerably lower. Tyll suggested 1/8th or lower.
The downside to this is that people became overly obsessed with low output impedance after Astell&Kern's original AK100 player had an output impedance that was 30 Ohms, to the point that people would lose their minds if it were not near zero Ohms. However, if it's
too low (<1 Ohm for headphone amps) sensitive IEMs especially can easily pick up hiss from the electronics. If it's too high (<5 Ohms maybe?) then some multiple driver IEMs with a non-flat impedance curve will often end up with frequency response changes as a result. My experience suggests 2 Ohms for headphone amps is about ideal, but there are quite a few other factors in this to be considered.
I remember reading a post by Marv over at SBAF in regard to the ifi Zen Dac Signature (original version), which he describes in some detail. The bottom line is that he liked it (and he's very discriminating) and felt that it was a good value at the price. I think it was $249.99US at the time. What surprised me was that Marv also thought it sounded better than the Schiit Modi Multibit (this was pre MM2), which has been very well received since its inception. I know that Marv has access to some very nice gear, and from what I have read, he's not easily impressed. Which makes his comments regarding the Zen Dac Signature V1 even more impressive.
I rather like the iFi BB/TI-based DACs and how they sound, so I concur with this idea, if they've fixed the issue with their PSUs constantly failing. However, Marv also helped tune the HD8XX, and we all know how that turned out. I don't recommend putting people on pedestals in this, or any hobby, including me.