Sure, speakers have much more troubles than headphones when it comes to fidelity. I think I also said it before.
But I stand by what I said about headphones vs DACs. I'm not telling anybody to get a 30$ DAC and a 9k$ headphone. You do what you want.
I am saying that transducers are the weak link in the chain for most audio variables. Something that, again, you can measure and show as factual. And improving the worst element(usually the headphone) will give the biggest improvement. It's not an opinion. That how things are.
I take my extremely powerful calculator and real numbers picked at random online on real gear specs at 1kHz. Consider the headphone to have THD at -66dB(0.05%) while the DAC has THD at -86dB(0,005%) not a great DAC but a big improvement doing nothing will better illustrate my point. What is the total THD? -65.957dB. Predictably, nearly the same as the worst element.
I improve the DAC part with another that measures -122dB(0.0000794%) now we're talking. What is the total improvement out of the headphone? Sticking to 3 decimals, the total THD is -66dB(0.05%), the same as the headphone alone. Improving the DAC's THD by tens of dB, we went from a total of -65.957dB of THD to -60dB. \o/ it's something...
Now instead I improve the headphone part with another that measures -70dB(0.0316%) instead of the initial -66dB (so just an improvement of 4dB on the headphone). That gives us a total between the original DAC's THD at -86dB and our new headphone at -70dB, of -69.892dB. Instead of the starting combo with a total of -65.957dB. So a total THD improvement of 3.935dB. Nearly all the 4dB improvement at the headphone goes into improving the total signal.
QED like they say in movies with vampires.
To be clear, it has nothing to do with being upstream or downstream, and everything to do with improving the weakest link to make the biggest improvement. If the DAC was the weakest link for a given variable, then improving it would be the best solution for that variable.
It just so happens that headphones are much worse than DACs in most audio variables. So providing even a slightly better headphone is usually the easy, obvious way to significantly improve the signal at our ears. It's an obvious consensus for speakers, but it remains true for most variables with headphones. Sadly, because transducers are very hard to improve objectively, the best ones tend to be fairly expensive(but some very expensive ones can be crap, money is not a sure measure of fidelity, or taste for that matter).
There is no mystery of science here. It's about measurements of audio variables, and how dBs work. And I guess I'll leave it at that, as I can't think of more to do to convince you of clear measurements saying clear stuff.
Then you go change your DAC and you feel some other change for whatever reason affecting whatever variable. Ok, maybe it's true, maybe you shouldn't listen with your eyes, maybe you should check for volume level differences(not all DACs output exactly 2V RMS fullscale), in the end it doesn't change anything about what I said. For a given measured variable, the weakest link is what should be improved, and as there is probably no headphone as good as a good DAC, improving headphones will always remain a valid choice. And that, regardless of what you will decide to do about the DAC.
Wow, that was a fun read, right?

Hopefully the values are correct, and I didn't input butter fingers magic into the calculator. Most likely nobody will check anyway, so it's fine. ^_^