I understand your argument, but as far as amps, I don't think it's completely true. I definitely notice a pretty big difference in sound quality in my headphones with even a potable e11 amp, but it depends on the headphones too. I find that headphones with higher impedance benefit a lot more from an amp than those with a low impedance because the amp provides the power the headphones need to produce great sound.
The only portable amp I've tried was the E7 and I used it with DT 880 250ohm speakers. I connected it to my Iphone 3gs with the line out adapter and turned the Iphone all the way up, and the E7 to a level that was identical. I turned the bass EQ off and closed my eyes. I had my wife pause and swap the headphones from E7 to Iphone and then unpause, back and forth not telling me which and I could not tell a difference in terms of quality. There were other little clues that told me what I was listening to but it wasn't the audio quality or dynamics.
Now if I were able to take the Iphone apart and solder a connection to pick up the signal as it exited the Iphones DAC and before it got to the Op/Amp and then did this test, I'm confident even my wife would be able to tell the difference between the two. But even the line out connector still has the signal pass through the Iphones op/amp so the coloration is the same whether line out or headphone out.
Neither distorted appreciably.
I don't doubt that different amplifiers might have different 'profiles' where some might warm the sound, others might sound a little thinner. I do believe there is such a thing as a good pairing. IE a headphone that isn't very warm (like T1 Tesla) with a warm amplifier to offset this slight problem, like a tube amp. But those same differences can be reproduced on any amp using a little EQ.
It's not that I have a ham fisted ear canal either, I have been mixing and mastering music for over a decade and have a pretty analytical ear. For instance I can tell whether my DAC is plugged into a HUB or directly into a USB port. I can even tell if a short or long cable is used. But these things don't really change the sound signature, just other things like noise floor and how loud they can go without distortion etc.
The job of any amplifier is to have enough bandwidth that the entire input signal is output without being distorted, and enough discreteness in it's design not to introduce any noise or coloration of it's own. I think given those simple requirements, most headphone amps do the job regardless of price.
I laugh a little at how much audiophiles love tube amps because as a musician I look at the tube very differently. Musicians love tubes because when they are overdriven they do what is called 'soft clipping'. This is especially good for guitar and microphones. But for the output stage of an amplifier, it might give a little warmth but if you aren't over driving the output to distort will otherwise make no difference in the sound signature. This is assuming the impedances are matched correctly and if they aren't you have bigger problems.
If I might make another analogy....Before we had LCD's and Plasma, most televisions used CRT's. People would often get into arguments about which set had the most accurate picture, best color fidelity etc. All CRT's use phosphors and pigments to make an image and those phosphor formulations are nearly identical from set to set. The differences people were seeing were in signal manipulation done inside the set. Provided you could get into the service menu's you could make about any TV 90% of the way to a studio monitor at 1/10th the cost.
Now that isn't to say that a Walmart RCA tv can do the same things that a Sony Wega could. The Wega's were awsome because they didn't restrict the video bandwidth as much, which let more detail through without artificial sharpening, and because they physically constructed the shadow mask differently (trinitron) they looked different than non trinitrons, complete with the little shadows. And audio amplifiers differ like that too, mostly in class, D, A, B, A/B, etc.
But unlike video equipment back in the day, audio equipment needs much less bandwidth, 20-20,000hz to fool our ear so that even inexpensive components and super thin cables can handle that. Contrast that with the hundreds of megahertz of bandwidth that a video signal needed and you begin to see how good we have it.