That, and the ear pads. This is purely an anecdote.
The pads of my old headphone is so worn that it fell off. The effect is immediate. You notice the sound being thin and not musical and all that.
New pads. Voila. Good as new.
I thought I could identify the difference between odac and fiio e10 . These two dacs measures well ( odac is better numerically of course, but e10 fits my applications as well) and I am quite confidently say if you do a proper double blind test on me, I would not be able to differentiate the both of them consistently with my current headphone.
The pads on the other hand though, you have to be deaf to not be able to tell if it is working or not.
I agree most of the difference that can be improved or become worse is in the headphones. There is only so much improvement that can be done on amp/source. A crappy headphone remains crappy regardless of source. Although a very good sounding headphone can sound bad if not sufficiently amped. I think this can be easily rectified without costing much.