There was a big realisation in the 80s Hi Fi consumer market in the UK that speakers weren't the most important thing. Speakers could not improve a poor signal. The computer phrase "garbage in garbage out" was used to emphasis this.
I think the same is very true of headphones. In fact, personally I think it is even more true. The headphones have tremendous advantageous over speakers for resolving information because they are not undermined by room acoustics and they do not split the signal across different speakers for different parts of the sound spectrum. So even average headphones often have tremendous resolving capabilities and this will be heard if they are fed with a high quality signal.
Personally I think that using a good source and amplifier with average headphones is likely to get better results than an average source and amplifier with good headphones.