...
Speakers - especially multi-way speakers with their crossovers etc- have pretty complicated impedance vs. frequency curves and this causes certain amplifiers to interact in various ways. If the curves really dip low, you're going to need something with a lot of current. And if the amp employs much negative feedback you may see some unexpected transient interaction with the speaker; and tube amps, with their output transformers, often show rather large differences of power delivery over the range of audio frequencies. Sometimes these interactions serendipitously produce wonderful sound. Speaker-amp matching can be quite an art.
I think we bring the expectations of system-matching that we developed in the speaker world to the headphone world; I think there's an expectation that with just the right amp, a given headphone will show truly superior performance.
However, headphones don't have the complicated impedance curves of speakers; in general, there's no crossover and no bass reflex or transmission line diaphragm loading to cause hard-to-drive phase angles, etc. Some headphones - the planar types- look like plain resistors to the driving amp, they have almost no reactance at all at audio frequencies. So really headphone amps aren't faced with loads that will bring out the kind of differences in them that speakers can bring out from speaker amps. ...