20mW as an average power is already very loud, but music is not like continuous test signals (which are used to measure output power, impedance, and so on).
Attacks, for instance, create a surge current draw that (relatively) weak amplifiers cannot provide (relative to the average power of the music). As a result, the attack is not sharp/tight and the overall impact is tamed. As a side effect, the imaging (instrument separation / overall definition) is decreased due to the lack of transient in higher frequencies (which the planar headphones are loved for) and the bass becomes bloomy.
The more you crank up the volume, the less headroom you get and the sound becomes blurry/bloomy and all the side effect that comes with it (harshness, ...).
As a matter of fact, powerful amplifiers have more headroom than weaker ones and thus, do a better job overall at reproducing transients (treble) and slam (bass) and have the headroom to retain these qualities at higher volumes.
The HE-500 is rated at 89dB, which, for headphones, is still really low.
The DT-880, for instance, is rated at 96dB. It's not the most efficient phone either but it already requires 5x less power than the HE-500.
Planars have an advantage however: their impedance is (almost) purely resistive and quite low which makes for an easy load to drive (in the amp's "point of view").
