Well for starters, headphones are very sensitive, it doesn't take much energy to move that driver, unlike a speaker, so they introduce things like resistance to essentially give load to the amplifier so it doesn't wildly throw a driver around that is sensitive in the 100db/mw range type thing, where speakers in the 86~90db/mw range are "efficient" relatively, at low resistance, while in headphones that sensitivity rating qualifies a headphone as inefficient to the point of needing stupid amounts of power (HE-6 for example). A headphone amp is designed to essentially work without a fan usually, be as absolutely quiet as possible operation wise (such as the power supply) so that it doesn't translate into the super sensitive headphone driver; and it's designed to output low power into various loads.
Plug a 200 watt amp into a 32ohm headphone that has a rating of 114db/mw sensitivity and see what happens for example.
I would however be curious how some of these amplifiers would work with higher impedance headphones.
I know back in the day people used "dummy loads" basically to keep a headphone from being destroyed by a speaker amplifier and just jacked headphones into normal speaker amps with them, since they didn't really have a lot of "headphone amps" at the time like today. Market has allowed for this to evolve essentially.