It has nothing to do with power and everything to do with implementation. You can have oodles of power and a top tier chip but if the circuit design is crud the headphone will sound like crud. Something else to note, you can have a top tier pricey amp and one headphone can sound terrible while another will sound fantastic. It's called synergy, don't ask me why it happens it just does. If you're looking for a good mobile sized amp Leckerton and ALO have nice offerings. I own both of Leckertons amps as well as the ALO RX MK II and I'm quite happy with them. JDSLabs is another dealer you can look into. The O2 amp gets lots of praise on here.
It happens because the amps are s***. Seriously though, just like a good DAC can be ruined by implementation, a good OpAmp stage can also be ruined by design. That doesn't really say much, however. With low impedance phones you want to look for amps with output impedance less than 2-3 Ohms, and even that is pushing it. The rule of 8 is important because if you combine high output impedance with low input impedance you can screw up the dampening factor, and get that "sloppy" or anemic bass. On that same amp,a headphone working @ 150 Ohms may sound fine - even great!
The same goes for ruthlessly efficient IEM - which not only can get by with practically no power, but then can dampen outside noise by as much as 20 dB. The unusual isolation will make static, pops, and hisses a lot more apparent. Also, the low-impedance drive might reveal problems with channel balance or even distortion present with less volume on the dial (or more likely, distortion with lots of volume on the dial!).
In my opinion, as a consumer of higher-end head audio, the whole industry is frustrating and weird. In car audio, generally speaking, all speakers and Amps are designed to work between 2-4 Ohms. Only a few manufacturers are designing 3 Ohm speakers which actually risk damage if you have the wrong amp, and only subwoofer setups sometimes drop into the 1 Ohm range. In Home Audio, 8 Ohms is relatively standard (6 Ohms for Bose), and 4 is pretty rare except at the high end. So, designing a good amp with the right amount of power is probably pretty well understood, since resistance is generally either going to double, or be cut in half. Headphone amps are conceivably going to drive headphones from 16-600 Ohms, which is a dramatic resistance range. Also, they have to do it with inaudible distortion, and enough power to cover all volume needs.
Given the various factors that matter when it comes to headphone amps, the biggest problem isn't that amps aren't out there. The biggest problem is that they don't give you jack for information to even begin wondering if you've found a good match for your cans (in terms of power / impedance - I assume everyone wants low distortion). I would never by a car audio amp that says it supplies "50 watts between 1-1,000 Ohms" - right, so, 50 watts somewhere in that range. I used to want a Creek Headphone amp, but got tired of the "10mW between 32-300 Ohms".
I knew I wanted about a half a watt of power @50 ohms for the HEs. I knew if I could get that I'd easily drive other headphones also. So I went with an O2. It had published specs that made sense, and it was a lot cheaper than alternatives. Piece of Pie.