I agree that the amount of power a headphone amp can cleanly deliver to the headphones of your choice is quite important.
Some headphones are really very efficient and work fine with just about any outboard amp, power-wise. I had a pair of Grado GS1000's and even though they were "low impedance" at 32 ohms, they seemed to work with any amp I tried them on at the time. (Although I did not have a Bottlehead Crack at the time, I would guess it would be "iffy" on that amp.)
Some OTL tube headphone amps can't provide enough power into low-impedance phones, the Bottlehead Crack is one of these- it is not intended to work with low-impedance phones. So, if you have low impedance phones you have to make sure that the amp you choose will be happy driving low impedance loads.
This "low impedance vs. high impedance" thing is not the same thing as the interaction between an amp with the frequency-dependent impedance curve a given set of headphones. This is simply an issue of power- can the amp put out the current to drive low impedance phones. Some just can't, like the Bottlehead Crack.
So, there's that.
Then there is the issue of their being several well-like headphones out there that require a lot more power to drive than most other 'phones. The AKG k1000, the HiFiMan HE-5 and especially HE-6, and the Audeze LCD-2. My HE-6 just doesn't play loud enough on any amplifier not capable of at least one watt output into their 50 ohm impedance. HiFiMan says the amp should have at least 4 watts output capability, but in my experience an amp with one clean watt - like the Musical Fidelity XCAN V3 for example- can drive these OK. The portable amps I've tried don't have the power to drive these, and neither do the OTL tube amps I've tried- Bijou and Little Dot MKIII were sorta OK on the HE-6, but left me wanting more power. Even the M³, an otherwise capable solid-state amp, ran out of steam driving the HE-6. The Audeze LCD-2's are not quite as power hungry as the HE-6's, but they want more power than the Crack can deliver, and the Bijou starts to fold up in the bass when played loud into the LCD-2's. LCD-2 is also a 50 ohm 'phone, and of lower efficiency than most other headphones. LCD-2 and HE-6 are planar type headphones, like a Magneplanar speaker, and this design is less efficient therefore requiring more power.
Aside from this handful of headphones that requires power fed to them in WATTS, most other headphones are happy with a few hundreds of milliwatts. (A milliwatt is 1/1000 of a watt.) An iPod can't deliver that much power, but almost any external hi-fi headphone amp can. Phones like Sennheiser HD600/HD650 and HD800 can play loudly off just about any headphone amp, although typically to have good bass the amp should have enough power capacity so that it is not straining to drive the headphone at your chosen volume level. So my experience with these headphones (I've owned all three) is that they are OK on just about any headphone amp, but you get extra goodness with amps that have MORE than enough power.
So, really, the power thing comes down to:
1. Can it drive the headphones to your desired loudness at all?
-and-
2. Does the amp have enough power not only to drive the headphones, but to drive them without ANY strain?
Amps that satisfy #1 let you listen and enjoy, and then there's #2 which gives you that extra "more" that audiophiles are always seeking.
So, yeah, power does make an important overall difference.
In my experience, headphone amps vary from about 100 mW at 50 ohms to about 8 watts (!) at 50 ohms. At 300 ohms, you'll see more like 300 mW to 4 or 5 watts as the range of what various amps can deliver.
When I was talking about impedance curves, though, I was talking more about "voicing" type interactions between the driving amp's source impedance and the headphones load impedance, both of which vary somewhat with frequency.