True. It's not so much the coloration an amp should have, but even if your amp is technically a "wire with gain," if it's perfectly suited for driving the bass-heavy headphone (output impedance, output power-sensitivity, current or voltage), then that will at a minimum prevent distortion. Note also that bass frequencies require a couple of things for reproduction - surface area and excursion - and for these you need lots of power and good damping factor to control the sub (plus a strong cone that won't physically and audibly distort at high excursion). That's why subwoofers tend to be mostly 8" to 10", and in cars more than home systems, they go up to 15". Specialty subs for blasting Lil Jon or demolishing your house while watching the Battle of Minas Tirith (or convincing the neighbors there's a police chopper on top of your apartment while watching Erica Bana run around Mogadishu) are available at 18" from Kicker and Velodyne (heck, MTX has a 22"). Smaller subs made for a lot of bass tend to have really thick surrounds vs larger subs with comparable output, ergo larger magnets and spiders, ultimately more expensive (and soemtimes offset what some people think will allow for tight installs, particularly in SQ set-ups where bass up front is achieved through really crazy installs that put the subwoofer itself in front). Compare for example the specialty JL Audio and TangBand 6" subs vs 8" subs that cost less. I've been to an autosound show where, during the SPL contest, a Kia Pride (Ford Fiesta to those outside Asia) rolled up sporting two 12W0's and a 1,000w amp in separate ported chambers register at 126db, windshield and rear glass looking like the adhesive wouldn't hold them; next car was a Nissan 350Z that was also competing in the SQ category, with a JL 8W7 in a large sealed box with a 1,500 watt amp, the car was so solid (come on, it's a Z vs a Fiesta) we were all shocked when the meter said 138db. That's 12db more than the two 12's, for 500w more (plus a solid car). In either, or any case, a lot of people using this much on their subs typically use around 75w to 125w on their front speakers. Compare that to HT subs of the same price as the amp and sub, and you're probably only at 500w.
Now, putting the bass drivers at your ears might seem like it's drastically different, but not really - just because an amp isn't making 1,500watts for a sub doesn't mean you won't need an amp that has a lot of output and headroom, if at least among CD players with headphone drivers or even dedicated headphone amps. If anything, the main difference is it'll actually be more demanding, because instead of a sub playing 150hz (or lower) and below, you have relatively large fullrange drivers. Considered at scale, these can be on the really large side (like the XB1000), and when excursion happens, it has to reproduce the midrange while pumping full tilt. An amp with the amount and the kind of driving power such a headphone needs will not only prevent distortion, but can also allow for more headroom if you EQ to add more bass (that way you don't pump up the volume and have the midrange going up with the bass when you want to have bass rattling your jaw (the lyrics you're listening to, if any, are probably, mostly, YEEEEAAAHH!'s, Cadillacs, and grillz anyway).
What many might get confused with is how much output you can actually get from, say, a dual battery CMOY, especially with a sensitive enough bass headphone, before your ears bleed. Because unlike cars and HT set-ups, where you have several people like those people standing on the sidewalk waiting for a bus because they're too poor to buy an Escalade with rimmz and tricked out with Audiobahn, them headphonez being fullrange is offset by their position on your head, and the fact that if you try to share that bass with people standing at that bus stop with you means your headphones are probably measuring well over 100db at your earlobes. I've seen a guy using an LCD-2 with a Corda Stapdance, and you can see the earcups pumping and his cheeks vibrating from all that bass. Of course, I can't hear any of it from across the room, unlike the Escalade with an Audiobahn system in it that I can hear from two blocks away.