This isn't true. Look up measurements of iPods (the most popular device). The conclusion is always that they're surprisingly good. Look at the Sansa Clip+, it has fantastic measurements, and bests much more expensive headphone amps. Yet it sells for $40, and is tiny. Sure, the voltage swing may not be large enough to attain sufficient volume on very high impedance cans, but that is a different issue than it sounding bad. If your headphones cannot get loud enough, the headphone's specs do not pair with those of the amp. Those specs (power output of the amplifier, headphone impedance and sensitivity) are given by the manufacturer. But I rarely see anyone citing power ratings or mentioning sensitivity here. It has always been "high impedance headphones? BUY AN AMP NOW", as if something magical happens when an amplifier has it's own aluminum housing.
In my experience, many portable devices do have enough voltage swing for 300 ohm headphones with reasonable sensitivity. Many of these headphone companies realize that selling >600 ohm headphones with 50 db/mW is stupid because consumers use things like iPods primarily. Why do you think so few companies sell electrostatic headphones? They sound fantastic, but require substantial amplifiers. So I am not surprised that I can drive my HD600s well with every source I have. That is simply good engineering on Sennheiser's part.
A very low impedance headphone is much more likely to sound bad due to the source+amplifier than a high impedance headphone, but the opposite is constantly stated, which is why it is a myth. I've tried low impedance IEMs on my laptop... they sound absolutely terrible. They pickup all kinds of weird noises and sound super muddy... but you don't find people here saying that sensitive IEMs need a dedicated amplifier with low output impedance, do you?