Generally speaking, it is easier to design a headphone that performs better that has a high impedance. A higher impedance allows for more turns of wire to be used in the voice coil and this results in a better 'motor system', higher impedance headphones also require less current, which reduces distortion. Furthermore, higher impedance headphones are more resistant to poorly-designed amplifiers that have high output impedance. If your amplifier has an output impedance that is more than 8x lower than the headphone's impedance, you can experience some degradation in sound quality, particularly in the bass as damping is reduced.
So for example, if you are using a 600 Ohm headphone, your amp's output impedance needs to be lower than 75 Ohms. If you have a 32 Ohm headphone, the output impedance needs to be lower than 4 Ohms. Nearly all amps available today do not have output impedance that exceeds 75 Ohms, but it is somewhat difficult to find an amp that has an output impedance that is lower than 4 Ohms. Ideally, an amp should have an output impedance as close to 0 Ohms as possible, but in practice this is rather rare.
Nowadays, since we have a lot of very good portable sources, headphone makers often design their flagships with low impedance in order to make them compatible with portable gear.
In the end, it all depends on your amplifier. If you have a well-designed amplifier with low output impedance, it is safe to assume that low-impedance headphones will perform very well with them. On the other hand, if your amplifier has high output impedance, it is a bad idea to use low-impedance headphones with them as you will get degraded sound quality. Furthermore, many tube amps also are known to perform poorly with low-impedance headphones.
With today's technology, it is entirely possible to design a low-impedance headphone that performs extremely well, you just have to be aware that some of your older sources might not be compatible with them. I actually think a lot of people drive their low impedance headphones with amps that have high output impedance and this is what contributes to the general consensus that they sound worse than high-impedance headphones.