To my own understanding, output impedance affects in two ways:
First, dampening. Transducer is a mechanical device, so it has inertia. When you push the diaphragm out with a signal, ideally you want it to go back to its original position immediately after the signal ends before the next signal comes. But in the real world, it will vibrate just a tiny bit before it fully stops. The vibration becomes the noise since it isn't there in the electric signal. There is already mechanical dampening in most transducer design (i.e. the rubber ring that connects the diaphragm to the metal housing), but electronic dampening is better. As the vibration itself creates a tiny bit of voltage and will get fed back to the amp section, the amp section will then compensate / resist the voltage change which forces the vibration to stop quicker. However, this kind of dampening usually works best if there is a big enough difference between the output impedance and the load impedance. 8x is the minimum but some will say 10x.
Second is impedance interaction. The impedance of a headphone changes with frequency. Most of what you read in spec is measured in 1kHz tone. A 32ohm @ 1kHz headphone can have a 16ohm or 64ohm impedance @ 500Hz, or 150ohm @ 15kHz. This is referred as a impedance curve. Sometime you can also have headphone with very linear / flat impedance curve and maintain 32ohm from sub-bass to treble (though it most case, impedance is usually higher at upper treble). The problem is when headphone has a curvy impedance curve - for an example, if a headphone is 4ohm @ 100Hz, 16ohm @ 1kHz and 120ohm @10kHz. When you drive it with an 10ohm output impedance, the headphone will see much less power at 100Hz (as the 10ohm output impedance is big in ratio compare to the 4ohm impedance of the headphone so most of the power will be waste), a little better at 1kHz but fine in 10kHz. This effect will change the final frequency curve of the headphone, thus adding coloration. This is particularly true for multi-way IEM since they tends to have very curvy impedance curve due to the use of crossover circuit. Again, a 8X difference is suggested in order to make sure the output impedance of the amp is far enough away for the impedance curve of the headphone so interaction will be minimum. In the case of IEM, less than 1 ohm is usually preferred since it will make sure the ratio is almost always greater than 8x.
Though might not always being the case, many dynamic driver have more linear impedance curve so dampening is more important. You can have a 32ohm dynamic headphone sounding very linearly / no coloration with amp of 10ohm output impedance, but you might not get as tight a bass since dampening is minimum at best. On the other hand, since balanced armatured tends to be very well mechanically dampened, impedance interaction is more important factor to consider when dealing with multi-way BA.
Hope these help to explain it a further bit.
