The output impedance (Zout or Zsource) and the headphone's impedance form a voltage divider. High Zout (= low dampening factor) will cause frequency response variations.
Simple example:
A headphone has an impedance of 50 ohms at 1000 Hz and 100 ohms at 100 Hz.
Zsource is also 50 ohms (= high), that results in a damping factor of Zload/Zsource = 1.
Signal is 1V, 1000 Hz: the load is 50 + 50 ohms, the headphone will see 0.5V.
Signal is 1V, 100 Hz: the load is 50 + 100 ohms, the headphone will see 0.66V.
Thats a difference of about +2.5 dB at 100 Hz.
(like using a parametric eq with a peak at 100 Hz with +2.5 dB, bandwidth depends on the impedance peak of the headphone)
Headphones often just have one broad impedance peak around 100 Hz and rising impedance towards 20 kHz.
Speakers have more peaks, which can be quite narrow and that can be a bigger problem.
I've read in an article (blind test with different output impedances) that with speakers a factor of at least 10 is desirable.