Thanks for the answer about voltage but I also wonder about output impedance. Is it the same output impedance on the headphone output and the XLR outputs when configured as single-ended? And I assume that the impedance is different when using the rear outputs as balanced compared to SE.
The relevance of the output impedance is that it affects the frequency response of the headphone (if the headphone impedance is frequency dependent and that is common).
You are correct to state that the impedance of the source, driving a signal into a non resistive load (headphone), does alter the flatness response. If a headphone were purely resistive load, then a resistive source will only cause attenuation, and the impact would be the same across the audible frequency range. But headphones are not a pure resistive load.
As a "rule of thumb", a lower load impedance and a higher headphone impedance tend to minimize the flatness response issue. Also, as a rule, a resistive source is better then inductive or capacitive.
The XLR outputs heave a limited current drive. They offer a lot of voltage. They are not meant to drive low impedance loads (they can drive 300 Ohms OK, but not much lower). The 75 Ohms is a pretty standard thing to do - the purpose is to enable driving long lines that end up looking like capacitive loads at audio frequencies. The series resistance "buffers" that capacitive load to prevent oscillations.
For a headphone output, the issue is not about oscillation prevention. The cables are not very long to start with, and different headphone introduce a very wide range of load impedances. Indeed, with a very low headphone impedance, a 75 ohms in series is not the way to go. Close to zero Ohms provides a near flat response...