Low output impedance with high input impedance delivers voltage amplification. High output impedance with low input impedance delivers current amplification.
You generally want your headphone impedance to be at least x8 greater than the output impedance of the amplifier. If the amplifier's impedance is too high, you will start to hear background noise on more sensitive headphones.
Since headphone impedance changes with frequency, the different voltages of different output impedance levels will also change with frequencies. Larger output impedance will result in greater frequency response deviations, and can lead to noticeable sonic changes. May tune this to your preference.
Also, larger output impedance has reduced electrical damping. Decreasing electrical damping will most noticeably effect the bass performance, leading to a boomy looser bass with slower transient response and rolled-off extension. This can result in a warm tube-like sound. I believe this is why people say that high gain can give you a warm bass boost, while low gain offers a more transparent sound.
Generally, people seem to think that low impedance/low gain is superior for accurate reproduction ("solid-state amp sound"), while higher output impedance/low gain may make your music sound rich, full, and 'euphonic' ("tube amp sound"). Really personal preference I guess, so you can play around with the settings and see what you find & like.
Hope this was helpful! :)