For preamps, a moderately high output impedance shouldn't make a big difference. If the output impedance is too high (above 10k), however, your system will be much more susceptible to picking up RF/EM interference, and you'll need to be careful to use well-shielded cables. Also, with such devices, it's wise to make the interconnects as short as possible, because you risk having high frequency rolloff from the capacitance of the interconnects.
With headphone amps, you want an output impedance less than the headphone's impedance. The damping factor of an amplifier is given by headphone impedance / amplifier output impedance. Generally larger damping factors give better control over the bass. The output impedance of an amplifier may also affect the frequency response of the headphones. For more info on that, see the article on the Meier Audio website.