Kryogen,
There are two main effects of impedance on an amp:
1. Current drive. The amp has to put a certain voltage across a driver to make it move. The amount of current necessary to develop a certain voltage depends on the impedance of the driver. Ohm's law says: I = V/Z, where I is the current, V is the voltage, and Z is the impedance. As you can see, the lower Z is, the more current is required.
2. Because drivers are driven by voltage, you have to worry about the voltage divider formed by the output Z of the amp, and the input Z of the driver, ie. the impedance of the driver. The voltage presented to the driver is multiplied by Zd/(Za+Zd), where Zd is the impedance of the driver, and Za is the impedance of the amp. This assumes that cable impedance is negligible, which is not a safe assumption with headphones because many cables for 'phones are so small that they have significant resistance. If cable Z is significant, it's in series with the amp Z, so the multiplier becomes Zd/(Za+Zd+Zc). For example, Etymotics cables seem to be customized for each different model. And all you Sennheiser HD-600/-580 cables upgraders, take note!
The tricky thing about the voltage divider effect is that Zd often varies over frequency (because it's not just a plain resistor), and Za can vary over frequency. This means different voltage divider values at different frequencies, leading to an altered frequency response that's specific to a particular headphone/amp combination.
What does this mean to you? If you want to drive headphones with the amp being as little in the way as possible, get a headphone amp that has as low an output impedance as possible, and make sure its output impedance is as resistive as possible, meaning, it doesn't change with frequency. This is doubly important for low-Z phones like the Grados, because the voltage divider effect kicks in at lower Za for those phones than high-Z phones like Senns.
(Voltage divider effects kick in at lower Za, because Zd is lower, and lower Za have greater effect, numerically --- plug in a few numbers and try it out, eg. compare Za = 10 Ohms with Zd = 60 Ohms and Zd = 300 Ohms, and see what the multiplier is).
--Andre