The definition of a buffer is a device that allows a circuit to drive a lower impedance than it could otherwise.
An opamp has a maximum current that it can output. Sometimes this is expressed on the datasheet as Output Short-Circuit current. If you load an opamp with a low-impedance load that would draw in excess of the max output current, the opamp will misbehave. The output current may collapse or the output may clip.
So you put in a buffer. This will prevent the overall circuit misbehaving up to its max output current, i.e. with a low impedance load. What happens now if the load is high impedance (draws a small current)? Nothing, or very close to nothing. The buffer may slightly reduce the output voltage, it may slightly increase the distortion, but although more current is available, the load doesn't make use of it, so the buffer is unnecessary, until you want to use a low-impedance set of phones.
That's all she wrote (on that subject).
The audibility of opamps is a highly contentious subject. This thread http://www.diyaudio.com/forums/everything-else/245455-ultimate-opamp-shootout-where-you-get-decide.html tends to suggest that there are audible differences in blind testing, but the test loads the opamps heavily (~580 ohms), sometimes outside what they are rated for, then uses 2 circuits in series to exaggerate the effect, so the conclusions are of dubious value. It does look, however, as though at least some individuals can detect an NE5532 in these circumstances