Lejaz, your electrician friend is quite right but not in the context of headphones.
Matching the input impedance of the load to the output impedance of the source is necessary in electrical power transmission for maximizing power transfer and minimizing reflections from the load.
The ratio of the input impedance of your headphone to the output impedance of the amplifier should be as high as possible to avoid interacting with the load. I am guessing that your adapter is increasing this ratio by raising the input impedance of the headphone. This can be useful for an amplifier with a high output impedance. A guy called Paul Spelz sells such a device for loudspeakers, typically when driven by tube amplifiers.