Not if the impedance is matched correctly. I find that modern audio components are designed to be audibly transparent... inaudible levels of distortion, flat response, super low noise levels, etc. It isn't like it was when I was starting out in this hobby in the late 60s. That is nice, because you can focus on what really counts... the transducers.
People who hear differences between amps probably are using them with cans they weren't designed for.
Speaking of impedances - not to go too OT here - a few of my good old buddies(yeah, right!) over on Usenet rec.audio.pro said that those impedance numbers on the backs of consumer amps, receivers, and speakers don't matter, as long as you don't drive concert level volumes for hours.
I replied in the context of, well, they're there for a reason, just like the recommended cold tire inflation pressures on the sticker on the door pillars of most cars nowadays. They said, basically, your loss, as in limiting yourself to speakers that match only the impedances of your amp. So I asked them, then what are they there for? "As a guide to showroom salesmen", they replied!
So according to them, with any home audio gear made in the last, say, thirty years, anything goes! "Wanna drive 3-4ohm speakers on your 8-16ohm rated receiver? Knock yourself out." I stated my respectful disagreement with them on the subject, to which 'None' added that I was "too fu-king dumb to read numbers anyway, and to put my helmet on and get back on the short bus".
Nice bunch a fellas, ehh?