That would be part of it. I've read from few different people who professionally design high end speakers that past a damping factor of 20 or so, the difference can sometimes be measurable, but is generally not audible. I have no idea how much of this applies to to headphones though. Since the diaphragms have far less mass, electrical damping may not be quite as important so the threshold of audibility for it may be lower than with loudspeakers, but I don't really know.
Headphone drivers are resistance controlled as opposed to mass controlled, and the relatively flat impedance curves of most headphones (every headphone I've ever seen) means that "damping factor" which is a non-sensical, non-quantitative measurement, has far less bearing on headphones. Generally speaking, if the Zout of the source is less than the impedance load of the headphone, you'll hear very little difference in sound. To put it into perspective a damping factor of 5 produces insignificant changes in FQ response with the DT880. A damping factor of 5 with speakers however (which have a more varied impedance curve) will have far more FQ response fluctuations.
I believe the desired term is "impedance bridging", as opposed to "impedance matching", which is reserved for the telecommunications industry. Until somebody (anybody please) can support the hypothesis that the higher impedance beyers measure better than the lower impedance beyers with any physical evidence, I stand by my position that there is no correlation between impedance and sound quality as far as audibility is concerned.
Hundreds of posts on the subject, and all there is is anecdotal evidence (which sucks as far as evidence is concerned).