I've got a few more questions now:
Do you mean that all all-pass filters cause large phase shifts, but only in multi-driver headphones? What about an L-R filter, which I've read behaves like an all-pass?
I looked at the phase response vs impedance on InnerFidelity for the 120 Ohm DT 48 E (http://www.innerfidelity.com/images/BeyerdynamicDT48E120Ohm.pdf) and it looks about as flat as I've ever seen on a headphone graph. Am I understanding the graph incorrectly?
Finally, I don't think my question a while back was addressed; it got lost at the end of page 3!