I fully believe that there is a huge difference between listening to music loudly through headphones versus being at a concert. Let me tell you a bit about my wife.
For years, I've been listening to her complain that I must be going deaf. Every time we're watching tv, listening to music, etc., she's always telling me to turn it down, it's hurting her ears.
Coincidentally, she happened to have an appointment with an ear doctor and had her hearing checked. Turns out.... she's some kind of bionic woman. Her hearing is registering approximately TEN TIMES what a normal person's hearing should be.
Getting to the point:
You can see my kit in my sig - I've got a Micro Amp and Senn 650's. I've had her listen to the cans when the Micro Amp is at a volume setting that I consider way too low for any kind of enjoyable listening - zero volume is about 6:30-7:00 on the Micro Amp's volume knob. I've had her listen at about 8:00. She couldn't even put the phones on her head - she had to pull the cans an inch or so away from her ears in order for it to not physically hurt her. This is at volumes that were really, really low by my standards.
However... She loves rock concerts. We've been to so many of them. Cranking loud. We saw Roger Waters in the front row. We've seen The Who from about 20 rows back. And many other shows. She didn't complain at all.
I don't think it's just about the volume. I'm no ENT, but it's got to have something to do with how close the source of the volume is to your ears. Don't discount Pete Townsend's comments here - he's been under serious care from ear specialists for decades now....