[exsomnis]

Here's a summary of what I've considered are the different factors governing evaluation of headphone gear.  
 
1. The music

1. Quality of sound recording
2. Quality of sound data
3. Genre
4. Types of instruments used
5. Volume
6. Recording era
7. Recording studio

 
2. Equipment

1. Reference equipment
2. Source
3. DAC (if applicable)
4. Amplifier
5. Equipment combinations
6. *Cabling only up to the extent that it is at the right rating for the current load required, with impedance due to cable length taken into account where significant

 
3. Quantifiable measurements

1. Measurement equipment
2. Measurement methodology
3. Measurement analysis/interpretation

 
4. Reviewer bias

1. Ability to discern differences in quality
2. Preference for certain characteristics of sound
3. Preference for specific genres
4. Subjective opinion
5. Brand perception
6. Experience
7. Hyperbole
8. Perception of headphone characteristics/limitations
9. State of mind

 
4a. 3rd party bias (evaluation based on another's evaluation)

1. Assumptions
2. Crowd mentality
3. Experience
4. Reference point
5. Unformed preference for sound characteristics

 
What else do you think should be in there?
 
 
*Added quantifiable measurements

 

[wuwhere]

I think that there should be some measurement data - frequency response, impedance curve, impulse, etc... Subjective and comparative reviews, I believe, are not enough, this is were the reviewer bias comes into play and dominates the review.

 

[exsomnis]

Quote:

I think that there should be some measurement data - frequency response, impedance curve, impulse, etc... Subjective and comparative reviews, I believe, are not enough, this is were the reviewer bias comes into play and dominates the review.

 
Good point - quantifiable measurements have been added to the mix.  
 
I'm starting to wonder how much of the differences we attribute to the 'synergy' of different set ups and audio signal processing design are actually due to a mismatch with the amplifier in terms of the required voltage + current to drive the headphones at their rated impedance, sensitivity and the desired loudness.