[Gr8Desire]

After many years away from HiFi, I decided to upgrade my aging Grado 325e's. Comparison graphs like those at http://www.headphone.com/pages/build-a-graph initially looked a like a good idea - particularly as I was starting pretty much at ground zero. I was hoping impartial measurements and consistent reviews, would let me make better decisions. But after some reflection, I am beginning to understand just how messed up the headphone industry appears to be.  

My limited experience has me asking some fundamental questions:
 

• Volume / SPL radically changes a response curve. I have noticed this more with planar headphones than other dynamic phones I have tried/owned. This means a response curve or reviewer's comment will be highly colored by different volumes levels.  But what are those review conditions?  Why are testers not using standard SPL levels for frequency response measurements and their reviews?
   
• Almost no one is doing double blind tests. Instead reviewers describe products in isolation or compare based on their memory of competing products. How can that possibly work? Isn't double-blind testing essential for both accuracy and credibility?
   
• No standardization. When I bought amps and speakers 30 years ago, standards allowed for some rudimentary comparisons. THD and RMS power measurements were common practice. When I started looking at headphones, I got almost nothing I could use. I continually read specs like 20Hz to 40kHz frequency response. What good is that?  Why do headphone specs seem to universally lack measurement constraints?

My impression: Despite the length of time headphones have been manufactured, the market is supremely immature. I have to ask: Why is there no demand for better measurements and more reproducible review procedures?

Am I doomed to relying on Amazon Prime to review new products and then return them 30 days later? This really doesn't work for me.  Without competing products to test, I don't trust my memory.  Who can? 

Does anyone know of web sites that follow standardized sets of measurements?

Are there any retailers who do double-blind headphone auditioning of a wide range of products? At this point, I would be willing to jump on a plane to do some proper comparisons.
 

 

[Music Alchemist]

I would like to see more objective data like this as well, but the simple fact is...it's hard. Not many people are willing to go to the trouble of all that, and even those of us who would are unable to due to budget constraints.
 
Here's a good place to start: http://www.innerfidelity.com/headphone-data-sheet-downloads
 
But just for future reference, a thread like this belongs in Sound Science.

 

[InebriatedGnome]

I found myself asking these questions a while ago. I really don't know why there's so little standardisation in reviewing practices; it's a little distressing considering how much money one can put down on a hifi setup. NwAvGuy had some interesting (and divisive) opinions as to why so many reviewers shy away from double-blind tests; if you missed the controversy surrounding his blog, that could be a good place to start. Still, he doesn't post anymore and I honestly haven't been able to find any professional reviewers who consistently blind-test their gear.
 
Audio stores in NYC have been helpful in that most will let me A/B/X-test a few pairs of headphones at once (on my own setup), which has been useful whenever I've tried to upgrade. However, I don't know of any specialist headphone stores with a wide inventory here, so if you're looking to break out of the Grado mold I can't help you. I wish you the best of luck with your search.

 

[AnalogSavior]

  I would like to see more objective data like this as well, but the simple fact is...it's hard. Not many people are willing to go to the trouble of all that, and even those of us who would are unable to due to budget constraints.
 
Here's a good place to start: http://www.innerfidelity.com/headphone-data-sheet-downloads
 
But just for future reference, a thread like this belongs in Sound Science.

 
I agree with the above, and second using innerfidelity as a starting point.
 
Quote:

Am I doomed to relying on Amazon Prime to review new products and then return them 30 days later? This really doesn't work for me.  Without competing products to test, I don't trust my memory.  Who can?

 
That's what I do, and while tedious, I end up with gear I like. The only way to know for sure is to listen to them yourself.  And keep a journal of notes of your impressions rather than relay on memory.
 
Final bit of unsolicited advise, objective data can tell you a lot, but let your ears decide what you like, not the graph.  Remember, the end goal is to enjoy the music.

 

[tomb]

I would like to see more objective data like this as well, but the simple fact is...it's hard. Not many people are willing to go to the trouble of all that, and even those of us who would are unable to due to budget constraints.

Here's a good place to start: http://www.innerfidelity.com/headphone-data-sheet-downloads

But just for future reference, a thread like this belongs in Sound Science.

Agreed. A thread like this is not a good idea for the beginners section.

 

[Gr8Desire]

Agreed. A thread like this is not a good idea for the beginners section.

Fair enough but I am a beginner .   I have bought a returned  half a dozen headphones and a pair of amps in 4 months.   That is the extent of my experience aside from buying a pair of Grados 10 years ago. 

In an age of self-driving cars I am bit surprised to find headphone market is so subjective. Where measurements are used, they are too variable and unconstrained to be of much value to me.

I am trying to understand why buyers in this market don't demand more from vendors. 

Cheers.

 

[tomb]

 

Agreed. A thread like this is not a good idea for the beginners section.

Fair enough but I am a beginner .   I have bought a returned  half a dozen headphones and a pair of amps in 4 months.   That is the extent of my experience aside from buying a pair of Grados 10 years ago. 

In an age of self-driving cars I am bit surprised to find headphone market is so subjective. Where measurements are used, they are too variable and unconstrained to be of much value to me.

I am trying to understand why buyers in this market don't demand more from vendors. 

Cheers.

 
OK, but maybe you could start out again with one question at a time.  It might be best to keep them in terms of simple questions, too, rather than stirring up an objective-subjective debate right away.  Also, regardless of where you stand on the issue, blind-testing becomes a political question in this arena.  More often than not, it becomes a hammer that people attempt to use in proving their position on objectivity/subjectivity.  A lot of what you express as opinion can start a fur ball of posts, similar to asking questions about cables.
 
The posters who've suggested you study up on innerfidelity.com are correct.  Tyll deals with a lot of these issues in detail, while in a controlled format.  He also offers up more measurements than you could ever think of requesting.  He is the entrepreneurial standard for headphone/equipment measurement.  Besides what Musical Alchemist suggested, that's the other issue - why would a mfr come out with some measurements based on an arbitrary standard that doesn't really exist?  They would immediately be put to disadvantage with another mfr who used a more favorable standard.  Perhaps Tyll's standards will achieve universal recognition someday.  I think that's what he would like to see, eventually, but the market is not there, yet.
 
I'm not a mod, so don't take this as anything other than friendly advice. 

 

[billybob_jcv]

I think the OP is fooling himself if he thinks that the lack of objective data is limited to headphone hi-fi. Headphones are no different than the rest of the audio world. There is a large and never-ending debate among the subjectivists and objectivists on hi-end speaker forums just as there is on high-end headphone forums. However, I think the real debates occur around the electronics, not the transducers - and those electronics debates are the same across all of audio.

Personally, I like to think I'm one of the silent majority that thinks both sides are zealots that like to hear themselves f@rt pearls of wisdom. I think it's cool when my own subjective opinions are backed-up by measurement data that makes logical sense, but I don't pretend that I understand all the mental biases and physical interactions that create the sounds I hear. Sometimes, what I hear is what I hear, regardless of the measurements, and I don't really care whether it's my imagination or an electrical/physical property that I do not yet understand. I close my mind to neither the science nor the perception.

 

[Gr8Desire]

   
OK, but maybe you could start out again with one question at a time.  It might be best to keep them in terms of simple questions, too, rather than stirring up an objective-subjective debate right away.  Also, regardless of where you stand on the issue, blind-testing becomes a political question in this arena.  More often than not, it becomes a hammer that people attempt to use in proving their position on objectivity/subjectivity.  A lot of what you express as opinion can start a fur ball of posts, similar to asking questions about cables.
 
The posters who've suggested you study up on innerfidelity.com are correct.  Tyll deals with a lot of these issues in detail, while in a controlled format.  He also offers up more measurements than you could ever think of requesting.  He is the entrepreneurial standard for headphone/equipment measurement.  Besides what Musical Alchemist suggested, that's the other issue - why would a mfr come out with some measurements based on an arbitrary standard that doesn't really exist?  They would immediately be put to disadvantage with another mfr who used a more favorable standard.  Perhaps Tyll's standards will achieve universal recognition someday.  I think that's what he would like to see, eventually, but the market is not there, yet.
 
I'm not a mod, so don't take this as anything other than friendly advice. 

 
I am not criticizing anyone who is trying to listen for themselves. That clearly is one of the test metrics that matter most.  But sharing related insight is difficult without some correlation to quantitive assessment. The two are not mutually exclusive. In fact, opinion and facts generally work very well together.

  

I disagree that testing headphones is like cables. Headphone test metrics are more fundamental. To me, buying a $1000 headphone without any comparative measurements, is like investing in a business with no accounting numbers.  Unfortunately, as a headphone newby, that's indeed what I am being asked to do.

 

[billybob_jcv]

How is this any different than being expected to know how speakers will sound in your room, when all you have is a FR graph obtained in an anechoic chamber?

 

[Gr8Desire]

I think headphones are a more controlled environment where measurement correlates better with actual user experiences.  

I have had good experience buying speakers back in the day, based on standardized NRC measurements http://www.soundstagenetwork.com/index.php?option=com_content&view=article&id=16 

That's why I find the headphone market somewhat backward.  The good manufacturers seem to be doing themselves a disservice by focussing so much on anecdotal metrics.  

 

 

[Music Alchemist]

Here's an interesting tidbit: two headphones can have the same frequency response, yet sound drastically different due to driver design and many other factors. Same goes for when you equalize different headphones to have the same (or at least very similar) frequency response.
 
But actually, one of the reasons I said this thread belongs in Sound Science is because we're not supposed to discuss blind testing, ABX tests, etc. in the other forums.

 

[ProtegeManiac]

My limited experience has me asking some fundamental questions:

 

Well, for one thing, reviewers especially in the internet age aren't really pros but consumers handed a review product. The only difference vs hi-fi magazine editors in general though is that the latter have that specifically as their job - I mean it's not like those magazine editors take their own measurements the same way Road and Track and MotorCycle editors would take data around a race track.
 
Quote:

 

• Volume / SPL radically changes a response curve. I have noticed this more with planar headphones than other dynamic phones I have tried/owned. This means a response curve or reviewer's comment will be highly colored by different volumes levels.  But what are those review conditions?  Why are testers not using standard SPL levels for frequency response measurements and their reviews?

 
Reviewers not being pros in the fullest sense of the term, many of them don't even have the tools necessary to take those measurements. And even if they did, like I mentioned in another thread (forgot already which one), even if all they do is take measurements, it's still subjective to each listener. Why? Clamping force and other fit/ergonomic factors can also affect headphones, and using a dummy head just means the reviewer has no idea if the dummy head is wearing it like he would, unless the dummy head is at minimum cast from the reviewer's own head (including earlobes) so instead of measuring pressure points and all that he can put them on first then put them on the dummy head.
 
In any case, there are standard measurements available on Headphone.com and Inner Fidelity.
 
Quote:

 

• Almost no one is doing double blind tests. Instead reviewers describe products in isolation or compare based on their memory of competing products. How can that possibly work? Isn't double-blind testing essential for both accuracy and credibility?

 

Here's another problem when you're not a pro: aside from not getting paid for it, you don't have a staff to help you, and it's hard to do a double blind test when there's only one person at it when you need at least three. I tried doing that at home, but everyone I asked to help me out got annoyed, sometimes after I got annoyed that they can't take my instructions properly at the start (thus wrecking the whole point of "double blind" when I have to tell them to play which track at which resolution, even when I have the playlist set-up already). Many of the reviews are thrown up there for a lot of basic info which is better than no info, but the problem is how they are presented and how people take them. On one hand you have people doing qualitative assessments with flowery descriptions and metaphors, then you have people reading it thinkign that's how it should be and don't take it with a grain of salt (like, at what dB are they listening, and how long can you do that - the O2 if you listen at safe listening levels won't be beat by a lot of pricier amps until you crank it up, and at that point youre not supposed to listen to that too long anyway).

 

Nowadays when I review amps and I like them what I say boils down to "unremarkable" but by that I mean "I don't hear it doing anything wrong with what headphon/speaker I tried with it."

 

Quote:

 

• No standardization. When I bought amps and speakers 30 years ago, standards allowed for some rudimentary comparisons. THD and RMS power measurements were common practice. When I started looking at headphones, I got almost nothing I could use. I continually read specs like 20Hz to 40kHz frequency response. What good is that?  Why do headphone specs seem to universally lack measurement constraints?

 

I don't think that's specific to headphones, I've seen speaker equipment do the same things. Ribbon tweeters quoted for up to 100khz while some dome tweeters are quoted for up to 30khz, subwoofers uoted down to infrasound frequencies but none of the reviewers ran out of the HT room after they "saw" ghosts, HT receivers that say "175watts @ 8ohm one channel driven," one amp says "50watts @ 8ohm, 10%THD" while another amp says "35watts @ 8ohm, 0.01% THD," etc.
 
 
 

  I think headphones are a more controlled environment where measurement correlates better with actual user experiences.  

 

Not really, the room might not be a factor but the head can still be. It's not just a matter of ergonomics and comfort, this can affect SQ too. When two people disagree on the imaging of a headphone it's not always because one is hallucinating and the other is objective (or that one is in a way toen deaf, except for imaging, given that's the last thing people learn to appreciate in hi-fi regardless of the transducer type used), but because one is wearing it differently. Just wearing a headphone with the drivers more aligned with the ear canal results in a more typical headphone soundstage; even headphones with angled pads or drivers can still get that problem if one wears any of them with the drivers smack over the ear canal since it's the commonsense way of wearing headphones (despite the fact that no one actually puts speakers at those firing angles).

 

Then there's the clamping force and earpad wear which can affect the overall tonal balance - I've always wondered how much of the HD650 and HD600 "dark sound" is attributable to confirmation bias helped along by trying them out at a meet after a dozen other people, which on top of a stock angle headband frame, would have been squished by all those people that used it already.

 

Speakers would actually be better in that regard though, assuming all people would actually invest on their listening room more than the speakers, but some want them in the living room or some other area of hte house that's supposed to look like a room in a house instead of a padded madhouse isolation chamber (which, according to the internet, is what it will feel like if you stay in the world's quietest room).

 

[tomb]

 

   
OK, but maybe you could start out again with one question at a time.  It might be best to keep them in terms of simple questions, too, rather than stirring up an objective-subjective debate right away.  Also, regardless of where you stand on the issue, blind-testing becomes a political question in this arena.  More often than not, it becomes a hammer that people attempt to use in proving their position on objectivity/subjectivity.  A lot of what you express as opinion can start a fur ball of posts, similar to asking questions about cables.
 
The posters who've suggested you study up on innerfidelity.com are correct.  Tyll deals with a lot of these issues in detail, while in a controlled format.  He also offers up more measurements than you could ever think of requesting.  He is the entrepreneurial standard for headphone/equipment measurement.  Besides what Musical Alchemist suggested, that's the other issue - why would a mfr come out with some measurements based on an arbitrary standard that doesn't really exist?  They would immediately be put to disadvantage with another mfr who used a more favorable standard.  Perhaps Tyll's standards will achieve universal recognition someday.  I think that's what he would like to see, eventually, but the market is not there, yet.
 
I'm not a mod, so don't take this as anything other than friendly advice. 

 
I am not criticizing anyone who is trying to listen for themselves. That clearly is one of the test metrics that matter most.  But sharing related insight is difficult without some correlation to quantitive assessment. The two are not mutually exclusive. In fact, opinion and facts generally work very well together.

  

I disagree that testing headphones is like cables. Headphone test metrics are more fundamental. To me, buying a $1000 headphone without any comparative measurements, is like investing in a business with no accounting numbers.  Unfortunately, as a headphone newby, that's indeed what I am being asked to do.

Try reading a little more closely sometime ...
 
Subject: "blind testing" ... note several sentences that come afterward - before a new paragraph is started.  As in this one: "A lot of what you express as opinion can start a fur ball of posts, similar to asking questions about cables."  The subject was still about blind testing.  As others have mentioned, discussions about blind testing are not acceptable in certain sections of this forum; discussions about cables are similarly limited, if memory serves.  As I tried to explain (read carefully), such questions can end up in discussions that have no good ending.  It detracts from the real effort here, which is to try to give people good advice (see below - read carefully again!).
 
Nowhere did I state that "testing headphones is like cables."  You chose to say that.
 
No one is asking you to buy a $1000 headphone without looking at measurements. Have you even looked at Tyll's site?  You've had multiple posts already telling you to do that.  You can find extensive measurements on probably every $1000 headphone out there on his site.
 
No offense, but if you choose not to follow the advice given, then your head must be in the sand or you really like being obtuse.

 

[Music Alchemist]

A few more resources for you:
 
http://www.headphone.com/pages/build-a-graph
http://en.goldenears.net/
 
The thing is, though, you will have to know HRTF curves (which are rarely superimposed on graphs) to really understand headphone measurements.
 
Here is an example of that:
 

 
This graph is for the Focal Spirit Professional, my current favorite headphone. Notice how closely the grey line (raw headphone measurements) follows the black line (HRTF curve that emulates how human ears hear a flat-tuned speaker system), especially in the bass, with only one major deviation in the higher frequencies.