Headphone Measurements: The New Standard, Part 1

[playmusic]

[...] the solution to measurement variances is to repeat the measurements multiple times and take an AVERAGE of the data, something which you have no doubt done in high school science experiments.

I wish I could agree because it would simplify things.
But in this case of headphone measurement it is a bit more complex.
The placement of the headphone varies between measurements.
So, it is necessary to either define a single fixed placement standard as suggested by @Thorsten Loesch or to define a specific method for random selection of placements. You might want to have a look at Bertrand's probability paradox which shows that taking averages of random placements is not well defined.

Either way, the problem derived from @Thorsten Loesch 's observation remains: Different placements (even if very small) lead to different measurements but to equal (or very similar) listening impressions. Why would the listening impression be adequately characterized by a single defined placement or by the average of a random selection method of placements?

 

[nanaholic]

I wish I could agree because it would simplify things.
But in this case of headphone measurement it is a bit more complex.
The placement of the headphone varies between measurements.
So, it is necessary to either define a single fixed placement standard as suggested by @Thorsten Loesch or to define a specific method for random selection of placements. You might want to have a look at Bertrand's probability paradox which shows that taking averages of random placements is not well defined.

Either way, the problem derived from @Thorsten Loesch 's observation remains: Different placements (even if very small) lead to different measurements but to equal (or very similar) listening impressions. Why would the listening impression be adequately characterized by a single defined placement or by the average of a random selection method of placements?

You are over-thinking the issue.

Even simple experiments like measuring temperature of liquid with the thermometer being placed slightly off would technically caused variances that shows up in the data but we can't feel with our skin. The laws of average is absolutely acceptable in scientific measurements. In science as long as you can repeat the experiment and consistently repeat the results within a certain amount of variance, then the result is considered to be credible, methodology is there to minimise variance, but everyone knows it is not possible to completely eliminate it. The way you are framing the issue, you are dismissing science entirely, not just headphone measurements.

 

[Thorsten Loesch]

Thank you for your extensive reply.

I only recently started to look into audio measurements.
If the statement in the above quote is correct (and I have at the moment no reason to doubt it) then this is a very fundamental problem:
Small variations in placement change the measurements significantly but do not lead to audible differences for the listener.
This would lead to the question: Why make measurements of headphones at all?
Even if a protocol to place the headphone is part of the standard (as suggested in the post above) then I wonder what would be the point of it.
This would be a standardised single instance of a spectrum of possible measurements for a headphone.
I do not see how this can be useful in assessing or comparing headphones.

Some weeks ago I thought that the fundamental question in comparing headphones is as follows:
Given two reasonably decent headphones (i.e. no extreme frequency response across the audible range and sufficiently low distortion), is it possible to equalize one headphone so that it sounds like the other headphone?
(Background for this question: I was inclined to think that a Stax SR-009s sounds fundamentally different to a, say, Sennheiser HD 800S. So they would not differ only by frequency response, but the 009s sounds 'somehow' much more detailed.)

But now @Thorsten Loesch 's post makes me wonder why to have headphone measurements at all.

> Small variations in placement change the measurements significantly but do not lead to audible differences for the listener.

This is my own and no doubt many others experience in the daily use of headphones. I can easily tell the difference between a Sennheiser HD600 and a Fostex T-50RP, but I do not perceive any notable change in sound from placement, other than the seal of pads.

The variability of test results depends on the type of fixture. If using a HATS type device it is largest, the smallest variance I find with IEM's using a 711 Coupler with an aid to ensure consistent insertion depth and for over ear headphones the IEC60318 Fixture is much more consistent than HATS style devices.

If you look at the IEC60318 fixture from B&K it will be clear why consistency is better than HATS:

TYPE 4153 ARTIFICIAL EAR WITH IEC COUPLER

> This would lead to the question: Why make measurements of headphones at all?

It has been shown that frequency response of playback systems (speakers and hearables) has the largest influence on preference. So obviously, a designer would like to design so that their product meets this preference. A reviewer would like to show the frequency response to illustrate listening impressions and give consumers an objective means to compare headphones. So taking measurements is valid, but accuracy AND repeatability/portability of such measurements is important to make them meaningful.

> Given two reasonably decent headphones (i.e. no extreme frequency response across the audible range and sufficiently low distortion),
> is it possible to equalize one headphone so that it sounds like the other headphone?

It appears this is possible, at least so that the difference cannot be reliably detected by small scale Double Blind Listening tests.

I would however suggest that even after equalisation other factors will remain, that may be detected by experienced listeners even under blind conditions, specifically distortion and impulse response.

Thor

 

[Thorsten Loesch]

You are over-thinking the issue.

Even simple experiments like measuring temperature of liquid with the thermometer being placed slightly off would technically caused variances that shows up in the data but we can't feel with our skin. The laws of average is absolutely acceptable in scientific measurements. In science as long as you can repeat the experiment and consistently repeat the results within a certain amount of variance, then the result is considered to be credible, methodology is there to minimise variance, but everyone knows it is not possible to completely eliminate it. The way you are framing the issue, you are dismissing science entirely, not just headphone measurements.

> The laws of average is absolutely acceptable in scientific measurements.

Yes, however, when performing a wide range of audio tests on a wide range of devices the variance between repeated measurements is VERY SMALL, small enough to not need to measure 20 times and then take an average. This does include loudspeakers.

When using known inaccurate test systems (711 Coupler, 43 Coupler, IEC60318 Coupler) the consistency is also very good with a little care. Yet these systems have severe problems in the measured response above a few kHz, which are documented and known.

Hence my question how we can call a device accurate if repeated measurements show a wide variance?

A good test system for hearables should be both accurate and consistent, in fact, if it is not consistent, I would not accept the claim to accuracy.

So while the B&K 5128 HATS has much improved accuracy for such a system at high frequencies, the accuracy does not extend to repeatability.

Hence my suggestion that a better fixture design coupled with the accurate acoustic system from 5128 would offer a better solution to testing hearables (and likely would result in a lower price as well) than is represented by the 5128 HATS.

Thor

 

[Adnen Ayed]

It's been busy at Head-Fi HQ, but I am still working on Part 2 of the article, and will post it as soon as I can, and it will of course include more measurements.

On the topic of doing headphone measurements: So that we could fit the Brüel & Kjær 5128 into the isolation enclosure with the 45CA, we reconfigured the 5128-C into a 5128-B for now (using the Brüel & Kjær UA-2180 support foot). On the full torso only one fixture or the other can fit on the pneumatic platform inside the enclosure -- but with the disembodied 5128, both fixtures fit on it.

​

​

Fig.1: Brüel & Kjær 5128-B (using UA-2180 support foot) with the GRAS 45CA​

Again, there will be more measurements in Part 2. And after it's posted we will thereafter be posting measurements regularly (we'll try posting them weekly to start). We have many 5128 measurements dating back to last December (though there weren't many performed during our lockdowns here in Michigan between March and June), and hundreds of measurements from the 45CA over the years.

Also related to doing headphone measurements: When we've done headphone measurements with visiting industry guests, they'll regularly comment about the measurement notes we log in our Audio Precision APx projects. One headphone engineer I recently screen-shared with in a video call suggested we post examples of these on the forums, thinking a few of you may find these interesting, too.

Here's an example of measurement notes for IEMs (Fig.2 below). Each line represents one sweep (from which many measurements can be derived). Each seating is noted, and we usually run at least two sweeps per seating. We record which fixture was used, whether the measurement was uncompensated or compensated, the voltage level, who did the seating, which ear tips were used, and any other noteworthy things.

​

​

Fig.2: Screen shot of some of the measurement notes for IEM measurements​

When we do AE/OE (around-ear/on-ear) measurements (Fig.3 below), we also note anything worthy of mention about position over the ears, headband size setting in clicks away from the smallest setting if the headband sizer has detents (or in estimated distance if it doesn't).

NOTE: I previously discussed how headband size settings can affect sensitivity, with measured examples: Measuring headphone sensitivity

​

​

Fig.3: Screen shot of some of the measurement notes for AE/OE (around-ear/on-ear) headphone measurements​

As you can imagine, these project notes definitely come in handy.

Again, I'll be posting Part 2 as soon as I can, and then regular measurements thereafter. Thanks for your patience, everyone.

Jude, If you need a Sony mdr-R10 for measurement, I can happily lend you mine. Light bass version.

 

[pfzar]

My 2 or 3 cents.

This new 5128 is a tool, just like the other mics, analyzers, and meters we use for tests. We have a lot of new work to do with this new tool to shape our ideas.

The pursuit of knowledge of a topic often leads to having to reevaluate our understanding.

Just because we can measure a difference in a response from different seated positions doesn't mean it matters in the design or our understanding of the product. But it doesn't mean that we should not measure it.

Microphones or simulators are not humans. They are Mechanical devices that TRY and simulate a fraction of our understanding of a HIGHLY COMPLEX hearing system. What one does with data is up to their personal understanding. I welcome this innovation. It will challenge my understanding, and I will challenge it.

 

[playmusic]

The way you are framing the issue, you are dismissing science entirely, not just headphone measurements.

Oh, quite a bold statement.
If you derive this from my post then I better leave you to your opinion.

Some other forum members are maybe interested to look into Bertrand's probability paradox (e.g. at Wikipedia) which shows that taking averages can converge but the result might depend on the method of random selection.

 

[castleofargh]

Oh, quite a bold statement.
If you derive this from my post then I better leave you to your opinion.

Some other forum members are maybe interested to look into Bertrand's probability paradox (e.g. at Wikipedia) which shows that taking averages can converge but the result might depend on the method of random selection.

From the wiki:

...when the domain of possibilities is infinite

Not the case here.

 

[nanaholic]

Oh, quite a bold statement.
If you derive this from my post then I better leave you to your opinion.

Some other forum members are maybe interested to look into Bertrand's probability paradox (e.g. at Wikipedia) which shows that taking averages can converge but the result might depend on the method of random selection.

1) this paradox is about random probabilities - measurements with purpose is not random probabilities.
2) as pointed out above - placing a headphone onto a dummy head is not a domain of infinite random possibilities, defining the methodology of headphone placement to minimise variances (ie the very process of limiting the possibilities) already eliminates this paradox you are quoting

Again, you are over thinking the issue, and by framing the issue as "you can't trust all measurements to be correct, then why measure at all" is EXACTLY dismissing science entirely and getting into the domain of existential philosophy.

 

[pfzar]

To add a little bit more perspective on the measurement of headphones.
An ear simulator is not the only means of measurement of a transducer. Some others looking at what a headphone response is are similar to traditional loudspeaker measurements just scaled to size.

Unloaded near field response taken at 10cm or less.
Suitable drive to reach 1Pa.Log chirp or MLS. Derive impulse.
Loaded near field response taken at the same level and distance.
Derive impulse.

Splice at convergence.

Divide response by target a Diffuse, free field, or Harman or other Hrtf target.
this will make your measurement look like an ear simulation measurement. You also do not need to divide.

this allows you to see the driver's response in different conditions without an ear simulator acoustic load.

Lots of correlation to be done if you go this route, but I have been doing this for many years. But I have the luxury of comparing these methods to hats methods.

IEMs would be something similar. But the loading part would be different.

Edited for grammar from writing on the phone.

 

[Adnen Ayed]

Again reading this thread and posts.
It took me back 24 years ago when I witnessed the measurement of acoustic properties of the three sound processing post production studios of Sony Pictures back in 1996. It led to the development of the current Sony amplifiers Sound Processor. I was working for Sony Audio in Shibaura, Tokyo, then.
I think I'll write about the experience. If anybody is interested, please read it.

 

[Adnen Ayed]

Again reading this thread and posts.
It took me back 24 years ago when I witnessed the measurement of acoustic properties of the three sound processing post production studios of Sony Pictures back in 1996. It led to the development of the current Sony amplifiers Sound Processor. I was working for Sony Audio in Shibaura, Tokyo, then.
I think I'll write about the experience. If anybody is interested, please read it.

Wrote it. Please feel free to read if interested.

https://www.head-fi.org/threads/sonys-measurement-of-acoustic-properties.945398/

 

[IgeNeLL]

Wrote it. Please feel free to read if interested.

https://www.head-fi.org/threads/sonys-measurement-of-acoustic-properties.945398/

I have read but your experience for a different purpose
I'm looking forward to furthering information on this topic.

 

[Adnen Ayed]

I have read but your experience for a different purpose
I'm looking forward to furthering information on this topic.

Please send a message if you require any clarification.

 

[433521]

Let me see a LCD-X So i can feel good knowing its still better than any mr.speakers product.

I listen to my Mr Speakers (now Dan Clark Audio) more than any of my 3 Audeze headphones, combined, including the LCD XC. And it's their cheapest sett they sell..