[Brooko]

@Brooko I certainly never said that whoever had the most $$$$ was the best, that's all you.

I did thank Jude and I mentioned his experience. Are you questioning that?

People have a tendency to believe whatever they see in the form of a graph and whatever comes from an apparatus, regardless of whether that's justified. I do think some humility from those do it yourself rigs toward people who have put more time and experience into it would be great. Jude (and Tyll) deserve respect on this and it's not just more data, but rather better data, that should receive the most attention. That understanding seems to be missing, although you are a step ahead by talking about the limitations of your apparatus. That's not enough, though.

If people with a starter measuring apparatus know that their measurements are limited to below 9khz (and maybe above 70hz, bass on these rigs often isn't quite right), then one way to decrease the garbage factor and gain respect would be to cut your graphs off at those frequencies. Why post information you know to be misleading? That's a real question and one that must be answered. This is a real step you could take to make your data better. We should ask the same of others with the same apparatus. It's not good when twister6, who has the same rig as you, I believe, posts measurements here and then later says he won't post them on his reviews for lack of faith in their accuracy.

And, just to return to topic, I've picked up a pair of ie800S and will be listening for myself as soon as they arrive. It was the comments of people listening to them, primarily, that helped the most in making that decision, plus my experience with the original ie800. Jude's measurements certainly have been helpful as well.

I'll just drop this in here - so we go back to what you said:

Jude is really showing us that there are a lot of garbage measurements around. While I'm sure there is always further to go and he will refine his measurements with experience, already we can see that this is miles ahead of what's been posted.

1. You automatically assumed Judes measurements are correct - and everyone else's are wrong (garbage is the term I believe you used) - this would naturally include Tyll's as well, as his are different.
2. You talk about experience - yet Tyll's graphs are different and he has had years more experience with measuring and refining those measurements. Jude is a comparative newcomer in that regard.

You made a blanket call aimed at hobbyists. I called you out on it. Nothing more nothing less. And yes - sorry - I mentioned the $$$$$ set-up, and thats on me. I'm just disagree with associating Judes set-up automatically with "the truth" and discounting all the other measurements which up to now we hobbyists have used (responsibly) to help us explain what we think we're hearing. They've definitely brought accuracy and objectivity to my own reviews. I provide disclaimers to my measurements - and I'm certainly not going to cut graphs off (if you have a problem with that - then ignore my reviews - in fact ignore most of those on the net including Tylls). People understand its a hobbyist measurement (albeit a calibrated one). On my rig I find it is a great tool for showing comparative frequency responses, and I've spent literally 100's of hours trying to get it as close to IEC711 as I can.

If you look at the graphs posted to date, they aren't that different. Peaks and valleys in the right areas. Its just that one area, and we have people saying they are hearing 10 kHz peaks. That could be down to canal physics - but in that case who is the most accurate?

 

[jude]

@Brooko, you are correct that our headphone measurement rig is not like most others you're seeing measurements from on the web. As I said in the previous post, I think some of the measurement commonalities you'll see between different enthusiast measurements on the web is due to commonly shared opinions and recommendations on how to make and tune do-it-yourself measurement rigs. Also, most of the DIY measurement rigs we've seen make no attempt to simulate the interaction of the headphone/earphone and the human that wears it -- not in terms of external influencing anatomy, nor in terms of attempting to load the headphone or earphone in the same way as the human ear.

Also, in many cases, we have no idea how a web measurement we're looking at was generated (how was position over the mic determined, for example?), what gear was used, what kind of coupler or mic was used, whether or not the system is calibrated, etc. For example, from one of your previous messages:

...Another interesting thing is that there were more measurements showing a 10 kHz peak around the net with that pair of headphones - in fact I think Jude's may have been the only measurement that wasn't. So when something like this comes up, I'd be holding off any judgement.

The person that had the measurement showing the highest level 10 kHz peak with that headphone (the Sony MDR-Z1R) posted this about that measurement:

I made the last measurement. That spike is much bigger than others. I would not necessarily trust mine completely. My measurement rig is a piece of leather and some felt tapped to a broken lamp. But it does seem relatively accurate for most measurements.

That was a candid description of what he used, but, best I could find, it wasn't known until that comment was posted. Because so many are looking to find quick, definitive answers in numbers and lines, then the graphs and specs can have an outsize influence on the perception of a product. While measurements can certainly be useful (otherwise I wouldn't be wasting time and money doing them), I do think it's important to discuss how they're useful, considerations that need to be weighed, their limitations, and simply checking for yourself by listening whenever you can.

...Pretty much all measurements on the internet including Tyll's measurements of various IE800 units show a peak at 10kHz. Now if the 800S follows a similar response, it makes us stop and think when a measurement of the 800S doesn't show a peak there. Even if the measurement is from a industrial level rig. May be Jude's measurement is correct and the IE800S doesn't have the peak. If that is the case, we need to investigate why there are differences instead of discrediting other measurements....if Tyll's measurements of IE800 all show a peak at 10kHz and if the IE800S possibly has the same peak (as stated by some trained listeners), would you start calling Jude's measurement garbage, because his doesn't show a peak there? You can't. We would have to investigate the possible reason for the discrepancy and understand why a certain measurement has the peak or doesn't.

"10 kHz" will forever be a hot topic 'round here! Again, while some may hear a 10 kHz peak of the magnitude shown in some of the IE800S measurements, I do not. There are some differences in our measurement setup here (versus most of the web measurement setups) that may explain some of the differences in some of the measured results. Again, Tyll discussed this briefly in a post on InnerFidelity (and in the video interview with Paul Barton that accompanies the post):

https://www.innerfidelity.com/content/canjam-rmaf2017-people-i-love-paul-barton

Getting back to the reliance on numbers and lines, and a headphone measurement example that's not about frequency response:

In a recent discussion about sensitivity of the new Sennheiser HD660S, there was some concern expressed when our sensitivity numbers varied a bit from Sennheiser's. Questions about the accuracy or integrity of their sensitivity spec arose (despite the fact that their lower sensitivity number actually would be viewed by most as disadvantageous not advantageous). It was clear to me at that point that there was probably not a clear understanding about how sensitivity measurements are done, and how variability is inevitable, affected by many different factors (mic position, earpad conditions, how tight or loose the headphone is, etc.). I never anticipated taking an in-depth look at headphone sensitivity, but, since it came up, we performed sensitivity measurements of the headphone, and then posted about it in some detail. You can read my post about this at the following link:

https://www.head-fi.org/threads/sennheiser-hd650-impressions-thread.197776/page-2669#post-13803816

Again, I think in our pursuit of easy, definitive answers, we sometimes fixate too much on numbers and lines, and finding out that there's usually variability and many other factors to consider -- that you can't just say that line or number is the gospel truth for all -- can be dissatisfying.

Getting back to web headphone measurements, primarily frequency response (and distortion):

Because I did not feel that a lot of the headphone measurements I was seeing on the web reflected what I was hearing (both based on normal subjective music listening impressions, as well as doing frequency sweeps to see if I was hearing the displayed peaks and troughs), we wanted to take a look at what was available -- and what was coming down the pike -- in terms of audio measurement technology, thinking that perhaps we could assemble a headphone measurement setup with the hope of providing measured results that make sense to most people, independent of preference.

Which brings me to this:

In my opinion, then, the best that one can currently do for the kind of headphone measurements we’re seeking as a community is to approximate the headphone on (or in, as they sometimes are) an average human -- how it wears on the human, simulating the anatomy and the working load that the headphone is subject to in situ. Since we can’t reasonably get a precise measurement microphone down a bunch of humans’ canals to the eardrum, the best we can currently do is a fixture or manikin that offers some amount of human dimensional simulation on the outside (and inside, where possible), and to simulate the working load the headphone is faced with when worn.

So...is there a fixture or manikin that can perfectly simulate the average human in all these respects? Do we at Head-Fi have systems that perfectly simulate the average human in all these respects? No. And no. What we are doing is employing the best currently commercially-available technology with that goal in mind. It's an asymptotic goal, perhaps, because it's one we can get closer and closer to, but probably never perfectly reach.

Now -- and this very important -- if hypothetically someone developed a measurement system that perfectly represented the human average for these purposes, what would be the best we could expect from it? The best we could expect from such a system would be measurements that, independent of preference, would make sense to most of those who have heard the headphones that were measured on it. Notice I said "most" -- not "all." I didn’t say “all” because there are always going to be those who deviate enough from the average that even a hypothetically perfectly-executed headphone measurement on the hypothetically perfect human average may not represent what those people (who deviate far enough from the norm) are hearing.

Even if you had this hypothetically perfect human average headphone measurement system, would the measurements from it -- even if on-the-whole they make sense to most -- be representative of every peak and dip for every one of these people? No. Absolutely not, and perhaps especially as you move higher up in frequency, and the wavelengths shorten, where positional and other physical variables can effect increasingly substantial differences. Meaningful headphone measurements above 8 kHz have been a big challenge for decades, and I'll say more on that shortly.

So what do we use in our measurement systems? I've posted a description of our audio measurement systems at the following link:

https://www.head-fi.org/threads/a-headphone-measurement-discussion-that-started-in-the-ie800s-thread.865189/page-3#head-fi_audio_measurement_lab

...Nexdt thing to note - Alex is using my compensation curve for my Veritas on my set-up. It's been calibrated as well as I can to mimic an IEC711 raw measurement, and I already acknowledge that at 9-10 kHz on my rig, my calibration measures lower than it should. Above 10 kHz on my set-up its a crap shoot - the Veritas is pretty inaccurate. However - up to around 8 kHz my measurements on my rig seem to be pretty good. How do I know? Because I had help (from Ken) in getting some accuracy. We both measured the same IEMs (we mailed them back and forth) and I could check and recheck. He also gave me his raw data and I used that to build the curves. Again - from 9 kHz onward I wouldn't call mine accurate - but I have already had 2 manufacturers (other than Ken) tell me that my rig is pretty close to what they are seeing on their own professional rigs at 8 kHz and under...

I've not yet used a Vibro Veritas, and I'm not sure one can simply EQ it into something approximating a 60318-4 simulator, as the 60318-4 (or 60711) ear simulator was designed to mimic the input and transfer impedance of a human ear. It consists of a main volume and side volumes connected to the main volume by thin slits, and is a pretty precise piece of kit. Here are a couple of cut-through views of a standard 60318-4 coupler:

​

I'll order a Vibro Veritas to play with it and compare with it, as it's only $79 with mic.

IMHO, it's much more likely that deviations among listener's ear canal anatomies are the reason behind these discrepancies. Have you ever compared ear impressions for custom IEMs?...

Actually, here's a link to a short discussion about ear canals (and eardrum impedance) that I had with someone at InnerFidelity:

https://www.innerfidelity.com/comment/514162#comment-514162

...The statement about KEMAR having "anatomically correct ear canals" is a bit misleading in my book. Of course they're closer to human ear canals than perfectly cylindrical ones, but (like HRTF compensation targets) they're still based on averaged data, collected from individuals that may vary significantly....

They state openly that their anthropometric pinnae / ear canals are based on 260 three-dimensional scans of human ear canals, with some adjustments made to adapt the ear canal to interface with the ear simulator. These pinnae include the first bend and the second bend of the canals, with "flesh" all the way to the mics. And of course they're averaged.

And, as you stated, they're closer to human ear canals than perfectly cylindrical metal ones. If you've ever tried to put an anatomically angled IEM into a standard measurement pinnae, you'll know it can be challenging, and, with some in-ears, almost impossible. With the anthropometric pinnae, most in-ears (whether straight or anatomically angled) fit into the ear very much like they would with most real human ears. Just from the standpoint of fit, it helps with more realistic placement relative to the "eardrum," as well as more realistic deformation of the eartips upon insert (which is most obvious when you pull foam-tipped IEMs out of them). Like I said earlier in this post, I think that the best one might reasonably do for the purpose is to approximate an average human, and these more realistic pinnae/canal assemblies seem like a step in the right direction that's long overdue.

In a LinkedIn discussion about headphone measurements, where the discussion of testing transducers apart from the headphone came up, Christopher J. Struck of CJS Labs (who is a living library and a wealth of knowledge, especially if you have audio and electroacoustics challenges you need solved) said this:

“All of these points are well taken. Dimensionally, the standard manikin offers median dimensions. In any commercial deign, one must also be concerned with fitting a reasonably wide range of head and ear sizes in order to have a sellable product. With respect to acoustic response, the same can be said: The median ear simulator, pinna and head size in the manikin is intended to provide a response corresponding to the theoretical median listener. This should be in the center of your 'fit range'. If that is not right, everything else is biased accordingly. As Flannigan points out, testing the transducers apart from the system is a separate issue. Ultimately, the transducer performance in-situ is what matters, so any preliminary transducer testing should correlate to the application, i.e., how the transducer will be used in the system or product: e.g., circum-aural, supra-aural, intra-concha, insert, open (low impedance), sealed (high impedance), etc.”

Again, we're doing our reasonable best to find and use the best of what's currently available to provide the most meaningful audio measurements we can. You'll be seeing more posts and videos about this from us, as there is so much to discuss. Yes we've been posting measurements periodically for a couple of years, but, as far as I'm concerned, this is just the start, and with some key developments in the last year or two, and with more coming, it's an exciting time to ramp this up.

 

[Brooko]

Snip for brevity

Jude - personally I welcome anything that can start giving us higher levels of accuracy. An informed potential buyer is always going to do better than someone who is guessing on suitability based on subjective impressions (refer my recent discussions about the Kinera H3).

What I was objecting to is the notion that the graphs provided by the hobbyists should be disregarded as rubbish. As you’re well aware, we can get pretty good readings on frequency response from 30 Hz to around 8 kHz (and some will be even better / go higher).

Personally I put little emphasis on freq response above 10 kHz as once you get into this rarified air, you’d need massive spikes to have any really significant difference with most music.

As I said on the phone to you a couple of years ago - I share your excitement regarding both our ability to measure more accurately and also to promote better understanding of the correlation between perception and measurements.

If you do get the Veritas (I have and continue to use the older model) - my method continues to be:

• Always using same DAC and amp set-up with impedance under 1 ohm
• Always using same ARTA set-up , same settings
• Always use same foam tips - medium bore setting. I use these primarily because with silicone it is impossible to get consistency in the coupler
• Always use approx same tip depth and orientation (and this is as upright/ straight into the couple as I can get)
• Always use it at night in essentially a very quiet room with no fans or other disturbances
• Always measure multiple times until I can get consistently repeatable results
• The calibration was done using exactly the same IEM on a fully calibrated IEC set-up and then applying compensation to equate the two measurements. Like I said earlier, it has been quite gratifying to get feedback from more than one source that my curves are not too far away from one of the industrial rigs being used.

I certainly believe there is a place for the hobbyist, and would encourage those participating to continue what they are doing, and strive for more accuracy over time. Until such time as you start measuring all the gear out there (including a lot of the budget stuff), the hobbyist set-ups are going to be the only way to check between some of the very subjective opinions out there, and a more measured objective approach.

Last point - I don’t generally use my rig for full sized earphones (so the Pinna does really not come into play) - simply because I could never have any degree of accuracy. I do see some affordable alternatives out there - so it’s exciting to think this could also help standardise the measurements of full sized headphones and earbuds
https://www.minidsp.com/products/acoustic-measurement/ears-headphone-jig

 

[jude]

Jude - personally I welcome anything that can start giving us higher levels of accuracy. An informed potential buyer is always going to do better than someone who is guessing on suitability based on subjective impressions...

Actually, given the discrepancies I've seen between web-posted measurements and subjective opinions, I'm apt to trust subjective opinions a lot of the time.

I know "higher levels of accuracy" in reviews has been a rallying cry for you of late, and you believe very strongly that measurements are perhaps the answer. In one of your earlier posts you said this to one of the other community members who writes reviews:

...have you thought about getting some measurement equipment? It will help with the accuracy. the more accurate you are, the better the calls you can make...

Again, given that measurements I've seen on the web can vary about as much as subjective opinions can, the above statement is a recommendation that should be wrapped in caveats. I don't believe reviewers need measurement rigs to review, and I don't always believe that a review should be so strongly influenced by them.

As I said in my previous post, so much of the time we have no idea how the measurements we find on the web were made or what they were made on -- and sometimes finding out only raises more questions (or at least should).

...What I was objecting to is the notion that the graphs provided by the hobbyists should be disregarded as rubbish...

I know you're probably referring to someone else's comments, because I don't feel that to be the case at all. All measurements (including the ones we do here) need to be weighed against the limitations of measurements. And the limitations of measurements vary, and I'll trust some results more than others, depending on who did them, how they were done, what they were done with, etc. Again, for frequency response (for example), even the hypothetical best headphone measurement won't track everyone's every peak and dip, but hopefully will avoid what would be egregious, strange, or even non-existent peaks and dips for most people.

...Personally I put little emphasis on freq response above 10 kHz as once you get into this rarified air, you’d need massive spikes to have any really significant difference with most music...

I feel very differently about this than you do. While I've regarded most headphone measurements above 8 kHz or 10 kHz suspect -- and, again, some more than others -- I feel getting accurate measurements above that should definitely be pursued. It's part of the reason we've chosen the ear simulators we've chosen, and we'll be sharing more measurements from these newer ear simulators over time that we hope have better subjective correlation above 10 kHz than we're seeing most of the time now.

I think most of us here definitely want the engineers and designers working on the headphones we listen to to be concerned with the performance of their products above 10 kHz. And so, in doing headphone measurements as a part of assessing headphones, I very much want to be able to accurately measure headphone performance above 10 khz as best I can.

...As I said on the phone to you a couple of years ago - I share your excitement regarding both our ability to measure more accurately and also to promote better understanding of the correlation between perception and measurements.

And this is still true for me, which I hope is becoming more obvious.

...The calibration was done using exactly the same IEM on a fully calibrated IEC set-up and then applying compensation to equate the two measurements. Like I said earlier, it has been quite gratifying to get feedback from more than one source that my curves are not too far away from one of the industrial rigs being used...

One can't simply EQ the Vibro Veritas to turn it into something that will approximate a 60318-4 simulator from headphone to headphone. If that could be done so simply and so affordably, it would be the industry standard -- it's not for fun or prestige that a company spends considerable sums on measurement gear (including ear simulators, the good ones pretty much always being expensive). The 60318-4 (60711) ear simulator was designed to mimic the input and transfer impedance of a human ear. In other words, its role in the interaction with a headphone or earphone when coupled to the rig is a physical and individual one -- individual in the sense that I do not believe every headphone or earphone will react the same (at every point across the frequency spectrum, not to mention aspects other than frequency response) to the human-simulated input and transfer impedance of the ear simulator. So while you may successfully use an equalizer to force the shape of the Veritas' frequency response output with one headphone to match the output from another measurement rig with that exact same headphone, subsequent measurements of other headphones and earphones from those two rigs would not necessarily match up consistently, if at all.

If you're an AES member, there's an interesting paper from 2004 titled Simulation of the IEC 60711 occluded ear simulator by Søren Jønsson et al. (Downloading the paper is free for AES members, and $33 for non-members.) I believe the purpose of the paper was to examine methods of modeling the 60711 ear simulator so that virtual testing can be done ahead of manufacturing. While it's not entirely analogous to what we're discussing here, I think it gives some idea as to the complexity involved. There may be more relevant papers, more specific to this discussion, but that one came to mind as one of the ones I read as I was first learning about ear simulators.

Again if it could be done that easily and that inexpensively, that's what everyone'd be doing instead of trying to clear budgets to make room for new measurement gear (which is something many engineers I talk to are regularly trying to do). We have a headphone engineer coming to our office in the next week or two in the hopes of trying the new GRAS RA0401 ear simulators. His team already has standard 60711 ear simulators, and if he could simply use an equalizer to turn their 711's into RA0401's, he wouldn't waste his time with the visit -- we'd use the time and money saved to eat more sushi and drink more beer. If we could take our standard 60318-4 ear simulators and EQ them into RA0401's, we'd have done it.

...I certainly believe there is a place for the hobbyist, and would encourage those participating to continue what they are doing, and strive for more accuracy over time. Until such time as you start measuring all the gear out there (including a lot of the budget stuff), the hobbyist set-ups are going to be the only way to check between some of the very subjective opinions out there, and a more measured objective approach...

I agree that it's remarkable, cool, and remarkably cool that people are passionate enough about audio engineering to build their own measurement rigs. I think it's just as important, though, that those trying to interpret the outputs from these systems understand the limitations of them, and even the limitations that would come from a theoretically perfect headphone measurement system.

I don't think the most important point here is that those with lab-grade rigs measure more stuff. (Though I will say that we endeavor to measure as much gear -- electronic and electroacoustic -- as we reasonably can here.) I think the more important point should be that people should not necessarily believe every single graph and number -- every single dip, peak, and ridge they see on the web -- and start asking more questions and understanding how the measurements were made and what they might mean (especially if one is apt to be the type to accept the graphs and numbers as gospel), and, again, definitely to understand the limitations of even the best measurements. That said, for nearly three years, we've been learning from and working with people and companies who have the experience and expertise to help us perform audio measurements using the best available techniques and technologies -- and we'll continue to do so.

I've visited several headphone companies and have seen some impressive facilities with even more impressive knowledge-filled people who work in them. At some of these places, I have watched them do measurements, and, on occasion, even participated in them myself. Even the companies I've visited who use measurements most extensively place great value in the subjective assessments of their products -- that is, they also do extensive, exhaustive listening tests. I haven't met an experienced headphone audio engineer yet who said anything like, "Our measurements of this headphone are fantastic, and so everyone will love them."

For these engineers, obviously measurements do matter. There are companies that go to great lengths and great expense to stay ahead of the curve when it comes to measurement technologies and techniques. I've recently discussed examples with engineers of products gone great and products gone wrong, in part due to both successes and failures relating to measurements. Some time ago, one engineer told me of an ANC headphone product that went to market and was met with very poor reviews, in part because the measurements during the development of that headphone were done on systems that did nothing to simulate the human hearing system -- a mistake he said they're not likely to ever make again. Conversely, another engineer at another company told me very recently of big successes they've enjoyed (also related to ANC headphones) that was in part attributable to improvements they'd made in their measurement labs to help specifically with that product class (some of it involving measurement gear we're intimately familiar with here) as well as extensive listening and use tests.

Anyway, to end this post, here's a fun thing we'll be doing: Soon we'll be posting about a very interesting, very simple experiment we did here with the Sennheiser HD650 that involves measurements, listening, and how different people can have very different experiences listening to the same headphone at the same level, with the same test signal independent of preferences.

 

[Kunlun]

Great post, Jude.

You've expressed both what I want to see from further refinements in measurements, as well as what I am concerned about in terms of hobbyists' understanding the limitations of various measurement rigs.

 

[Brooko]

Actually, given the discrepancies I've seen between web-posted measurements and subjective opinions, I'm apt to trust subjective opinions a lot of the time.
I know "higher levels of accuracy" in reviews has been a rallying cry for you of late, and you believe very strongly that measurements are perhaps the answer.

Depends on the measurements of course - I know that a lot of the Veritas measurements out there are completely uncalibrated. You can see it particularly in the upper mid-range and lower treble. But what i am talking about is when reviewers claim increased bass or increased treble due to things like going balanced - when you can measurably show this does not occur. IMO the standard of reviewing has slipped in the last couple of years - less accuracy, no proper volume matching when making comparisons, talking about perceived "easily heard" differences between two earphones when there are none. You must see it Jude - yet Head-Fi promotes any review to the front page as long as it has a high ratings, nicely laid out and has some nice pictures. In other words rather than trying to set a higher bar, Head-Fi seems to be encouraging the completely subjective and often highly inaccurate reviews that seem to be becoming the norm. I'll trust subjective reviews when I know the reviewer and he/she has good methodology and reasonable accuracy

Again, given that measurements I've seen on the web can vary about as much as subjective opinions can, the above statement is a recommendation that should be wrapped in caveats. I don't believe reviewers need measurement rigs to review, and I don't always believe that a review should be so strongly influenced by them.

A review is and always should be a persons opinion on the headphone they are reviewing - so it will always be subjective. But you and I both know how inaccurate that filter (bias) is that the brain puts on everything. So whats wrong with listen first, take your notes, then measure and use the measurements to check your impressions, and also try to explain them. Surely - as long as you've invested time into trying to get a measurement system to a reasonable level of accuracy - the resultant review should be better? Would we rather not be properly informed, or should we just blindly go into a purchase where a reviewer calls a DAP "U shaped" when it is measurably flat? Or talks about smooth treble when there are obvious peaks? Or talks about differences with going balanced (in terms of frequency changes) when using the same gear you can prove that there is no frequency change?

As I said in my previous post, so much of the time we have no idea how the measurements we find on the web were made or what they were made on -- and sometimes finding out only raises more questions (or at least should).

All measurements (including the ones we do here) need to be weighed against the limitations of measurements. And the limitations of measurements vary, and I'll trust some results more than others, depending on who did them, how they were done, what they were done with, etc. Again, for frequency response (for example), even the hypothetical best headphone measurement won't track everyone's every peak and dip, but hopefully will avoid what would be egregious, strange, or even non-existent peaks and dips for most people.

Agree on all of this which is why some of us who are more serious about this spend so much time seeking a level of accuracy which can make our reviews better. Most of us won't get to anywhere near the level of accuracy as Tyll or yourself (we don't have the budget for it), but its amazing how far the hobbyist sector has come.

I feel very differently about this than you do. While I've regarded most headphone measurements above 8 kHz or 10 kHz suspect -- and, again, some more than others -- I feel getting accurate measurements above that should definitely be pursued. It's part of the reason we've chosen the ear simulators we've chosen, and we'll be sharing more measurements from these newer ear simulators over time that we hope have better subjective correlation above 10 kHz than we're seeing most of the time now.

I think most of us here definitely want the engineers and designers working on the headphones we listen to to be concerned with the performance of their products above 10 kHz. And so, in doing headphone measurements as a part of assessing headphones, I very much want to be able to accurately measure headphone performance above 10 khz as best I can.

My comment was not on accuracy of measuring above 10 kHz (although I agree with you) - it was purely on the audible effect of peaks and valleys (in real music) above 10kHz. By all means strive for accuracy - if I had the equipment I would. But lets not lose the intent of my comment. I've invited more than one reviewer to take normal music, then change an EQ slider in bands from 20 kHz down to around 13 kHz - and see how much actual effect it has. Once again I stand by my comment - for most music and for most people - upper treble frequencies have to have dramatic changes to be perceived or to have noticeable effect.

One can't simply EQ the Vibro Veritas to turn it into something that will approximate a 60318-4 simulator from headphone to headphone. If that could be done so simply and so affordably, it would be the industry standard -- it's not for fun or prestige that a company spends considerable sums on measurement gear (including ear simulators, the good ones pretty much always being expensive). The 60318-4 (60711) ear simulator was designed to mimic the input and transfer impedance of a human ear. In other words, its role in the interaction with a headphone or earphone when coupled to the rig is a physical and individual one -- individual in the sense that I do not believe every headphone or earphone will react the same (at every point across the frequency spectrum, not to mention aspects other than frequency response) to the human-simulated input and transfer impedance of the ear simulator. So while you may successfully use an equalizer to force the shape of the Veritas' frequency response output with one headphone to match the output from another measurement rig with that exact same headphone, subsequent measurements of other headphones and earphones from those two rigs would not necessarily match up consistently, if at all.

If you're an AES member, there's an interesting paper from 2004 titled Simulation of the IEC 60711 occluded ear simulator by Søren Jønsson et al. (Downloading the paper is free for AES members, and $33 for non-members.) I believe the purpose of the paper was to examine methods of modeling the 60711 ear simulator so that virtual testing can be done ahead of manufacturing. While it's not entirely analogous to what we're discussing here, I think it gives some idea as to the complexity involved. There may be more relevant papers, more specific to this discussion, but that one came to mind as one of the ones I read as I was first learning about ear simulators.

Again if it could be done that easily and that inexpensively, that's what everyone'd be doing instead of trying to clear budgets to make room for new measurement gear (which is something many engineers I talk to are regularly trying to do). We have a headphone engineer coming to our office in the next week or two in the hopes of trying the new GRAS RA0401 ear simulators. His team already has standard 60711 ear simulators, and if he could simply use an equalizer to turn their 711's into RA0401's, he wouldn't waste his time with the visit -- we'd use the time and money saved to eat more sushi and drink more beer. If we could take our standard 60318-4 ear simulators and EQ them into RA0401's, we'd have done it.

Remember - I'm talking about earphones - not headphones - so the Pinna has little effect. Our canals are all quite different - so all we can do is get a reasonable average. And I'd suggest to you that within general tolerances for the gear we have - YOU CAN - do exactly what I've done, and get very consistent measurements which are reasonably close to an IEC standard. I've shown it with the gear both Ken and I measured - exactly same earphones (in the end there were about 6 of them). I calibrated based on one (the Nova) and we got very similar curves for all the rest. His will always be more accurate, and I never claim my rig is. But I can consistently get curves which are pretty close to many manufacturers rigs - based on this one reading, and subsequent checking with others. I purposely used a very low impedance system to negate the overall effects of impedance - and the results I'm getting tend to tell me that I'm not too far away.

And lets not throw the baby out with the bath water here. When I use the rig, I show channel matching (which seems to be pretty good as long as I'm seating properly), and base frequency response (which I mainly use to show relativities within the primary audible range - sub-bass to lower treble). I use it to show relativities within the frequency response, and also show relative comparisons to other earphones. Nowhere do I make claims of accuracy above anyone else's measurements. I am very careful with my disclaimers. So I guess I'll just feel free to disagree with your statement. maybe I just "got lucky".

I agree that it's remarkable, cool, and remarkably cool that people are passionate enough about audio engineering to build their own measurement rigs. I think it's just as important, though, that those trying to interpret the outputs from these systems understand the limitations of them, and even the limitations that would come from a theoretically perfect headphone measurement system.

I don't think this was ever in question. Measurements will never tell you how a headphone or earphone will sound to an individual. But they can help explain what you're hearing, and they can give pointers to where issues are.

I don't think the most important point here is that those with lab-grade rigs measure more stuff. (Though I will say that we endeavor to measure as much gear -- electronic and electroacoustic -- as we reasonably can here.) I think the more important point should be that people should not necessarily believe every single graph and number -- every single dip, peak, and ridge they see on the web -- and start asking more questions and understanding how the measurements were made and what they might mean (especially if one is apt to be the type to accept the graphs and numbers as gospel), and, again, definitely to understand the limitations of even the best measurements. That said, for nearly three years, we've been learning from and working with the people and companies who have the experience and expertise to help us perform audio measurements using the best available techniques and technologies -- and we'll continue to do so.

And I hope you continue. In the meantime - perhaps you'll remember that all this discussion kicked off from a single member's post who basically stated (and forgive me, I'm paraphrasing) - "well thank goodness you've measured them Jude - because everyone else's is rubbish".

If you agree with this sentiment (and condone/encourage it) then I'd have to ask why I'm still here? I trust that's not the case. You've always been straight with me - as I have with you.

Ultimately it comes down to pursuit of accuracy. Like I stated in my blog on this site - one of the best compliments you can get as a reviewer is "I trust your opinion xxxxx" or "I trust you xxxxx because you tell it like it is", and this is right up there with someone saying they based a buying decision on your review, and wanted to get in touch to thank you for nailing the description. I believe that my own personal standard and also understanding of audio lifted as I started developing my rig further and understanding how changes of frequency could very much affect my perception of what I was hearing. If you personally feel this is not the case, I'd be interested in talking further - perhaps off-line.

Can I close in suggesting that it might be a good idea if a Moderator prune all the measuring discussion - and maybe re-site it on your blog Jude. Then we could leave the people wanting IE800S discussion in peace

 

[pfzar]

Long time lurking here. Its great to see so many people interested in measurement of audio. I commend Jude and his team on the efforts of exploring this wild world of headphone measurements.

The underlying problem with headphone measurement is that the industry does not have a target shape. We know a loudspeaker in anechoic chamber should measure flat.

We do NOT know what a headphone can and should do. We have microphones that can't simulate the hearing transfer function accurately beyond ~12k. Its an approximation based on standards. The IEC-60318-4 or 711coupler transfer function was based on a dataset of 2 people.

The measurement industry is small and requires extensive capital to do the required R&D to create these new tools. It took 10 or so years to create the new B&K HATS. https://www.bksv.com/en/News/launching-high-frequency-HATS-type-5128

The groups and forums that head-fi and others have created are in fact creating demand of newer technologies.

So thank you all for being so passionate about finding answers.

We are all in pursuit of the same thing. To understand how audio is perceived and measured. We are in search of ways to correlate microphone measurements with human perception.(which is a whole other can of worms.)

Measuring something with microphones, analyzers, or humans are just tools. They give us a graph and we try to map that to an experience.

We are all on the never ending pursuit of knowledge.

 

[castleofargh]

simulating an average ear is very relevant and interesting when trying to set a standard(and we really could do with some new one for headphones and IEMs, as pretty much nobody finds diffuse field compensation to sound neutral). averages are not as interesting or significant when we're concerned about our own ears, because chances are we're not a clone of the perfectly average guy(based on hundreds of human heads). there is no one fit all answer to those stuff, but that doesn't make them irrelevant. they just help answer different questions.
about measurements, while I'm clearly one to think we never have enough, this time I have to side against Brooko. over reliance on some pseudo objective approach is what I tend to think of as "new objectivist"(every year I hope that I'm out of it for good, and the next year I realize that I wasn't ^_^). just because out of 2 approaches, one is rubbish for most purposes outside of personal preferences(sighted tests), doesn't mean we should become over confident about the other one. the repeatability of a measurement can give a false sense that we're full on scientific method. but doing it wrong 500 times in a row with the same IEMs doesn't make it scientific or factual.
the Vibro Veritas is not an IEC711 coupler, that was more or less clear from the start. even if calibrated properly (based on what?), the resonances are not where they should be up to 10khz. so what will work for some IEMs, will not for others.

about the IEM, it's not hard to use a sweep and tell if in our ears we get a significant spike or not. even those who aren't used to sine sweeps and their own equal loudness contour,they can just take an EQ and play around the area of the expected spike with some white or pink noise to see if right before and after, the signal does go down abruptly. or if maybe there is no spike, or a spike but at a different frequency due to insertion, tip, ...
what is the result for this IEM for those who have it? it seems more relevant than to know if the perfectly average dude would get a spike or not. ^_^

 

[thefitz]

Does nobody find it odd that pretty much every single measurement of the IE800 shows a pretty massive 10kHz spike? OK, great - this may or may not be a bad thing depending on taste - but are we really denying it's there? I'd bombard you with proof but I know how sensitive this site is over external links.

On one of these aforementioned external links was a remark that the 10kHz spike "goes away" with very shallow insertion (something about 6cm away from the plane of reference.... whatever that means in human terms), with measurements to back that up. Personally, I still felt it even with the largest tips and as shallow as an insertion as I could get, but perhaps that's just my head geometry.

But this got me to thinking: a stack of measurements show the IE800 with a 10kHz spike. Head-Fi says "nothing to see here". A stack of measurements show the MDR-Z1R with a 10kHz spike. Head-Fi says "nothing to see here".

This kind of reaction brought out a whole lot of "they're rigging the measurements to appease sponsors!!!" but maybe - just maybe - there's some convenient and co-incidental voodoo surrounding 10kHz spike resonances and the geometry of the G.R.A.S. system. Perhaps fitting the G.R.A.S. with earbuds puts them 6mm away from this "plane of reference", naturally (and legitimately) getting rid of this spike. Maybe this geometry applies to all headphones, and cancelled out the Z1R's spike.

This is of course just fun science fiction... but since Schiit's Gadget, why not?

 

[jude]

Does nobody find it odd that pretty much every single measurement of the IE800 shows a pretty massive 10kHz spike? OK, great - this may or may not be a bad thing depending on taste - but are we really denying it's there? I'd bombard you with proof but I know how sensitive this site is over external links.
On one of these aforementioned external links was a remark that the 10kHz spike "goes away" with very shallow insertion (something about 6cm away from the plane of reference.... whatever that means in human terms), with measurements to back that up. Personally, I still felt it even with the largest tips and as shallow as an insertion as I could get, but perhaps that's just my head geometry.

But this got me to thinking: a stack of measurements show the IE800 with a 10kHz spike. Head-Fi says "nothing to see here". A stack of measurements show the MDR-Z1R with a 10kHz spike. Head-Fi says "nothing to see here".

This kind of reaction brought out a whole lot of "they're rigging the measurements to appease sponsors!!!" but maybe - just maybe - there's some convenient and co-incidental voodoo surrounding 10kHz spike resonances and the geometry of the G.R.A.S. system. Perhaps fitting the G.R.A.S. with earbuds puts them 6mm away from this "plane of reference", naturally (and legitimately) getting rid of this spike. Maybe this geometry applies to all headphones, and cancelled out the Z1R's spike.

This is of course just fun science fiction... but since Schiit's Gadget, why not?

Regarding headphone measurements you may have seen on the web, and the fact that many will show measurement characteristics in common: I think some of the measurement commonalities you'll see between different enthusiast measurements on the web is due to commonly shared opinions and recommendations on how to build and tune do-it-yourself measurement rigs. The gear we use to measure headphones here at our office, and the methods we use, are probably quite different than what was used to generate most of the measurements you're seeing on the web from do-it-yourself measurement rigs.

A couple of years ago, I made a post somewhat related to this (the discussion of different headphone measurement setups, and different results), which you can read at the following link:

Headphone Measurements: Different Setups, Different Results

If you search the forums and web you can find some information about, and photos of, the DIY measurement rigs from which many (perhaps even most) of the headphone measurements you've seen online have come from. Some of these can be seen below, with my post continuing after the photos:



As you can see, there are many things in common that these rigs have, probably guided by sharing of instructions and suggestions about how to make one's own rig. For example, to explain why almost all of them have a similarly sized piece of felt or foam around the microphone, we can probably look to this description by Marvey (aka purrin) about this:

...In the middle of the foam layer, a piece of felt about the size of an ear. Without this piece, I found measurements too ringy. Covered completely with felt, I found measurements too damped. The felt the size of an ear also serves another function. Proper measurement of supra-aurals that rest on the ear...

This would suggest that the intent of this piece of felt is to serve as a sort of pinna. I don't think a flat piece of felt or foam cut to about the two-dimensional area size of an ear could reasonably serve this purpose.

Microphone placement for most of these rigs tends to be flush with a flat plate. None of the rigs shown above are using ear simulators, nor are any using pinnae or canal assemblies. Given the length of the microphone that most are using, some of the rigs are quite wide, and that may also have an effect on headband tension and tightness of fit. For most, there is no support underneath the top of the headband. There are other things in common, too, obvious from the photos above.

Again, I think some of the measurement commonalities you'll see between different enthusiast measurements on the web is due to commonly shared opinions and recommendations on how to build and tune do-it-yourself measurement rigs.

---

Since we're talking about differences in measurements from others you've seen on the web, it's probably worthwhile to go over the current measurement systems we're using at Head-Fi's office, as they're quite different than the ones shown above. We've been putting together our measurement systems and techniques for nearly three years now, with a lot of help, knowledge, and feedback from industry mentors that include acoustical engineers and others who make their livings in/around audio measurements. There's always going to be much more learning ahead, no matter how much we do, how much we read, how much we're taught.

Anyway, here's what we're currently using at the office for audio measurements:

GRAS KEMAR, 45CA, Ear Simulators, and Pinnae / Canals​

For headphone measurements, we're working with a head or fixture with more or less fixed dimensions (defined by international standards) representing average human dimensions, and those are the GRAS 45CA and the GRAS 45BB-12 KEMAR manikin.

In the photo (above left), KEMAR is face-forward in our measurement lab, inside a custom Herzan acoustic/vibration isolation enclosure (more information below).​

The pinnae / ear canals we use on the GRAS 45BB-12 KEMAR are anthropometric, based on 260 three-dimensional scans of human ear canals. These pinnae include the first bend and the second bend of the canals, with "flesh" all the way to the mics. Because they're more anatomically representative than traditional measurement pinnae, they have (among other features) a more realistic, more oval-shaped entrance point. Here's a photo of our current GRAS 45CA ears (which currently use standard measurement pinnae):

​

Here are a photo showing the new anthropometric pinna / canal:

​

​

Another advantage of the new pinnae is their increased realism externally. If you've ever felt measurement pinnae, they're typically stiffer than human pinnae and don't readily compress against the head, which is why many who measure headphones often have difficulty measuring supra-aural (on-the-ear) headphones with them. (They can also present problems with measuring shallow-cup circumaurals.) The new pinnae feel and move much more like human pinnae and compress against the head much more like (most) people's pinnae do. Here is a photo I took of the supra-aural Audeze Sine on the standard measurement pinnae on a GRAS 45CA:

​

Here's the same headphone on the new anthropometric pinnae on the GRAS 45BB-12 KEMAR:

​

Where in-ears are concerned, we've found the new pinnae/canals to help tremendously with more realistic and consistent placement, as the pinnae/canals are definitely more human-like now. With this improved realism we've found, for example, that characterizing the differences between eartip types via measurements (relative to our subjective experiences) is improved.

Additionally, I think the dimensional characteristics of the rig we're currently using (a GRAS KEMAR) might also contribute to some of the differences (especially versus the type of DIY rigs shown in the photos above). KEMAR has head shape characteristics that lead to more dimensional limitations on placement -- more human-like limitations, in my opinion. Whereas on a flat plate coupler you can place the headphone in any number of places and still maintain a seal (and thus maintain bass, the loss of which is one of the primary indicators that fit has gone wrong). On humans -- and on KEMAR -- if you go too far back, the curvature of our head can break the seal. Too far down on a human (and KEMAR) and you can also lose seal. In other words, the dimensional limitations of our anatomy -- and the larger the headphone, the more this may come into play -- play a role in guiding and limiting the placement range of the headphone over our ears in actual use. In the frequency ranges we're talking about (as the wavelengths get shorter), minor shifts in placement and dimensions can have substantial effects.

Unlike the DIY rigs shown in the photos that began this post, the measurement manikin (GRAS 45BB-12) and fixture (GRAS 45CA) use ear simulators to simulate the input and transfer impedance of a human ear. The GRAS 43BB ear simulators in this specific KEMAR configuration are quite different than standard 60318-4 simulators. While they still meet the IEC 60318-4 tolerances, the single high-Q resonance above 10 kHz is replaced by two more balanced, more damped resonances. The splitting of the one high-Q resonance into two low-Q resonances may present an advantage in decreasing the uncertainty in the measurements around the resonance (above 10 kHz). Also, the GRAS 43BB is highly sensitive, and very low-noise, and extends the lower dynamic range below the threshold of human hearing. Given its extremely low-noise nature, the 43BB can be used to measure and characterize things like the self-noise of an active headphone (both with and without active noise canceling), which is something we'll be increasingly interested in with the growing prominence of high-fidelity wireless headphones and earphones. It can also help in measuring low-level distortion in headphones and earphones. NOTE: One thing to consider with this low-noise simulator is that it's not suited to very-high-SPL measurements, with an upper limit of the dynamic range to about 110 dBSPL. This hasn't been an issue for us, though, as most of our measurements are set at 90 dBSPL (at 1 kHz).

Here's a whitepaper about the GRAS 43BB Low Noise Ear Simulator

A few weeks ago, GRAS announced still another ear simulator designed specifically for measuring high-resolution headphones. On Friday (two days ago) we took delivery of the new GRAS RA0401 High Resolution Ear Simulators, and we'll be installing them on our GRAS 45CA, along with the new anthropometric pinnae for the GRAS 45CA.

​

This is still another very exciting development for headphone measurements, as obtaining meaningful measurements above 8 kHz with most systems can be enormously challenging. This new GRAS RA0401 High Resolution Ear Simulator also meets the IEC 60318-4 tolerances, but GRAS was able to design it so that its performance from 10 kHz to 20 kHz is substantially improved, that range through which it has a tolerance of +/- 2.2 dB. Here is a graph showing the RA0401's response (including the IEC 60318-4 tolerances) compared to a standard 60318-4 ear simulator:



Again, meaningful headphone measurements above 8 kHz or 10 kHz have been a major pain point for decades, so I think these new GRAS High Resolution Ear Simulators may prove an important development in the world of headphone testing.

Here's a whitepaper about the GRAS High Resolution Ear Simulator

Unfortunately, we haven't had a chance to run our own measurements with the new RA0401 High Resolution Ear Simulators yet, as we shipped our measurement mic preamp power supply back to GRAS for a check-up and any necessary calibration (as it's now nearly three years old, and has been jostled around quite a bit). We should have that (GRAS 12AQ) back in the next few days, and we'll fire up these newest ear simulators just as soon as we do.

To help improve the quality of the measurements, we wanted to maximize environmental isolation. Though we obviously do not have the space (not to mention the budget) to build a full walk-in anechoic chamber, we still wanted to achieve as much acoustic and vibration isolation as reasonably possible. Skylar Gray (formerly of AudioQuest, now with Definitive Technology) recommended we contact Herzan. We worked with Herzan to carefully spec out a custom-built acoustic and vibration isolation enclosure. Our Herzan enclosure has thick walls, made with 11 variable density layers of sound-damping material, and the interior of the enclosure is lined with acoustic sound absorption foam. This enclosure is (by design) fairly massive, weighing around 1200 pounds -- the more mass there is, the more energy it takes to excite the system. The enclosure has two cable ports, both covered with solid machined metal screw-down blocks that are damped, and also have soft gaskets that allow full sealing around the cables. (See photo below.)

The headphone measurement manikin or fixture being used at the time is placed on a Herzan Onyx-6M vibration isolation table, to further help isolate the system from vibrations caused by foot traffic, HVAC systems, vehicle traffic, etc. The Onyx-6M is essentially a steel tabletop supported by pneumatic isolators, and provides isolation beginning at 4.5 Hz.

(Above left) Closed left-side cable port on the Herzan acoustic/vibration isolation enclosure.​

Even with the Herzan enclosure, we still make sure to keep it as quiet as we can in the office while doing measurements. Both of the office HVAC systems are switched off when we measure. You'll frequently hear the tongue-in-cheek cry, "Measurement in 3...2...1...fire in the hole!" before we start a measurement, and everyone remains quiet until an all-clear is given. We're particularly careful about this when using the GRAS 43BB ear simulators because, again, their dynamic range extends below the threshold of human hearing -- so even the very faintest sounds you can hear can be heard by the 43BB's.

The Audio Analyzer: Audio Precision APx555 and APx1701​

At the center of our audio measurements -- whether we're doing electronic measurements (another topic for another time) or headphone measurements -- is an audio analyzer. The audio analyzer generates a stimulus of known characteristics, and then analyzes the response. (Wikipedia has an entry for "audio analyzer" which you can see at the following link for more general information about them: audio analyzer.)

We use the Audio Precision APx555 audio analyzer. In terms of its analog performance, the APx555 has a typical residual THD+N of -120 dB and over 1 MHz bandwidth, which exceeds the analog performance of all other audio analyzers. It will also do FFTs of 1.2 million points and full 24-bit resolution. The Audio Precision APx555 is an incredible tool, whether we're measuring DACs and amps (again, another topic, another time) or doing electro-acoustic tests, which is obviously more relevant to this thread's topic.

For some reading about the importance of low test system noise floor, Dan Foley from Audio Precision (one of my audio measurement mentors for over two years) wrote an article for audioXpress titled "Test and Measurement: 'I Can Hear It. Why Can't I Measure It?'" In this article, Dan uses headphones as a key example, as he discusses the human hearing threshold and how it relates to the noise floor of a sound card interface compared to an audio analyzer's noise floor. You can read this article at the following link:

"Test and Measurement: 'I Can Hear It. Why Can't I Measure It?'"

We've also added all of Audio Precision's Electro-Acoustic Test Options to the APx555 through their Electro-Acoustic Research & Development option (APX-SW-SPK-RD), which is a suite of measurements intended for designers and engineers who develop electro-acoustic audio products. You'll be seeing some measurements specific to this module from us in the near future, along with explanations of them.

Earlier I mentioned testing wireless headphones. Another update we're making to our measurement lab very soon is the addition of Audio Precision's new Bluetooth Duo Module to the Audio Precision APx555 audio analyzer. This new Bluetooth module can act as source and sink for AAC, aptX, aptX-HD, aptX-LL, and SBC. Yes, I said aptX-HD, and yes I'm very excited about that. Last year we had two aptX-HD-enabled wireless headphones in our office. Now we have many more, and it will be exciting to be able to measure these headphones at their wireless best. We'll be covering this module more over time.


In the photo above, the Audio Precision APx555 Audio Analyzer (bottom) and Audio Precision APx1701 Transducer Test Interface.

For amplification (to drive the earphones and headphones), we use the Audio Precision APx1701 Transducer Test Interface. The APx1701 integrates instrument-grade amplifiers and microphone power supplies, but we do not currently use its mic power supplies (we use GRAS's power modules). The APx1701 is made to drive both loudspeakers and headphones, includes current-sense resistors in the amplifier outputs for impedance measurements, has a signal-to-noise ratio of 134 dB, THD+N is ≤ -105 dB, output impedance is 0.13Ω, and it can drive up to 100W per channel into 8Ω. Given the APx1701's fixed gain of 20 dB, we will sometimes use the Rupert Neve Designs RNHP headphone amplifier (set to unity gain) to drive more sensitive headphones and earphones.

Keeping all of the measurement equipment calibrated and maintained is obviously important. We do the calibrations we can do at the office. We use GRAS pistonphones to regularly calibrate the measurement microphones and systems. Ideally, we would do this before every measurement session -- we do it no more than once per month, corrected for temperature and ambient static pressure at the time of the calibration. We run Audio Precision's APx Self Test utility on the APx555 from time to time to confirm proper operation of its analog and digital sections. (We have not yet run that utility on the APx1701, which is still quite new.) We do occasionally measure the Neve RNHP on the APx555 just to confirm the consistency of its performance and to confirm unity gain. We intend to adhere to the manufacturers' maintenance and calibration schedules, and will send the gear to them for check up and calibration accordingly.

We're in the process of shooting Head-Fi TV videos now to discuss the components of the measurement systems in Head-Fi's audio measurement lab, including some examples of them in actual use.

---

Getting back to some of the measurements you're discussing:

Does nobody find it odd that pretty much every single measurement of the IE800 shows a pretty massive 10kHz spike? OK, great - this may or may not be a bad thing depending on taste - but are we really denying it's there? I'd bombard you with proof but I know how sensitive this site is over external links.
On one of these aforementioned external links was a remark that the 10kHz spike "goes away" with very shallow insertion (something about 6cm away from the plane of reference.... whatever that means in human terms), with measurements to back that up. Personally, I still felt it even with the largest tips and as shallow as an insertion as I could get, but perhaps that's just my head geometry.

But this got me to thinking: a stack of measurements show the IE800 with a 10kHz spike. Head-Fi says "nothing to see here". A stack of measurements show the MDR-Z1R with a 10kHz spike. Head-Fi says "nothing to see here".

This kind of reaction brought out a whole lot of "they're rigging the measurements to appease sponsors!!!" but maybe - just maybe - there's some convenient and co-incidental voodoo surrounding 10kHz spike resonances and the geometry of the G.R.A.S. system. Perhaps fitting the G.R.A.S. with earbuds puts them 6mm away from this "plane of reference", naturally (and legitimately) getting rid of this spike. Maybe this geometry applies to all headphones, and cancelled out the Z1R's spike.

This is of course just fun science fiction... but since Schiit's Gadget, why not?

Looking back at some of the IE800S measurements that were shared by others in this thread:

The first two are perhaps characterized more by sizable scoops out of the frequency response moving from 1 kHz to around 6 kHz, with the first one being more severe. Coming out of this, the rise to 10 kHz looks more pronounced. We've gone over this already, but let's do it again. If you listen to a frequency sweep through that range, are you hearing a scoop of that magnitude? Again, some may, I think most will not. On the third graph, the rise to 10 kHz is about 12 or 13 dBFS above the average level leading to it. Again, sweep through this range. Do you hear it? Some may, I think most will not. (I do not.)

As for the Sony MDR-Z1R, here are some recent quotes from Tyll related to that discussion (comments that may also apply to the above):

You may remember the Sony MDR-Z1R measurement discrepancy between Jude's rig and mine. You'll recall the biggest discrepancy was at around 10kHz. Jude uses the G.R.A.S. KEMAR head and torso simulator with anthropometric pinnae that includes anatomically correct ear canals; the ear canals in my head are perfectly cylindrical. I've been thinking about it quite a bit, and I've got to think the anatomically correct ear canals have lower-Q (more damped) resonances than mine. We may not be used to seeing the curves as presented on Jude's rig, but they may actually be more representative of what we hear.

In a later post there he said:

Many readers will be familiar with the controversy regarding the Sony MDR-Z1R and the disparity between my measurements and Jude's, much of it surrounding the significant difference of the measured peak at around 10kHz. I think much of the curfuffle can be written off with the understanding that measurements from differing systems really can't be compared. I continue to believe these headphones have too much energy in that area, but if you look at all the InnerFidelity measurements you can see a clear tendency for my head to show a peak in that area...and I think that's a measurement accuracy error.

Is that the answer? I think it may be part of it, some of the details of which were discussed in the first half of this long post.

Simply put, I think it's primarily the differences in measurement systems that generate the different results (and the common results among like DIY systems), some of the details of which we've gone over in this post.

Again, a couple of years ago, I made a post somewhat related to this (the discussion of different headphone measurement setups, and different results), which you can read at this link.

 

[Rob80b]

Thanks Jude...
Some may have missed the point but from my understanding if the methods for measuring have not taken in to compensate for the irregularities created by the ear canal, which do alter the final signal arriving at the ear drum,…especially the higher frequencies, the rise in the upper frequencies which some have recorded may in fact have been purposely induced by Sennheiser (we may never know) through their R&D to create a flat response relative to the original signal arriving at the ear drum by the “average” listener.

Bottom line: I don’t believe anyone is disputing hearing capabilities.
I’m not an acoustic engineer but was involved in electronic music production and recording for about 30 years and what I believe is being questioned is the validity of 10khz of anything being easily perceived. At that frequency we should be dealing strictly with “greatly” diminished overtones volume wise, with traditional instruments, non electronic, the highest fundamentals which are very rarely used in any music is around 4 kHz or the highest note on the piccolo or pipe organ.

Cymbals are usually the first culprits as they do create massive amounts of overtones, and are difficult to capture, especially “digitally” ….compounded by mediocre dacs by the end listener and you’ve got an irritating noise which most higher revealing headphones will be more than happy to reveal

 

[Rob80b]

….one important point not currently discussed is the consistency of manufacturing and driver matching otherwise measurements are moot and become a fairly contentious issue especially with TOTL headphones.

Believe it is generally agreed Sennheiser is quite consistent and JFYI the ie800 and most likely the ie800S are not an exception as shown in passing here at the 3:30 mark of this factory tour, note the hand written notice. : )

 

[jude]

Thanks Jude...
Some may have missed the point but from my understanding if the methods for measuring have not taken in to compensate for the irregularities created by the ear canal, which do alter the final signal arriving at the ear drum,…especially the higher frequencies, the rise in the upper frequencies which some have recorded may in fact have been purposely induced by Sennheiser (we may never know) through their R&D to create a flat response relative to the original signal arriving at the ear drum by the “average” listener...

@Rob80b, I haven't measured the IE800S outside of KEMAR yet. Perhaps I'll ask someone at Sennheiser.

Regarding the ear canal with respect to what we're hearing. Someone (in the comments area) at InnerFidelity posted an interesting comment that speaks to a couple of key questions. First of all, why are high-frequency measurements all over the place? Secondly, does the answer require measuring beyond the eardrum? He posted this:

The eardrum itself with the tympanic cavity and bones behind it should significantly affect the overall transfer function of the ear at the tympanic window where motion enters the cochlea. That is the only location that really means anything when considering measurement and it is rather difficult to get to. For this reason I consider all current methods and devices to measure 'phone response to be largely invalid.

Such incomplete plots have and can have no absolute value nor can they be interpreted in any way other than to compare the plot of one phone relative to another phone (convolutionally divide the impulse responses.) The ratio can be meaningful even if the actual plot isn't.

The region above 8k responds to the most minor factors. Simply taking the 'phones off and putting them back on radically effects the measurements I take right at my eardrum with a .1" Knowles mic. I have found the repeatability of such measurements to be most elusive.

To me this was a very interesting comment, as I'd wondered the same thing in the past. That is, might we eventually find out that measuring at the eardrum is not sufficient? Last year at the 2016 AES International Conference on Headphone Technology in Aalborg Denmark (which Tyll and I both attended), someone actually addressed this question (among other things), and also brought the discussion of ear canals into his talk in some interesting detail. I thought it would be of interest to DonGately, so I responded:

At the 2016 AES International Conference on Headphone Technology in Aalborg, Denmark (which Tyll also attended), Andrew Bright (then of Goertek, now the Director of Audio Systems Engineering at Apple) gave a very interesting talk called "Headphones, known-knowns and unknown-unknowns," and, among other things, he discussed this.

In his summary he said, "For me, when I undertook this, I was kind of expecting to conclude that we should try to measure something in the cochlea, but I was surprised that measuring the eardrum pressure seems to be sufficient. This was a surprise to me."

Among other things during the talk, he cited papers by Hudde et al. titled "A Full-Size Physical model of the Human Middle Ear" and "Measuring and Modeling Basic Properties of the Human Middle Ear and Ear Canal. Part III: Eardrum Impedances, Transfer Functions and Model Calculations," in which Bright was discussing the question of whether or not we should be measuring something more directly analogous to pressure into the cochlea, and that maybe doing so would give us something more consistent when measuring at high frequency. From these and other studies he concluded that the answer is that it's "probably not necessary."

Among the findings from Hudde et al., Bright said "at high frequencies (>3 kHz) the eardrum presents a largely reflective surface," and that "individual characteristics of the eardrum, middle, and inner are unlikely causes of individual differences in headphone response." He further said, "The transfer function from eardrum pressure to vestible (cochlea) pressure, while not flat, does not exhibit the wide variations seen in person-to-person measurements. This is an unlikely cause of individual differences in headphone response. It is thus likely that pressure at the eardrum is a good measure of hearing."

So, in his summary, he stated that eardrum pressure is likely a good measure of hearing. He also said that differences in a headphone's response between individuals are caused primarily by variations in ear canal length and area profile (area vs. length), which he did go into more detail about during his talk.

I'm abbreviating a very compelling hour-long talk by Bright, and perhaps not well enough. Given your post, though, I thought you'd find this part interesting.

DonGately did respond, and you can see his response in the comments area of the post at this link (note that comments are not visible in InnerFidelity's mobile view, so you'll have to be in desktop view to see this discussion in the comments).

In a sea of interesting talks at the conference, that one stuck with me the most for perhaps obvious reasons. It was encouraging to hear his findings (and how he arrived at them with the help of others' studies) that "It it likely that presure at the eardrum is a good measure of hearing." If we can feel reasonably comfortable with that conclusion, then we can perhaps move on to the other variables that affect individual differences in headphone response, and he did that, too, by further examining the discussion of the ear canal's role.

Like pfzar said earlier, "We are all on the never ending pursuit of knowledge."

 

[Brooko]

You've absolutely nailed a lot of the issues Jude. And especially the higher frequency issues. I'd be willing to bet that even on your much more sophisticated equipment, changes in angle and insertion depth (with IEMs) will significantly change 10 kHz+, and more-so than other frequencies. Its one of the hardest things to get right on a hobbyist rig. Interesting to hear that engineers have similar issues, and acknowledge that what makes us unique will also make the sonic experience unique.

If you do get any time, and its not too much trouble - would you be able to do this (change angle and insertion depth) and show the changes graphically (multiple curves on a single graph) because it would highlight some of the issues with making blanket assertions about subjective impressions of an IEM. Likewise the differences with tips (BTW do you use one standard tip for your measurements?)

I know one of the learning experiences for me was with the newer qJays where I couldn't understand why people were calling them sibilant, and others weren't (I heard no sibilance). It was only afterwards when it was explained to me that for such a small IEM, insertion depth, angle, and canal physiology could have significant effects on the higher frequencies that I had that "aha" moment. Until then I'd been a bit of a prat about it - afterwards was able to appreciate both sides, and completely modify my stance from that time forward. One of the benefits of some of the expertise on the forum. You're always learning, and slowly modifying your thoughts and understanding with reviewing. An enlightening and often humbling journey.

 

[piotrus-g]

@jude what's your take on Hi-Res thing and do you think 40kHz extension matter for IEMs at all?

I know this might be even more controversial than discussing measuring rigs or 10kHz accuracy. I'm just wondering what is your view.