Jude - personally I welcome anything that can start giving us higher levels of accuracy. An informed potential buyer is always going to do better than someone who is guessing on suitability based on subjective impressions...
Actually, given the discrepancies I've seen between web-posted measurements and subjective opinions, I'm apt to trust subjective opinions a lot of the time.
I know "higher levels of accuracy" in reviews has been a rallying cry for you of late, and you believe very strongly that measurements are perhaps the answer. In one of your earlier posts you said this to one of the other community members who writes reviews:
Brooko said:
...have you thought about getting some measurement equipment? It will help with the accuracy. the more accurate you are, the better the calls you can make...
Again, given that measurements I've seen on the web can vary about as much as subjective opinions can, the above statement is a recommendation that should be wrapped in caveats. I don't believe reviewers need measurement rigs to review, and I don't always believe that a review should be so strongly influenced by them.
As I said in my previous post, so much of the time we have no idea how the measurements we find on the web were made or what they were made on -- and sometimes finding out only raises more questions (or at least should).
...What I was objecting to is the notion that the graphs provided by the hobbyists should be disregarded as rubbish...
I know you're probably referring to someone else's comments, because I don't feel that to be the case at all. All measurements (including the ones we do here) need to be weighed against the limitations of measurements. And the limitations of measurements vary, and I'll trust some results more than others, depending on who did them, how they were done, what they were done with, etc. Again, for frequency response (for example), even the hypothetical best headphone measurement won't track everyone's every peak and dip, but hopefully will avoid what would be egregious, strange, or even non-existent peaks and dips for most people.
...Personally I put little emphasis on freq response above 10 kHz as once you get into this rarified air, you’d need massive spikes to have any really significant difference with most music...
I feel very differently about this than you do. While I've regarded most headphone measurements above 8 kHz or 10 kHz suspect -- and, again, some more than others -- I feel getting accurate measurements above that should
definitely be pursued. It's part of the reason we've chosen the ear simulators we've chosen, and we'll be sharing more measurements from these newer ear simulators over time that we hope have better subjective correlation above 10 kHz than we're seeing most of the time now.
I think most of us here
definitely want the engineers and designers working on the headphones we listen to to be concerned with the performance of their products above 10 kHz. And so, in doing headphone measurements as a part of assessing headphones, I very much want to be able to accurately measure headphone performance above 10 khz as best I can.
...As I said on the phone to you a couple of years ago - I share your excitement regarding both our ability to measure more accurately and also to promote better understanding of the correlation between perception and measurements.
And this is still true for me, which I hope is becoming more obvious.
...The calibration was done using exactly the same IEM on a fully calibrated IEC set-up and then applying compensation to equate the two measurements. Like I said earlier, it has been quite gratifying to get feedback from more than one source that my curves are not too far away from one of the industrial rigs being used...
One can't simply EQ the Vibro Veritas to turn it into something that will approximate a 60318-4 simulator from headphone to headphone. If that could be done so simply and so affordably, it would be the industry standard -- it's not for fun or prestige that a company spends considerable sums on measurement gear (including ear simulators, the good ones pretty much always being expensive). The 60318-4 (60711) ear simulator was designed to mimic the input and transfer impedance of a human ear. In other words, its role in the interaction with a headphone or earphone when coupled to the rig is a physical and individual one -- individual in the sense that I do not believe every headphone or earphone will react the same (at every point across the frequency spectrum, not to mention aspects other than frequency response) to the human-simulated input and transfer impedance of the ear simulator. So while you may successfully use an equalizer to force the shape of the Veritas' frequency response output with one headphone to match the output from another measurement rig with that exact same headphone, subsequent measurements of other headphones and earphones from those two rigs would not necessarily match up consistently, if at all.
If you're an AES member, there's an interesting paper from 2004 titled
Simulation of the IEC 60711 occluded ear simulator by Søren Jønsson et al. (Downloading the paper is free for AES members, and $33 for non-members.) I believe the purpose of the paper was to examine methods of modeling the 60711 ear simulator so that virtual testing can be done ahead of manufacturing. While it's not entirely analogous to what we're discussing here, I think it gives some idea as to the complexity involved. There may be more relevant papers, more specific to this discussion, but that one came to mind as one of the ones I read as I was first learning about ear simulators.
Again if it could be done that easily and that inexpensively, that's what everyone'd be doing instead of trying to clear budgets to make room for new measurement gear (which is something many engineers I talk to are regularly trying to do). We have a headphone engineer coming to our office in the next week or two in the hopes of trying the new GRAS RA0401 ear simulators. His team already has standard 60711 ear simulators, and if he could simply use an equalizer to turn their 711's into RA0401's, he wouldn't waste his time with the visit -- we'd use the time and money saved to eat more sushi and drink more beer. If we could take our standard 60318-4 ear simulators and EQ them into RA0401's, we'd have done it.
...I certainly believe there is a place for the hobbyist, and would encourage those participating to continue what they are doing, and strive for more accuracy over time. Until such time as you start measuring all the gear out there (including a lot of the budget stuff), the hobbyist set-ups are going to be the only way to check between some of the very subjective opinions out there, and a more measured objective approach...
I agree that it's remarkable, cool, and remarkably cool that people are passionate enough about audio engineering to build their own measurement rigs. I think it's just as important, though, that those trying to interpret the outputs from these systems understand the limitations of them, and even the limitations that would come from a theoretically perfect headphone measurement system.
I don't think the most important point here is that those with lab-grade rigs measure more stuff. (Though I will say that we endeavor to measure as much gear -- electronic and electroacoustic -- as we reasonably can here.) I think the more important point should be that people should not necessarily believe every single graph and number -- every single dip, peak, and ridge they see on the web -- and start asking more questions and understanding how the measurements were made and what they might mean (especially if one is apt to be the type to accept the graphs and numbers as gospel), and, again, definitely to understand the limitations of even the best measurements. That said, for nearly three years, we've been learning from and working with people and companies who have the experience and expertise to help us perform audio measurements using the best available techniques and technologies -- and we'll continue to do so.
I've visited several headphone companies and have seen some impressive facilities with even more impressive knowledge-filled people who work in them. At some of these places, I have watched them do measurements, and, on occasion, even participated in them myself. Even the companies I've visited who use measurements most extensively place great value in the subjective assessments of their products -- that is, they also do extensive, exhaustive
listening tests. I haven't met an experienced headphone audio engineer yet who said anything like, "Our measurements of this headphone are fantastic, and so everyone will love them."
For these engineers, obviously measurements
do matter. There are companies that go to great lengths and great expense to stay ahead of the curve when it comes to measurement technologies and techniques. I've recently discussed examples with engineers of products gone great and products gone wrong, in part due to both successes and failures relating to measurements. Some time ago, one engineer told me of an ANC headphone product that went to market and was met with very poor reviews, in part because the measurements during the development of that headphone were done on systems that did nothing to simulate the human hearing system -- a mistake he said they're not likely to ever make again. Conversely, another engineer at another company told me very recently of big successes they've enjoyed (also related to ANC headphones) that was in part attributable to improvements they'd made in their measurement labs to help specifically with that product class (some of it involving measurement gear we're intimately familiar with here) as well as extensive listening and use tests.
Anyway, to end this post, here's a fun thing we'll be doing: Soon we'll be posting about a very interesting, very simple experiment we did here with the Sennheiser HD650 that involves measurements, listening, and how different people can have
very different experiences listening to the same headphone at the same level, with the same test signal
independent of preferences.