Cut the blind testing crap  

With some luck I'll have dragged a few of the more "sciency" head-fi'ers in with the title for this post.

I want to propose a project to the more scientifically inclined crowd on head-fi, the "absolute fidelity index" project, as I'm calling it for now. I'm kind of frustrated that when I'm looking for a new pair of headphones, I need to read highly subjective reviews on what various people think of the sound quality of the headphones, in order to determine which one is probably the best at a given price range. This shouldn't need to be the case, there are objective ways to measure the absolute fidelity of a piece of audio equipment, there is nothing in sound that can't be measured. What I'm proposing, as most of you have probably already guessed, is that we objectively measure, with a percentage, how close the electrical impulses or sound waves produced by a piece of audio gear are. I'd like a few people to respond agreeing that they'd be interested in participating before I type out the full description of how we'd be doing this. Anybody have some good meters and the time to figure out how close the sound reproduced by a pair of cans replicates the electrical impulses from an amp?

All I'm proposing is a series of recordings played back at very hit bit rates / bit qualities to headphones on a binaural dummy head. Get both the DAC and the ADC to work using the same clock. Play back the 384 / 24 track over the headphones, then record it with the very carefully calibrated mics in the dummy head. Apply a reverse of the mics equalization curve to the recording that you produced, and then run it through a program to measure the difference between each "bar" of the new recording against the old one. Then, add up the values of all of the differences, and then divide that by the total of all of the values in the test track to find the percentage fidelity for the headphones. Shift the headphones around on the head, and repeat the process two to three dozen times. Then average the percentages, and give out a low percentage, a high percentage, a mean percentage, and a median percentage. This should account for varying placement of the headphones, as well as different head topography to get  a "typical" result.

You mention soundstage, those sonic cues come from the music itself, not the headphone. The ability to reproduce these cues is based entirely on the headphone ability to faithfully reproduce anything. As for the diffuse field equalization, then apply the reverse of that curve the recording as well.

Quote:

Originally Posted by poikkeus 

 

 

Your testing methodology is promising, but your statement on soundstage doesn't fit in with my experience with a pretty wide range of phones. Soundstage relates directly to the distance your ears from the headphone diaphragm. That's why it's just about impossible to get soundstage when you're wearing a pair of IEMs. I've also noticed increased headspace while using some open headphone; the excess sound leaks from your phones, giving the impression of space.

I don't think the leakage is the issue, so much as the minimized reverb inside of the housing, anyway, that would be accounted for as any echos would be picked up on the mic and would severely impact percentage fidelity. And soundstage isn't about the distance between your ears and the diaphragm, its about a ton of very complex sonic cues that should not be added in by the headphones. Thats all there is to it, a headphone should not try and apply HRTFs to make the sound seem more distant, this is just very, very bad practice when we're talking about reproducing the sound as it was recorded. You might be talking about a sort of natural "cross feed" which could very easily be measured by playing sound through only one side at a time and taking measurements from the other. Anyway, again, I'm concerned about fidelity, not soundstage. If someone else would pitch a good way to objectively measure soundstage, I already shot tyll an email about this asking for his opinion, then once the details are rounded out I'll start looking around to see who has the gear / how much would it cost to start this project.

The headphone graph is compared with an original graph made with just the amp alone. You could zero the test using the most distorted piece of crap amp you could hope to find from 1980 if it could produce all of the frequencies in the test sweep. The idea isn't to tell you exactly what a headphone will sound like, just how accurately the headphone reproduces the signal fed to it by an amplifier. I don't see how the impedance curves on an amplifier will change how accurately the headphone reproduces the signal that is being thrown at it. If you could explain that, it would be greatly appreciated.

Quote:

Originally Posted by tvrboy 

Truly flat headphones (audiometric) do not give a perceived flat frequency response to the listener. Try a Beyerdynamic DT-48. It's the flattest frequency response I've ever seen. But when listening, it has some problems including seriously rolled off bass and treble. There are also things like tonality that we haven't found good ways to measure yet. A transducer can have perfect frequency response yet have timbre totally unlike a real instrument. I think measurements are necessary, but with the technology we have now it's impossible to "listen" to a transducer with measurements alone.

 

1. That is a consequence of head acoustics. That first graph above was likely done with just a regular spl meter held up to the cups. The latter graph specifically identifies that it was adjusted for head acoustics. Neither of these situations are desirable, as they involve too much assumption and compensation. The test I proposed uses a dummy head like headroom does to account for head acoustics. 

2. I'm proposing this test as a way to measure tonality. Admittedly, we can't measure exactly how the cans will sound with this test. I'm sure if someone wanted to, they could create a complex testing method that solves this issue, I just want to know how accurately a pair of cans reproduces the signal fed into them by an amp. If the cans score. say, a 98% on this test, instruments are going to sound **** realistic. A 98% means that you get pretty much flawless reproduction of the sound, and if your reproduction is "near flawless" the instruments should sound exactly as they were recorded. The only reason the instruments wouldn't sound realistic if the headphones produced every "bar" the amp fed them without error is because of a lack of a decent soundstage, the fact that you can't tangibly feel the sound with headphones, and the distortion created by the recording equipment / your dac and amp.

"If you do a simple null to determine how similar the signals are, you're not going to be taking into account head acoustics. Every headphone is colored, so the ones that score best would be flattest. The question isn't which headphone is flattest, the question is which headphone most closely matches the ideal equalization to simulate instruments a certain distance away. Most songs are mixed on speakers X feet away, and will sound realistic from X feet away. So doesn't a headphone needs to produce the signal and account for the effects of the outer ear and attenuation caused by the air between the head and ideal listening distance?"

Thank you, this is an excellent point. The test would need to be done using a dummy head and reference studio monitors positioned exactly where the engineer usually puts them (which can vary from studio to studio) as the reference point to compare with the headphones. This is starting to get ridiculous, the testing method just has to compensate for too many variables to be truly effective. 

"P.S. As far as I know, Tyll at Inner Fidelity does use a dummy head. After all, he's the reason Headroom has graphs of their own. Those bottom curves are what the dummy perceives. The upper curve is the only thing that's compensated."

Ah, I saw "compensated" on the graph and assumed they ran a computer simulation. I guess not, thanks for clearing that up.

Quote:

Originally Posted by Head Injury 

You wouldn't have to account for every possible speaker position. No one does that with their speakers anyway, and I think engineers know that. The point is, you have to compensate for head acoustics and distance somehow. Even if a headphone measured perfectly flat in either the uncompensated or compensated graphs of Inner Fidelity, that still wouldn't be "neutral". Speakers are EQed flat because they're a certain distance away, not just because of head acoustics. After head acoustics, if a headphone measured flat, it would still sound like the musicians are in your ear.

 

It depends on what your reference for neutrality is. True flat response? Perceived flat response? Speakers in an imperfect room? Or musicians in an acoustically neutral room?

I was going to compensate for head acoustics from the headphones with a dummy head. The distance, I presume, could be compensated for by applying an ideal diffuse field equalization curve to the control recording? This way when the original recording is compared with the headphones, the "better" the diffuse field equalization on the headphones, the close they would come to the control recording? My reference for neutrality is the exact sound the engineer intended for listeners to get. I was thinking that the only impact the use of speakers would have on the sound would be limited to a larger soundstage without any notable effects on the timbre of the instruments, but I'm probably wrong.