[playmusic]

I am looking for some protocol that reduces my unconscious bias when I compare headphones in a quiet hifi shop.

I guess that level matching is the first step, but already this looks challenging:
Matching at what frequency? The choice of frequency has an influence - unless the two frequency curves are parallel.
Or matching by white noise?

Assume that the sensitivities (dBSPL) of the headphones at a rated frequency and voltage rms (say 1 kHz, 1 Vrms) are specified, and that the impedances of the headphones at that frequency and the output impedances of the amplifiers are known.
Then some calculations in combination with a multimeter to measure the unloaded voltage of the amplifier output can match the SPLs with a certain accuracy.
But even with this effort, I guess that an accuracy of ca. 1 dBSPL would be the best attainable by this method because of measurement errors of sensitivities, sealing issues with the headphones on my head etc.

So, a pragmatic approach is to have some sine tone and switch between headphones and adjust the volumes until the match - more or less.
Since hifi shops provide streaming as source material, e.g. Tidal, I still have to find a song that has a continuous sine tone.

Double blind testing will not really work because of the different pads, weights, pressures, headbands, etc. of the headphones.

I had listened to a Stax SR-009s in a shop and the following happened:
At first, I was underwhelmed. Only later when I 'got used' to the Stax and listened more carefully, I found that I hear a lot of details with the 009s that I hear with significantly less detail in, say, a Sennheiser HD 800s, e.g. how a hi-hat sounds or to be able to "zoom in" to single instruments in a symphony.
I now wonder whether I might hear the same level of detail when I learn to listen more carefully to a HD 800s.

Those headphones are at a significant price level (let alone the amplifiers for Stax).
Any guidance from you how to eliminate potential bias on my side (or to learn to hear more details with any headphone) would be highly appreciated.

 

[bigshot]

I don't think you can conduct a controlled test in a shop. But it doesn't matter because it is almost impossible to blind compare headphones properly anyway. You can feel differences, and it takes too long a time to switch from one to the other.

Here is how I do it. First, I chase away the salesman and handle the playback equipment myself. I've caught salesmen changing settings behind my back to try to skew the choice to the model he wants me to buy. Second I bring several CDs with different kinds of music that I am familiar with and methodically listen to all three on each headphone. Lastly, I record my thoughts on a yellow pad as I am listening. When I'm done, I eliminate the obvious non-starters and do a run off of the top three taking comfort and price into consideration as well as sound.

 

[castleofargh]

I am looking for some protocol that reduces my unconscious bias when I compare headphones in a quiet hifi shop.

I guess that level matching is the first step, but already this looks challenging:
Matching at what frequency? The choice of frequency has an influence - unless the two frequency curves are parallel.
Or matching by white noise?

Assume that the sensitivities (dBSPL) of the headphones at a rated frequency and voltage rms (say 1 kHz, 1 Vrms) are specified, and that the impedances of the headphones at that frequency and the output impedances of the amplifiers are known.
Then some calculations in combination with a multimeter to measure the unloaded voltage of the amplifier output can match the SPLs with a certain accuracy.
But even with this effort, I guess that an accuracy of ca. 1 dBSPL would be the best attainable by this method because of measurement errors of sensitivities, sealing issues with the headphones on my head etc.

So, a pragmatic approach is to have some sine tone and switch between headphones and adjust the volumes until the match - more or less.
Since hifi shops provide streaming as source material, e.g. Tidal, I still have to find a song that has a continuous sine tone.

Double blind testing will not really work because of the different pads, weights, pressures, headbands, etc. of the headphones.

I had listened to a Stax SR-009s in a shop and the following happened:
At first, I was underwhelmed. Only later when I 'got used' to the Stax and listened more carefully, I found that I hear a lot of details with the 009s that I hear with significantly less detail in, say, a Sennheiser HD 800s, e.g. how a hi-hat sounds or to be able to "zoom in" to single instruments in a symphony.
I now wonder whether I might hear the same level of detail when I learn to listen more carefully to a HD 800s.

Those headphones are at a significant price level (let alone the amplifiers for Stax).
Any guidance from you how to eliminate potential bias on my side (or to learn to hear more details with any headphone) would be highly appreciated.

Short answer: It's really hard. we're all doomed! Give up.

Your method of trying to match by ear will always limit the accuracy of the matching. With a sine at 2 or 3kHz I could get fairly consistently within 0.5dB(so +/-0.25dB). But even that isn't considered good for a listening test, because such levels are definitely enough to influence our impressions.
A mic(ideally some binaural mics in your ears) and some test tones would be better, but bring up other issues you're usually not going to to handle at a store.
And then of course as you pointed out, we're trying to match 2 headphones with different frequency responses. So a good match at one frequency, is a bad match somewhere else.

I don't remember if/how the matching of headphones in term of listening level is explained in some of the Harman papers in their massive headphone preference studies. "The relationship between perception and measurement of headphone sound quality" perhaps has such a protocol explained?
I couldn't find it online just now with my google-fu, but maybe someone else will have more skillzzz(or was clever enough to save it when it was available for free at AES). Or if @Tonmeister2008 happens to come around(don't hold your breath), he would answer better than I ever could.

 

[bigshot]

I wouldn't say give up... I'd make an effort to match volumes by ear, but I wouldn't worry about getting it to lab standards, just close enough for government work. Whenever a "standard" is set in home audio, it always gets its goalposts moved several times "just to be safe". 1dB used to be the audible difference, then it went down an order of magnitude to .5dB once audiophiles started stacking the deck to find out what ABSOLUTELY WITHOUT ANY EXCEPTION is the threshold. The truth is, for the purposes of listening to music in your living room, tolerances aren't as strict as listening to test tones in a lab. Just be aware that in certain circumstances, it could affect what you are perceiving. As long as you are doing it to answer questions for yourself, good enough is fine.

The big problem isn't volume matching... it's that you are going to try to make an informed decision in enemy territory. I don't know the high fi shop you are going to, but I can tell you that when I tried to do an organized comparison of speakers, the salesman deliberately adjusted the bass control behind his back where I couldn't see to skew my results. I caught him at it and told him that if he was going to monkey with the settings, it would take me all night to figure out what I was going to buy. He marched off in a huff and didn't speak to me again, which was fine with me. I decided which speakers I wanted... and went to another store to buy them!

 

[dazzerfong]

The big problem isn't volume matching... it's that you are going to try to make an informed decision in enemy territory. I don't know the high fi shop you are going to, but I can tell you that when I tried to do an organized comparison of speakers, the salesman deliberately adjusted the bass control behind his back where I couldn't see to skew my results. I caught him at it and told him that if he was going to monkey with the settings, it would take me all night to figure out what I was going to buy. He marched off in a huff and didn't speak to me again, which was fine with me. I decided which speakers I wanted... and went to another store to buy them!

Not really a problem with headphone stores - a lot of places let you plug them back and forth between whatever you want in your own manner. In fact, most of the time they leave you alone during the process.

 

[RickInHouston]

You can go back and forth between three different phones and find the one that has the sound you want.

...

But they may not be the ones you lose all sense of time with when listening to them after you take them home.

 

[bigshot]

Each set of cans has different impedance and sensitivity. There is no way to volume match. And the only stores where you aren’t bothered by salesmen are big box stores. They hide from customers!

 

[JDFlood]

Honestly, I think it is impossible. unless you are comparing maybe two pair of wildly different headphones. While you might note a few characteristics.. or subconsciously feel drawn to one sound type (this would be good). To really get an understanding how they are going to sound in your system you need hours in your system., not in someone else’s. Purchasing your first good set will be an educated guess. Then you can learn what they really sound like in your system, not just by critically analysis but by listening to the music, not the headphones. You really find out you have the right headphones when they disappear and you get lost in the music... well, unless you are one of those people that really enjoy the analytical process of evaluating equipment.

Take for instance my Focal Utopias. You could argue that the reviews are correct in their characterization (and I do)... but the differences i have heard are greater from the equipment feeding them than the differences in the headphones. This is more true the higher the quality of headphone. So for instance my Utopias and my HD800s really sound like my Aurender, Ayer DAC, and Woo WA5 amp more than the headphones. The difference between the sound through my HD800s and Utopia are surprisingly small... taking on the characteristics of great equipment I am fortunate to have feeding them.

There are a lot of reasons people own multiple headphones... for one, you can. I think in less than the top echelons headphones (And supporting equipment) they tend to do one kind of music better than others. Like Sennheiser and jazz, versus maybe Dennon and rock. Also, lots of listening is required to really characterize their sound.

At the higher levels of performance the characteristics converge... which makes sense if you think about it. You are trying to reproduce something with fidelity... using different materials and techniques... when you get really good they converge on the original.

So I think, read a lot, listen in the store as best you can. I find my subconscious better at the evaluation... which ones allow you to get lost in the music or emotionally grabs you. Grab them... they will probably not be your last. Enjoy what you get.

 

[sander99]

@JDFlood: New to the Sound Science forum?

Good thing you stumbled into the sound science forum, you might wanna do some reading here. The Sound Science subforum happens to be the only part of Head-Fi where certain well established truths can be told.

Short introduction:
Hearing perception is done mostly by the brain, and is influenced by other senses and all kinds of stuff going on in the brain. Because of this you (and also me and everyone else) can "hear" things or differences that are not audible! Placebo effect etc. van make you hear non-existent differences orders of magnitude larger than the smallest really audible differences (that can be descerned in a well controlled double blind ABX test).

So if a company sells some bull story with unsubstancited claims, and countless audiophiles repeat that story, and you believe that story, it can make one thing sound better to you than another thing even if there is no audible difference! Also many people will not even try for themselves, but just believe the story and buy the expensive item without any comparison.

Because of this many audiophile myths and claims could come into existence. Many of these myths and claims are in direct conflict with established science on the subject.

Read for example:
https://www.head-fi.org/threads/testing-audiophile-claims-and-myths.486598/

Now to your post above:
I have news for you. You are completely brainwashed by marketing bull and completely unaware of the enormous influence of expectatio bias etc. etc.
Most DACs are audible transparent, meaning a more expensive DAC won't sound any different (except when it is - deliberately - made to sound different = objectively worse), except in your mind. I won't go as far to say they all sound the same, but most do, including the built-in DACs in for example disc players, av receivers, etc.

Under certain conditions something similar can be said about amplifiers.

So in short: your story is completely backwards. The headphones (in general the transducers) is where the real differeces are, not the electronics.

And the worst advice one can give to @playmusic is to read a lot on head-fi outside the sound science subforum, or many other forums, where he will be bombarded with the subjective impressions of people completely unaware of above facts, parotting after each other or repeating marketing bull.

 

[playmusic]

Thank you for your feedback so far!
I particularly liked

Short answer: It's really hard. we're all doomed! Give up.

At least, I have been warned as was Dante:
Lasciate ogni speranza, voi ch'entrate!

Actually, when auditioning the headphones I listened to music pieces that I knew well. Checked.

Here are two of my assumptions:
I think that 44.1 kHz 16 bit recording is perfectly fine for music playback.
Amplifiers have no audible impact - unless there is sth fundamentally wrong, e.g. i) an electric guitar amplifier is operated outside its rated range in which case a tube amplifier sounds "better" than transistor or ii) a massive output headphone amplifier output impedance bends the frequency response of the headphone.

I am still concerned with the following topics:

1. If I hear a difference between two headphones: Is there really a difference?
2. After I convinced myself that two headphones sound different: Which sounds "better", i.e. which shall I get?

Ad 1: There are some less controversial differences, e.g. 2 $ earbuds that sound like a tin can or the difference between a Meze Empyrean and a Stax SR-009s when playing a bass heavy pop song. But then there is a large middle ground where I would not be so sure of my hearing.

Ad 2: Chances for a scientific answer on 2. are slim, even lower than for 1.
@RickInHouston even pointed out that chosing the headphone which has the "sound I want" may not be wise.

There is one headphone for which I am inclinded to think that 1. there is a difference and 2. it actually sounds "better" than the others is the Stax SR-009s - if bass heavy pop songs are excluded for the moment.
(I exclude also the Sennheiser HE-90 and HE-1 here because they are too pricey.)

Which still leaves the amplifier choice for the Stax open. I will have to be careful to not fall for amplifiers that are expensive without audible benefits.

Nevertheless, if you have further good advice on choosing/comparing headphones and avoiding expectation bias, I am looking forward to read it here.
After all, the 009s is a significant investment, so I will not buy it in the next few days.

 

[JDFlood]

@JDFlood: New to the Sound Science forum?

Now to your post above:
I have news for you. You are completely brainwashed by marketing bull and completely unaware of the enormous influence of expectatio bias etc. etc.
is to read a lot on head-fi outside the sound science subforum, or many other forums, where he will be bombarded with t parotting after each other or repeating marketing bull.

Thanks for setting us right with your superior knowledge. I do have two advanced degrees, have been a working scientist for over a decade and worked in the electronics industry for 40 years as well as being an audiophile for all that time. marketing hype... guess I should have seen that.

 

[sander99]

@JDFlood: It is not my superior knowledge, it is established science. Do you understand what audible transparency means?
Those expensive DACs can reconstruct the signal more accurate than cheap DACs, that is true. Instead of keeping imperfections 10-100 times below audibility they keep it for example 1000-10000 times below audibility. No audible difference.

If you know this and look more precise at some of the marketing it is kind of funny, for example Chord:
They say:
-our DACs reconstruct the signal more accurate (true)
-our DACs sound wonderfull (true, they sound just as wonderfull as any other audibly transparent DAC, audibly perfect)
They will never say:
-this DAC of ours sounds better than <some other DAC>
Because at the moment they do that they will be under attack, rightly so because at that moment they would be lying!
They leave it to the audiophools to make those statements for them in forums etc.

Anyway, if you think you really hear bigger or more important differences between different DACs and amplifiers than between headphones then why don't you go for the Carver challange (money to be won for who can recognise one amp from another under certain specific conditions)?

 

[KeithPhantom]

but the differences i have heard are greater from the equipment feeding them than the differences in the headphones.

I think it's the opposite, the gear you use most likely has lower distortion than the best headphones we can currently make. Moreover, even equipment with a SINAD of 55 dB can be frequently confused with transparent equipment by humans. We're pretty bad estimating non-linearities.

Like Sennheiser and jazz, versus maybe Dennon and rock. Also, lots of listening is required to really characterize their sound.

The FR of headphones is something that to a listener is subjective and cannot be correctly objectively categorized. I would say do not discriminate by brand and go and listen to headphones and choose the one you like the most.

 

[KeithPhantom]

Thanks for setting us right with your superior knowledge. I do have two advanced degrees, have been a working scientist for over a decade and worked in the electronics industry for 40 years as well as being an audiophile for all that time. marketing hype... guess I should have seen that.

Your degrees may help you, but they don't give you an automatic boost to your credibility. Evidence is needed even when a claim is presented by a top scientist in an area. As you say that you are a scientist, I may ask you to follow the scientific method when presenting a counterargument against the principles currently established.

 

[KeithPhantom]

Under certain conditions something similar can be said about amplifiers.

Actually, in most cases, amplifier should sound the same. Things that can make an amplifier sound different is the effect output impedance has on the transducer, level-matching, or extreme presence of non-linearities and their causes (usually SINAD < 55 dB).