Head-Fi.org › Forums › Equipment Forums › Sound Science › What are the arguments against double blind tests (incl. ABX)?
New Posts  All Forums:Forum Nav:

What are the arguments against double blind tests (incl. ABX)?

post #1 of 209
Thread Starter 
The argument for such tests is that if there's an audible difference, it should be detectable easily enough, even without any indication as to what is what (elimination of all bias and placebo effect). Failing that (the argument goes), any difference that was heard, was in fact imagined, due to various factors.

But what are the arguments against double blind tests? Why do some of the people who claim to hear differences refuse to submit to such tests? I found this one:
Quote:
I find differences among short interconnects difficult to hear. Speaker cable show some audible differences due to their longer runs. Blind A--B tests are not good at revealing small sonic differences. They show up more effectively in the long term listening when the ear/brain better latches on to (learns) the sonic signatures of components and cables.

That one is incompatible with claims of "night and day" differences though. Can you think of more like it?

I ask, because I feel there are many extremists in both camps (subjectivists vs. objectivists), and I (perhaps naively) hope to find arguments that can mitigate their differences.
Edited by skamp - 3/7/12 at 2:20pm
post #2 of 209

We should clarify that ABX and double blind refer to two different things. You can have an ABX test that is double blind, but it doesn't have to be. And you can definitely have double blind trials that are not ABX. 

 

With that said, I have no arguments against them. They are extremely useful tools. 

post #3 of 209

Double blind testing is the defacto gold standard for most research because it helps control for both participant, and researcher bias. The poster who mentioned that A/B testing isn't sensitive enough for small differences, instead suggesting these differences take time to become noticeable is actually describing the reality of expectation and acclimitization. As you spend more time attempting to hear a difference (and if you spent money, you are trying to hear a difference don't kid yourself), you are building up the "evidence" you need to confirm your expectations of a sonic improvement and you are also getting used to a knew sound which subjectively may mean the sound is improved for you, but that doesn't actually mean in an objective sense the sound has improved.

post #4 of 209

I don't see how there could be any arguments against it. 

 

If someone can't tell the which is which between two cables, for example, 1000 times out of 1000, then it's all in their head. 

post #5 of 209

Double blind tests are useful. The arguments against double blind that I think you are referring to (correct me if I'm wrong) are the listening tests that have been done. That's not anti DB but about the validity in how they have been done. 

post #6 of 209

Studies have furthermore shown that long-term listening, which is often the argument given, is not as good as short-term comparative switching back and forth.

 

I don't think there are any credible arguments to be made against double-blind testing for this kind of application (if there are, please go ahead), but certainly there are some arguments to be made against some specific studies that have employed blind testing, not necessarily against the double blind procedure itself:

1.  Music is not revealing enough or otherwise doesn't have qualities X, Y, or Z

2.  Playback setup is not revealing enough

3.  Test subjects don't have good ears or aren't trained

4.  Test subjects have insufficient time or control over what they are listening to, aren't familiar with the system or the setup or the music

 

and so on.  Generally, for a decently well-run study, the above are pretty much covered, so the objections seem weak at best.  A null result, with people being unable to tell beyond guessing in a double-blind test, isn't sufficient to be absolutely conclusive proof that there's no audible difference.  But in terms of quality of evidence, it sure beats all kinds of sighted-listening anecdotes, which of course have next to zero credibility for any serious investigation.

post #7 of 209

the anti controlled test people often confuse blinding for "not knowing what is being tested" - you can tell subjects what audio "dimensions" to focus on, you can train with exaggerated examples of the differences, give them every "advantage" except knowing which stimulus is present at each "X" trial

post #8 of 209

The only real argument against objective testing is that it reveals subjective bias, and many seem to really need their subjective biases. 

 

post #9 of 209
Thread Starter 
Quote:
Originally Posted by mikeaj View Post

1.  Music is not revealing enough or otherwise doesn't have qualities X, Y, or Z
2.  Playback setup is not revealing enough
3.  Test subjects don't have good ears or aren't trained
4.  Test subjects have insufficient time or control over what they are listening to, aren't familiar with the system or the setup or the music

Usually when someone is asked to submit to a double blind test, it's in their own (familiar) environment, on their gear, with music they know by heart…

I'd like to hear more from the subjectivists. Certainly they wouldn't believe in something if they accepted the validity of contradicting results in double blind tests, surely they must have arguments against it? I'd like to hear it straight from the horse's mouth!

Edit: it just occured to me that they might not visit the Sound Science forum :-/
Edited by skamp - 3/8/12 at 1:54am
post #10 of 209
Quote:
Originally Posted by skamp View Post


Usually when someone is asked to submit to a double blind test, it's in their own (familiar) environment, on their gear, with music they know by heart…
I'd like to hear more from the subjectivists. Certainly they wouldn't believe in something if they accepted the validity of contradicting results in double blind tests, surely they must have arguments against it? I'd like to hear it straight from the horse's mouth!
Edit: it just occured to me that they might not visit the Sound Science forum :-/


The double edged sword of being corralled into only having these types of discussions here. 

 

(edit: probably the wrong turn of phrase, since "double edged sword" that implies things cuts both ways and there is an upside... it doesn't - this is a one way marginalization of the conversation.) 


Edited by liamstrain - 3/8/12 at 7:27am
post #11 of 209
Quote:
Originally Posted by ILikeMusic View PostThe only real argument against objective testing is that it reveals subjective bias, and many seem to really need their subjective biases.


Yes, this is the right answer. There are people who are opposed to blind testing in principle. I suspect that deep down they know they will not be able to tell by listening alone the difference between expensive speaker wire and a coat hanger. So rather than admit they wasted their money on such nonsense, they make up excuses for why blind testing isn't valid.

 

--Ethan

post #12 of 209

There are only two arguments against blind tests:

 

1. The need for people who make overpriced junk to make money from said junk

 

2. Stupidity


Edited by scuttle - 3/8/12 at 12:06pm
post #13 of 209
Quote:
Originally Posted by scuttle View Post

There are only two arguments against blind tests:

 

1. The need for people who make overpriced junk to make money from said junk

 

2. Stupidity



Do people who need to justify their purchases fall into #2? It makes sense though. If I blew a handful of money on cables or something else "sketchy", why would I want to go through the steps to prove to myself & others that it was a waste?


Edited by Vkamicht - 3/8/12 at 12:45pm
post #14 of 209

Just to play devil's advocate.

 

Well, a lot of people purchase cables from places with generous return policies, so it would cost them virtually nothing to send them back (just their time), and many seem to keep them anyway. I'm a strong believer of double-blind ABX testing and a champion of the scientific method, skepticism as an approach to claims, etc, etc, but really often it's impractical.

 

For my needs, a striaght up AB testing performed by myself is good enough. I pick a test track I know very well and play a 10 to 15 sec clip in a loop and go back and forth and back forth, AB, AB, over and over again to see if I hear something I like in one and not in the other. I realize it's far from perfect but its practical for my needs.

 

Double-blind ABX would be ideal of course, but I would need to get at least a couple of people to help me go through the tedious process it might take hours if not days of help from others to collect enough data. I only have one girl friend... I guess I could get a 2nd for my ABX double-blind needs and try to explain this to the first girl... very_evil_smiley.gif

 

If I was developing a new technology for NASA to send people to Mars, and every equipment needed to be tested, then yeah it would be worth my time and the time of others to rigorously test everything, especially using double-blind ABX testing (when prudent to do so), but honestly, audio gear for my personal enjoyment... it's not really worth it for me.

post #15 of 209

So the (devil's advocate) argument against using rigorous testing to determine if something (often quite expensive) actually provides audible benefit to a system, "is it's not worth the trouble", not that there are any flaws in the process which would invalidate the results?

 

New Posts  All Forums:Forum Nav:
  Return Home
  Back to Forum: Sound Science
Head-Fi.org › Forums › Equipment Forums › Sound Science › What are the arguments against double blind tests (incl. ABX)?