What are the arguments against double blind tests (incl. ABX)?

We should clarify that ABX and double blind refer to two different things. You can have an ABX test that is double blind, but it doesn't have to be. And you can definitely have double blind trials that are not ABX. 

With that said, I have no arguments against them. They are extremely useful tools. 

Double blind testing is the defacto gold standard for most research because it helps control for both participant, and researcher bias. The poster who mentioned that A/B testing isn't sensitive enough for small differences, instead suggesting these differences take time to become noticeable is actually describing the reality of expectation and acclimitization. As you spend more time attempting to hear a difference (and if you spent money, you are trying to hear a difference don't kid yourself), you are building up the "evidence" you need to confirm your expectations of a sonic improvement and you are also getting used to a knew sound which subjectively may mean the sound is improved for you, but that doesn't actually mean in an objective sense the sound has improved.

I don't see how there could be any arguments against it. 

If someone can't tell the which is which between two cables, for example, 1000 times out of 1000, then it's all in their head. 

Double blind tests are useful. The arguments against double blind that I think you are referring to (correct me if I'm wrong) are the listening tests that have been done. That's not anti DB but about the validity in how they have been done. 

Studies have furthermore shown that long-term listening, which is often the argument given, is not as good as short-term comparative switching back and forth.

I don't think there are any credible arguments to be made against double-blind testing for this kind of application (if there are, please go ahead), but certainly there are some arguments to be made against some specific studies that have employed blind testing, not necessarily against the double blind procedure itself:

1.  Music is not revealing enough or otherwise doesn't have qualities X, Y, or Z

2.  Playback setup is not revealing enough

3.  Test subjects don't have good ears or aren't trained

4.  Test subjects have insufficient time or control over what they are listening to, aren't familiar with the system or the setup or the music

and so on.  Generally, for a decently well-run study, the above are pretty much covered, so the objections seem weak at best.  A null result, with people being unable to tell beyond guessing in a double-blind test, isn't sufficient to be absolutely conclusive proof that there's no audible difference.  But in terms of quality of evidence, it sure beats all kinds of sighted-listening anecdotes, which of course have next to zero credibility for any serious investigation.

the anti controlled test people often confuse blinding for "not knowing what is being tested" - you can tell subjects what audio "dimensions" to focus on, you can train with exaggerated examples of the differences, give them every "advantage" except knowing which stimulus is present at each "X" trial

The only real argument against objective testing is that it reveals subjective bias, and many seem to really need their subjective biases. 

Quote:

Originally Posted by skamp 


Usually when someone is asked to submit to a double blind test, it's in their own (familiar) environment, on their gear, with music they know by heart…
I'd like to hear more from the subjectivists. Certainly they wouldn't believe in something if they accepted the validity of contradicting results in double blind tests, surely they must have arguments against it? I'd like to hear it straight from the horse's mouth!
Edit: it just occured to me that they might not visit the Sound Science forum :-/

The double edged sword of being corralled into only having these types of discussions here. 

(edit: probably the wrong turn of phrase, since "double edged sword" that implies things cuts both ways and there is an upside... it doesn't - this is a one way marginalization of the conversation.) 

Quote:

Originally Posted by ILikeMusic The only real argument against objective testing is that it reveals subjective bias, and many seem to really need their subjective biases.

Yes, this is the right answer. There are people who are opposed to blind testing in principle. I suspect that deep down they know they will not be able to tell by listening alone the difference between expensive speaker wire and a coat hanger. So rather than admit they wasted their money on such nonsense, they make up excuses for why blind testing isn't valid.

--Ethan

There are only two arguments against blind tests:

1. The need for people who make overpriced junk to make money from said junk

2. Stupidity

Quote:

Originally Posted by scuttle 

There are only two arguments against blind tests:

 

1. The need for people who make overpriced junk to make money from said junk

 

2. Stupidity

Do people who need to justify their purchases fall into #2? It makes sense though. If I blew a handful of money on cables or something else "sketchy", why would I want to go through the steps to prove to myself & others that it was a waste?

Just to play devil's advocate.

Well, a lot of people purchase cables from places with generous return policies, so it would cost them virtually nothing to send them back (just their time), and many seem to keep them anyway. I'm a strong believer of double-blind ABX testing and a champion of the scientific method, skepticism as an approach to claims, etc, etc, but really often it's impractical.

For my needs, a striaght up AB testing performed by myself is good enough. I pick a test track I know very well and play a 10 to 15 sec clip in a loop and go back and forth and back forth, AB, AB, over and over again to see if I hear something I like in one and not in the other. I realize it's far from perfect but its practical for my needs.

Double-blind ABX would be ideal of course, but I would need to get at least a couple of people to help me go through the tedious process it might take hours if not days of help from others to collect enough data. I only have one girl friend... I guess I could get a 2nd for my ABX double-blind needs and try to explain this to the first girl...

If I was developing a new technology for NASA to send people to Mars, and every equipment needed to be tested, then yeah it would be worth my time and the time of others to rigorously test everything, especially using double-blind ABX testing (when prudent to do so), but honestly, audio gear for my personal enjoyment... it's not really worth it for me.

So the (devil's advocate) argument against using rigorous testing to determine if something (often quite expensive) actually provides audible benefit to a system, "is it's not worth the trouble", not that there are any flaws in the process which would invalidate the results?