Are blind tests bogus? Examples of blind tests with positive results.
Sep 20, 2010 at 7:43 AM Thread Starter Post #1 of 30

Prog Rock Man

Headphoneus Supremus
Joined
Jul 2, 2009
Posts
3,812
Likes
196
Some claim that blind tests are bogus because they are designed to create a fail or cannot be passed and they involve trickery or deception.

I do not think that is true because

- they can be passed, such as blind tests of speakers and bit rates. Use the same test with cables and the result is a fail. That is because there is no difference between cables, but there is one between speakers and bit rates. If the test was itself designed to fail, then why not fail with speakers?

- I do not like the blind tests that have been done where people are told they are listening to different cables, but in fact there has been no change. It is interesting, but again a bit dubious when something like a wire coat hanger is slipped into a cable test without anyone's knowledge. My preference is the simple two cables, let the subject see and hear both in action. Then once blinded use one and then say, 'I am may or may not change to the other cables now, please say if you can hear a difference or not?' Then repeat that for about 20 times. Where is the deception in that?

You can also decide if you do have 'golden ears' here

http://www.audiocheck.net/blindtests_index.php

So here are blind tests that have positive results where people could hear a difference.

1 - A blind test of speakers, passed by the subject. Interestingly, the subject failed to identify different crossovers, one more expensive than the other.

http://www.audioholics.com/news/editorials/axiom-blind-listening-test

2 - One of amps with 500 participants

http://www.stereophile.com/features/113/index.html

EDIT - it is debatable whether this is actually a pass or not.

3 - Head-Fi, by member Pio2001 between a Marantz integrated amp and Pro-ject Headphone amp

https://www.head-fi.org/threads/headphone-outputs-lots-of-measurments-and-one-abx.429619/

4 - An interesting Boston Audio Society article about two tests. The tweaked CD test is a fail, but read on and an amplifier blind test is a pass.

http://www.bostonaudiosociety.org/bas_speaker/wishful_thinking.htm

5 - A Hydrogen Audio test of different gauge speaker cables.

http://www.hydrogenaudio.org/forums/index.php?showtopic=14082&st=25

6 - PSB speaker blind test, the top of the range speaker won

http://www.psbspeakers.com/audio-topics/Birthplace-of-Good-Sound

Edit - link broken.

7 - ABX Comparator. A series of blind tests of different kit and cables.

Starting with the cables, differences were found with video cables over very long runs of 100 feet in comparison to a 6 foot one.

http://djcarlst.provide.net/abx_vid.htm

Then interconnect and speaker cables, five tests and no differences found.

http://djcarlst.provide.net/abx_wire.htm

A speaker test with a very large sample found 97% could tell the difference

http://djcarlst.provide.net/abx_spk.htm

CDPs and a DAC did less well

http://djcarlst.provide.net/abx_cd.htm

Power amps did a bit better

http://djcarlst.provide.net/abx_pwr.htm

But what was very noticeable was the likes of distortion, filters and a small change in volume

http://djcarlst.provide.net/abx_dist.htm
http://djcarlst.provide.net/abx_f4.htm
http://djcarlst.provide.net/abx_lvl.htm

8 - Matrix Hifi between two amps where two testers got all 30 tests correct, 60 attempts in all (in Spanish, Google translator used)

http://translate.google.com/translate?js=n&prev=_t&hl=en&ie=UTF-8&layout=2&eotf=1&sl=es&tl=en&u=http://www.matrixhifi.com/contenedor_trivsclat.htm&act=url

9 - 16bit vs 24bit, Gearslutz.com forum, 9 out 10 correctly identified

http://www.gearslutz.com/board/so-much-gear-so-little-time/358301-24-vs-16-bit-not-audible.html

10 - Sampling rate, 44.2kHz vs 88.2 kHz, AES May 2010

http://www.aes.org/e-lib/browse.cfm?elib=15398

11 -"Some amplifiers do sound different" report c1984

https://audiosciencereview.com/foru...ts-did-show-amplifiers-to-sound-different.23/

Two amps, an apparently well run test and the cheaper amp wins!

12 - "Blind listening to cables..." HiFi+ Magazine

https://www.nordost.com/images/review-images/review-pdf/15-hifiplus_issue34.pdf


Three listeners, 6 cables, it appears to be a well run test and there is a conclusion, with the most expensive cable winning.
 
Last edited:
Sep 20, 2010 at 8:21 AM Post #2 of 30
I agree that a blind test should be done with a certain method known to everyone and said method stuck to; cable A, B and then cable X and preferably nobody knowing what cable X is, so no 'surprises', otherwise the people listening will end up not knowing what is going on and will spend the time being paranoid and unfocused, "are they trying to make me look stupid", etc.
 
I think such a test properly done can only be a good thing, and I say that as someone who appreciates the value of being protected from their own delusions/bias/etc.
 
Sep 22, 2010 at 12:54 PM Post #3 of 30
The Skeptic's attitude to blind testing....
 
http://www.skepdic.com/control.html
 
It includes a blind test of a device for detecting people hiding or trapped. The device worked 10 times out of 10 when the maker knew where the person was and 6 out of 25 times when the person was actually hiding. Imagine the consequences of people buying into that device not knowing it does not actually work. Horrifying. The test with dowser who got none right is thankfully less scary.
 
The article concludes that it is no wonder certain groups shy away from blind testing. I would have to add cable makers to that group.
 
Sep 22, 2010 at 2:03 PM Post #4 of 30
I am going to keep going with positive blind test results and adding them to the first post to make them easier to find. I am sure that positive blind tests prove that blind tests are not rigged or are bogus.
 
Sep 24, 2010 at 1:51 AM Post #5 of 30
Many blind tests fail, although depending what you were trying to prove this may be success, because they begin with the stipulation that the devices under test be indistinguishable within some tolerance to non-human test equipment. 
 
After you verify electrical similarity of 2 things to within your predetermined tolerances its a question of whether the human listener is better at telling 2 things apart than the measurement equipment, and he/she never is. From this we can (incorectly) conclude no difference across the whole spectrum when all we have proven is that the listener is not as good as the machine at telling 2 pieces of gear apart.
 
If you remove the machine from the test It is not hard to make almost anything that makes an audible difference to the sound. Whether its better or worse is a matter of taste. Some things are simply screwed up, others are totally subjective.
 
Sep 24, 2010 at 3:05 AM Post #6 of 30
Interesting site none the less. I think it's useful to critically listen to things like this so when you make assessments on gear you can draw more accurate conclusions. The dynamic range test is especially useful to me. Some people don't realize that because of the plethora of background noise present in there listening environment, it greatly reduces there ability to pickup subtle nuances or mixing changes. A quiet listening floor is key to really extracting all the notes out of music. I did that dynamic test in two different rooms vastly different outcomes
 
Sep 24, 2010 at 11:10 AM Post #7 of 30


Quote:
Some claim that blind tests are bogus because they are designed to create a fail or cannot be passed and they involve trickery or deception.
 
I do not think that is true because
 
- they can be passed, such as blind tests of speakers and bit rates. Use the same test with cables and the result is a fail. That is because there is no difference between cables, but there is one between speakers and bit rates. If the test was itself designed to fail, then why not fail with speakers?
 
- I do not like the blind tests that have been done where people are told they are listening to different cables, but in fact there has been no change. It is interesting, but again a bit dubious when something like a wire coat hanger is slipped into a cable test without anyone's knowledge. My preference is the simple two cables, let the subject see and hear both in action. Then once blinded use one and then say, 'I am may or may not change to the other cables now, please say if you can hear a difference or not?' Then repeat that for about 20 times. Where is the deception in that?
 
You can also decide if you do have 'golden ears' here
 
http://www.audiocheck.net/blindtests_index.php
 
So here are blind tests that have positive results where people could hear a difference.
 
1 - A blind test of speakers, passed by the subject. Interestingly, the subject failed to identify different crossovers, one more expensive than the other.
 
http://www.audioholics.com/news/editorials/axiom-blind-listening-test
 
2 - One of amps with 500 participants
 
http://www.stereophile.com/features/113/index.html



For your 2nd citation it is important that readers read the 'letters part 4" where an experimental psychologist comments on procedural issues such as the human bias to report differences regardless and the non-balalanced number of same/different tests - this does undermine the statistical strength of the results to the point that the  rate of correct detection (52.3%)  is not significantly different from that expected by chance (50.9%).
 
Sep 24, 2010 at 11:15 AM Post #8 of 30
The point with the Stereophile test that made it seem one to put in as a pass was the number of people who did pass the blind test. But thinking back on it, that would be expected even in a random situation. Ah well. The hunt goes on for passed blind tests.
 
Sep 24, 2010 at 11:22 AM Post #9 of 30
Sep 24, 2010 at 12:20 PM Post #10 of 30
Thanks, added to the list.
 
Sep 25, 2010 at 10:17 AM Post #12 of 30
Thanks imyself. I have added the first test to the original post and will work my way through the rest. I will put positives here and negatives in this thread -
 
http://www.head-fi.org/forum/thread/486598/testing-audiophile-claims-and-myths
 
 
Oct 8, 2010 at 1:31 PM Post #14 of 30
Bump as more tests added. The winner in audibility by far is the speaker, the loser is the cable.
 
Oct 8, 2010 at 7:17 PM Post #15 of 30
The infamous Carver Challenge is worth listing here, although it's a bit different in premise from the typical blind test:
http://www.stereophile.com/features/the_carver_challenge/
 

Users who are viewing this thread

Back
Top