Head-Fi.org › Forums › Equipment Forums › Sound Science › Are blind tests bogus? Examples of blind tests with positive results.
New Posts  All Forums:Forum Nav:

Are blind tests bogus? Examples of blind tests with positive results.

post #1 of 20
Thread Starter 

Some claim that blind tests are bogus because they are designed to create a fail or cannot be passed and they involve trickery or deception.

 

I do not think that is true because

 

- they can be passed, such as blind tests of speakers and bit rates. Use the same test with cables and the result is a fail. That is because there is no difference between cables, but there is one between speakers and bit rates. If the test was itself designed to fail, then why not fail with speakers?

 

- I do not like the blind tests that have been done where people are told they are listening to different cables, but in fact there has been no change. It is interesting, but again a bit dubious when something like a wire coat hanger is slipped into a cable test without anyone's knowledge. My preference is the simple two cables, let the subject see and hear both in action. Then once blinded use one and then say, 'I am may or may not change to the other cables now, please say if you can hear a difference or not?' Then repeat that for about 20 times. Where is the deception in that?

 

You can also decide if you do have 'golden ears' here

 

http://www.audiocheck.net/blindtests_index.php

 

So here are blind tests that have positive results where people could hear a difference.

 

1 - A blind test of speakers, passed by the subject. Interestingly, the subject failed to identify different crossovers, one more expensive than the other.

 

http://www.audioholics.com/news/editorials/axiom-blind-listening-test

 

2 - One of amps with 500 participants EDIT - it is debatable whether this is actually a pass or not.

 

http://www.stereophile.com/features/113/index.html

 

3 - Power amp blind tests, two of which are positive.

 

http://home.provide.net/~djcarlst/abx_data.htm

 

4 - Head-Fi, by member Pio2001 between a Marantz integrated amp and Pro-ject Headphone amp

 

http://www.head-fi.org/forum/thread/429619/headphone-outputs-lots-of-measurments-and-one-abx

 

5 - An interesting Boston Audio Society article about two tests. The tweaked CD test is a fail, but read on and an amplifier blind test is a pass.

 

http://www.bostonaudiosociety.org/bas_speaker/wishful_thinking.htm

 

6 - A Hydrogen Audio test of different gauge speaker cables.

 

http://www.hydrogenaudio.org/forums/index.php?showtopic=14082&st=25

 

7 - PSB speaker blind test, the top of the range speaker won

 

screenOvenNRC.jpg

 

http://www.psbspeakers.com/audio-topics/Birthplace-of-Good-Sound

 

8 - ABX Comparator. A series of blind tests of different kit and cables.

 

Starting with the cables, differences were found with video cables over very long runs of 100 feet in comparison to a 6 foot one.

 

http://home.provide.net/~djcarlst/abx_vid.htm

 

Then interconnect and speaker cables, five tests and no differences found.

 

http://home.provide.net/~djcarlst/abx_wire.htm

 

A speaker test with a very large sample found 97% could tell the difference

 

http://home.provide.net/~djcarlst/abx_spk.htm

 

CDPs and a DAC did less well

 

http://home.provide.net/~djcarlst/abx_cd.htm

 

Power amps did a bit better

 

http://home.provide.net/~djcarlst/abx_pwr.htm

 

But what was very noticeable was the likes of distortion, filters and a small change in volume

 

http://home.provide.net/~djcarlst/abx_data.htm

 

9 - Matrix Hifi between two amps where two testers got all 30 tests correct, 60 attempts in all (in Spanish, Google translator used)

 

http://translate.google.com/translate?js=n&prev=_t&hl=en&ie=UTF-8&layout=2&eotf=1&sl=es&tl=en&u=http%3A%2F%2Fwww.matrixhifi.com%2Fcontenedor_trivsclat.htm&act=url

 

10 - 16bit vs 24bit, Gearslutz.com forum, 9 out 10 correctly identified

 

http://www.gearslutz.com/board/so-much-gear-so-little-time/358301-24-vs-16-bit-not-audible.html

 

11. Sampling rate, 44.2kHz vs 88.2 kHz, AES May 2010

 

http://www.aes.org/e-lib/browse.cfm?elib=15398


Edited by Prog Rock Man - 7/25/11 at 11:54am
post #2 of 20

I agree that a blind test should be done with a certain method known to everyone and said method stuck to; cable A, B and then cable X and preferably nobody knowing what cable X is, so no 'surprises', otherwise the people listening will end up not knowing what is going on and will spend the time being paranoid and unfocused, "are they trying to make me look stupid", etc.

 

I think such a test properly done can only be a good thing, and I say that as someone who appreciates the value of being protected from their own delusions/bias/etc.

post #3 of 20
Thread Starter 

The Skeptic's attitude to blind testing....

 

http://www.skepdic.com/control.html

 

It includes a blind test of a device for detecting people hiding or trapped. The device worked 10 times out of 10 when the maker knew where the person was and 6 out of 25 times when the person was actually hiding. Imagine the consequences of people buying into that device not knowing it does not actually work. Horrifying. The test with dowser who got none right is thankfully less scary.

 

The article concludes that it is no wonder certain groups shy away from blind testing. I would have to add cable makers to that group.

post #4 of 20
Thread Starter 

I am going to keep going with positive blind test results and adding them to the first post to make them easier to find. I am sure that positive blind tests prove that blind tests are not rigged or are bogus.


Edited by Prog Rock Man - 9/22/10 at 11:36am
post #5 of 20

Many blind tests fail, although depending what you were trying to prove this may be success, because they begin with the stipulation that the devices under test be indistinguishable within some tolerance to non-human test equipment. 

 

After you verify electrical similarity of 2 things to within your predetermined tolerances its a question of whether the human listener is better at telling 2 things apart than the measurement equipment, and he/she never is. From this we can (incorectly) conclude no difference across the whole spectrum when all we have proven is that the listener is not as good as the machine at telling 2 pieces of gear apart.

 

If you remove the machine from the test It is not hard to make almost anything that makes an audible difference to the sound. Whether its better or worse is a matter of taste. Some things are simply screwed up, others are totally subjective.

post #6 of 20

Interesting site none the less. I think it's useful to critically listen to things like this so when you make assessments on gear you can draw more accurate conclusions. The dynamic range test is especially useful to me. Some people don't realize that because of the plethora of background noise present in there listening environment, it greatly reduces there ability to pickup subtle nuances or mixing changes. A quiet listening floor is key to really extracting all the notes out of music. I did that dynamic test in two different rooms vastly different outcomes


Edited by Lucias_D - 9/24/10 at 12:20am
post #7 of 20
Quote:
Originally Posted by Prog Rock Man View Post

Some claim that blind tests are bogus because they are designed to create a fail or cannot be passed and they involve trickery or deception.

 

I do not think that is true because

 

- they can be passed, such as blind tests of speakers and bit rates. Use the same test with cables and the result is a fail. That is because there is no difference between cables, but there is one between speakers and bit rates. If the test was itself designed to fail, then why not fail with speakers?

 

- I do not like the blind tests that have been done where people are told they are listening to different cables, but in fact there has been no change. It is interesting, but again a bit dubious when something like a wire coat hanger is slipped into a cable test without anyone's knowledge. My preference is the simple two cables, let the subject see and hear both in action. Then once blinded use one and then say, 'I am may or may not change to the other cables now, please say if you can hear a difference or not?' Then repeat that for about 20 times. Where is the deception in that?

 

You can also decide if you do have 'golden ears' here

 

http://www.audiocheck.net/blindtests_index.php

 

So here are blind tests that have positive results where people could hear a difference.

 

1 - A blind test of speakers, passed by the subject. Interestingly, the subject failed to identify different crossovers, one more expensive than the other.

 

http://www.audioholics.com/news/editorials/axiom-blind-listening-test

 

2 - One of amps with 500 participants

 

http://www.stereophile.com/features/113/index.html



For your 2nd citation it is important that readers read the 'letters part 4" where an experimental psychologist comments on procedural issues such as the human bias to report differences regardless and the non-balalanced number of same/different tests - this does undermine the statistical strength of the results to the point that the  rate of correct detection (52.3%)  is not significantly different from that expected by chance (50.9%).

post #8 of 20
Thread Starter 

The point with the Stereophile test that made it seem one to put in as a pass was the number of people who did pass the blind test. But thinking back on it, that would be expected even in a random situation. Ah well. The hunt goes on for passed blind tests.

post #9 of 20
Quote:
Originally Posted by Prog Rock Man View Post

The point with the Stereophile test that made it seem one to put in as a pass was the number of people who did pass the blind test. But thinking back on it, that would be expected even in a random situation. Ah well. The hunt goes on for passed blind tests.


 

There ar some positive DBTs here...


http://home.provide.net/~djcarlst/abx_data.htm


Edited by nick_charles - 9/24/10 at 8:24am
post #10 of 20
Thread Starter 

Thanks, added to the list.

post #11 of 20

What about the positive one reported on head-fi distinguishing a headphone amp and the headphone out of an integrated amplifier?

http://www.head-fi.org/forum/thread/429619/headphone-outputs-lots-of-measurments-and-one-abx

 

Did you explore the full list (negative and positive) at hydrogenaudio?

http://www.hydrogenaudio.org/forums/index.php?showtopic=82777

post #12 of 20
Thread Starter 

Thanks imyself. I have added the first test to the original post and will work my way through the rest. I will put positives here and negatives in this thread -

 

http://www.head-fi.org/forum/thread/486598/testing-audiophile-claims-and-myths

 

post #13 of 20

10/10 on the 1dB test on my first try without listening to the test tones.  10/10 on the 5cent test on my second try. 10/10 on the C-scale on my first try.

post #14 of 20
Thread Starter 

Bump as more tests added. The winner in audibility by far is the speaker, the loser is the cable.

post #15 of 20

The infamous Carver Challenge is worth listing here, although it's a bit different in premise from the typical blind test:

http://www.stereophile.com/features/the_carver_challenge/

New Posts  All Forums:Forum Nav:
  Return Home
  Back to Forum: Sound Science
Head-Fi.org › Forums › Equipment Forums › Sound Science › Are blind tests bogus? Examples of blind tests with positive results.