Head-Fi.org › Forums › Equipment Forums › Sound Science › What are the arguments against double blind tests (incl. ABX)?
New Posts  All Forums:Forum Nav:

What are the arguments against double blind tests (incl. ABX)? - Page 9

post #121 of 209
Thread Starter 
Quote:
Originally Posted by Prog Rock Man View Post

If one person hearing a difference is a guarantee of audibility, then using the same logic one person not hearing a difference becomes a guarantee of inaudibility. Both claims are in fact logical fallicies.

That's not what I said. I claimed that a successful test is guarantee of audibility to the person who passed the test (and others might pass it as well), and a failed test is not a guarantee of inaudibility for every single person out there. I don't see a fallacy here, you seem to be misinterpreting what I'm saying.
Quote:
Originally Posted by Prog Rock Man View Post

Please also, with reference to specific tests show me how the positive ones are more valid than the negative ones in terms of their results.

It's a simple matter of logic. By design, passed tests give us certainty as far as human capabilities go, and failed tests in the absence of any successful test give us varying probability of inaudibility (in general). I can't explain it better than I already have.

Edit: I'm a proponent of such tests. Was that not clear?
Edited by skamp - 3/16/12 at 11:27am
post #122 of 209
Quote:
Originally Posted by skamp View Post

Is there any other type of tests that would undeniably prove the existence of audible differences? One that DBT/ABX detractors would approve of, hopefully?


Yes, null testing, a topic I have tried to discuss a couple of times in the past but there was no interest. With null testing you record a piece of music, make your change with the hifi (swop cables, or amp or whatever) and then make another recording and then subtract one from the other and see if you can hear what is left. This is a programme to that, but I am not computer literate enough to make it work

 

http://www.libinst.com/Audio%20DiffMaker.htm

 

 

 

post #123 of 209
Quote:
Originally Posted by skamp View Post


That's not what I said. I claimed that a successful test is guarantee of audibility to the person who passed the test (and others might pass it as well), and a failed test is not a guarantee of inaudibility for every single person out there. I don't see a fallacy here, you seem to be misinterpreting what I'm saying.
It's a simple matter of logic. By design, passed tests give us certainty as far as human capabilities go, and failed tests in the absence of any successful test give us varying probability of inaudibility (in general). I can't explain it better than I already have.
Edit: I'm a proponent of such tests. Was that not clear?


I wouldn't say a pass gives certainty. There's always a chance of guessing, no matter the number of trials. But with a P value of 0.05 the probability of guessing is low enough that we can assume the difference is audible.

 

We have to be careful of extremes at all times. Nothing is certain, even if it seems it should be. Gravity could suddenly reverse tomorrow, for all we know about physics. It's not likely by any means, but it's not certain it won't happen.

post #124 of 209
Quote:
Originally Posted by skamp View Post


That's not what I said. I claimed that a successful test is guarantee of audibility to the person who passed the test (and others might pass it as well), and a failed test is not a guarantee of inaudibility for every single person out there. I don't see a fallacy here, you seem to be misinterpreting what I'm saying.
Appologies I have got that wrong then.

It's a simple matter of logic. By design, passed tests give us certainty as far as human capabilities go, and failed tests in the absence of any successful test give us varying probability of inaudibility (in general). I can't explain it better than I already have.
But there have been successful as well as failed tests and such have been consistent. That must tell us both results are correct.

Edit: I'm a proponent of such tests. Was that not clear?


 

post #125 of 209
Thread Starter 
Quote:
Originally Posted by Prog Rock Man View Post

With null testing you record a piece of music, make your change with the hifi (swop cables, or amp or whatever) and then make another recording and then subtract one from the other and see if you can hear what is left.

That only shows an actual difference in the signal, but not that it's audible. A null test of a perceptual codec, for instance, is clearly audible, but it doesn't prove that the codec alone is not fully transparent. Now THAT is a fallacy often used by people who claim that even high bitrate lossy codecs sound like crap, when really they probably couldn't ABX them. Even if there are measurable differences between cables, it doesn't mean I'll be able to hear them. So I don't consider the null test a viable alternative.
Quote:
Originally Posted by Head Injury View Post

I wouldn't say a pass gives certainty.

Agreed. Again, I went too fast. A pass gives us probability so high that it is by all practical means equivalent to certainty. Well, that assessment is debatable, but you get my point. We essentially agree.
Edited by skamp - 3/16/12 at 11:46am
post #126 of 209

I disagree, a null test will show if the difference is A there and B audible.

post #127 of 209

It shows if the difference is audible, in isolation. Not if it is audible in situ. 

 

e.g., it is easier to hear an isolated 3rd order harmonic against a black background, than it is as part of a 4 part chord with full strings and piano. 

 

Is it audible? alone, technically, yes. This does not prove discernible - which is perhaps a better word than audible for this purpose.


Edited by liamstrain - 3/16/12 at 12:20pm
post #128 of 209
Thread Starter 
Quote:
Originally Posted by Prog Rock Man View Post

I disagree, a null test will show if the difference is A there and B audible.

Take a high bitrate lossy encode, say Ogg Vorbis -q 8. Do the null test with the FLAC file it was encoded from. Is the difference audible? Most likely.

Now run an ABX test with the Ogg Vorbis file. Will you pass it successfully? Probably not.
post #129 of 209

If no one can pick out something in an ABX test and a null test finds no audible difference, do any of you guys still think that something may still be audible?

post #130 of 209
Thread Starter 
Nope, but that's not helpful when it comes to differences that are real, yet inaudible, e.g. with DACs or perceptual codecs.
post #131 of 209

Blind testing will certainly root out "differences that are real, yet inaudible".

post #132 of 209
Thread Starter 
I guess it all comes down to the fact that the burden of proof lies on those who claim to hear a difference, and if they can't pass DB/ABX tests successfully, it is for them to find other means of producing adequate proof of their claims. I'm curious to know if there are any.
post #133 of 209
Originally Posted by skamp View Post

We're not scientists, but I believe that's what we're doing by debating the merits of DBT.
Do you agree that successful ABX tests are reliable, and that sighted tests are unreliable? And again, do you have anything better in mind?


Yes, I agree successful ABX tests are reliable at confirming differences. And poor at confirming preferences - e.g. the pepsi/coke results were skewed because the testers would be overtly influenced by sweetness in a quick swap test. Again, this limitation is more to do with the fast swap nature of typical ABX tests, rather than anything wrong with the ABX principle.

 

Yes, I agree that sighted tests are unreliable, because of expectation bias.

 

For what I call "Essentally Neutral Components", like cables, the most reliable test I know of is no test at all: Listen normally to your unchanged system for months, going through a range of circumstances (tired/bored/happy/sad/etc).

Then swap exactly one component and carry on listening to any of your recent heavy rotation albums - just enjoying the music.

 

If the new component is significantly better, the difference will hit you in the first few seconds. Any further listening is unnecessary, and will only confirm those first few seconds.

If you don't hear a difference in the first few seconds, then the new component is not for you.

If you then start swapping A with B, then you lose much of those months of "preparation" and quickly end up with all the problems of fast swapping tests.

 

For reasons I can't really explain, I don't appear to have an issue with expectation bias in the above test. On the basis that every single time I've been genuinely surprised with the results being different to what I had expected based on that component's price, sound signature reputation, etc.  It's possible this is a sort of complex double delusion, but the results have been so clear and against expectation that I'm willing to live with that possibility.

 

The above test is obviously impractical and time consuming, So most of the time, I have to make do with a reduced version, which introduced unreliability - such is life.

It doesn't follow that what works best for me will work best for anyone else.

 

 


Edited by TheAttorney - 3/17/12 at 5:47am
post #134 of 209

FWIW, my view is that by finding out the different results between sighted, blind comparison and ABX tests we can then make a more informed decision about our hifi and what we need.  The wine industry has not ignored blind testing like the hifi industry has. Wine tasters and reviewers have not banned blind testing like many hifi forums have. To them it is not a dirty secret to be hidden away. Expensive brands of wine still live next to supermarket branded wines and people are able and indeed encouraged to enjoy both.

 

So why not do the same with hifi? Let people know the different results between sighted and the blind tests. Let people know that when sighted a £100 USB cable can sound better than a £5 one in a sighted test. But in a blind test no one can hear a difference. Then people can make up their own minds as to which one to buy. I do not believe for one minute the hifi industry will then collapse. Just as the wine industry has not collapsed. There is a place for both a £100 and a £5 USB cable.

 

Fact is that the arguments against blind tests are more about the results they give than the test procedure itself. That some here will accept a positive result and not a negative result from the same procedure shows how people are repared to manipulate blind testing and ignore proper sceintific process. I also see that the various theories against blind testing have not been accompanied by any evidence or experimentation. The theories also ignore the sheer number of blind tests already conducted in different places, different equipment and different people which have consistently produced the saem results.

 

Indeed it is odd how so many audiophiles, as bright and well educated as many clearly are, behave. As Wapiti, head fi forum member said "It is striking how in a hobby based on the creations of science, science is readily and routinely both ignored and held highly suspect."

 

 

post #135 of 209
Thread Starter 
Quote:
Originally Posted by Prog Rock Man View Post

That some here will accept a positive result and not a negative result from the same procedure shows how people are repared to manipulate blind testing and ignore proper sceintific process.

Well, as we said earlier, positive outcomes and negative ones are two very different (shall I say, asymmetrical?) animals. I don't think it's unfair to readily accept the former, yet question the latter. My main problem is that detractors often ignore Occam's Razor, the likelihood in the midst of overwhelming evidence that there is no difference to be heard.

As far as Head-Fi is concerned, the presence of various sponsors makes it a little awkward, to say the least :-/
New Posts  All Forums:Forum Nav:
  Return Home
  Back to Forum: Sound Science
Head-Fi.org › Forums › Equipment Forums › Sound Science › What are the arguments against double blind tests (incl. ABX)?