What are the arguments against double blind tests (incl. ABX)? - Page 9

Quote:

Originally Posted by skamp 

Is there any other type of tests that would undeniably prove the existence of audible differences? One that DBT/ABX detractors would approve of, hopefully?

Yes, null testing, a topic I have tried to discuss a couple of times in the past but there was no interest. With null testing you record a piece of music, make your change with the hifi (swop cables, or amp or whatever) and then make another recording and then subtract one from the other and see if you can hear what is left. This is a programme to that, but I am not computer literate enough to make it work

http://www.libinst.com/Audio%20DiffMaker.htm

Quote:

Originally Posted by skamp 


That's not what I said. I claimed that a successful test is guarantee of audibility to the person who passed the test (and others might pass it as well), and a failed test is not a guarantee of inaudibility for every single person out there. I don't see a fallacy here, you seem to be misinterpreting what I'm saying.
It's a simple matter of logic. By design, passed tests give us certainty as far as human capabilities go, and failed tests in the absence of any successful test give us varying probability of inaudibility (in general). I can't explain it better than I already have.
Edit: I'm a proponent of such tests. Was that not clear?

I wouldn't say a pass gives certainty. There's always a chance of guessing, no matter the number of trials. But with a P value of 0.05 the probability of guessing is low enough that we can assume the difference is audible.

We have to be careful of extremes at all times. Nothing is certain, even if it seems it should be. Gravity could suddenly reverse tomorrow, for all we know about physics. It's not likely by any means, but it's not certain it won't happen.

I disagree, a null test will show if the difference is A there and B audible.

It shows if the difference is audible, in isolation. Not if it is audible in situ. 

e.g., it is easier to hear an isolated 3rd order harmonic against a black background, than it is as part of a 4 part chord with full strings and piano. 

Is it audible? alone, technically, yes. This does not prove discernible - which is perhaps a better word than audible for this purpose.

If no one can pick out something in an ABX test and a null test finds no audible difference, do any of you guys still think that something may still be audible?

Blind testing will certainly root out "differences that are real, yet inaudible".

Originally Posted by skamp 

We're not scientists, but I believe that's what we're doing by debating the merits of DBT.
Do you agree that successful ABX tests are reliable, and that sighted tests are unreliable? And again, do you have anything better in mind?

Yes, I agree successful ABX tests are reliable at confirming differences. And poor at confirming preferences - e.g. the pepsi/coke results were skewed because the testers would be overtly influenced by sweetness in a quick swap test. Again, this limitation is more to do with the fast swap nature of typical ABX tests, rather than anything wrong with the ABX principle.

Yes, I agree that sighted tests are unreliable, because of expectation bias.

For what I call "Essentally Neutral Components", like cables, the most reliable test I know of is no test at all: Listen normally to your unchanged system for months, going through a range of circumstances (tired/bored/happy/sad/etc).

Then swap exactly one component and carry on listening to any of your recent heavy rotation albums - just enjoying the music.

If the new component is significantly better, the difference will hit you in the first few seconds. Any further listening is unnecessary, and will only confirm those first few seconds.

If you don't hear a difference in the first few seconds, then the new component is not for you.

If you then start swapping A with B, then you lose much of those months of "preparation" and quickly end up with all the problems of fast swapping tests.

For reasons I can't really explain, I don't appear to have an issue with expectation bias in the above test. On the basis that every single time I've been genuinely surprised with the results being different to what I had expected based on that component's price, sound signature reputation, etc.  It's possible this is a sort of complex double delusion, but the results have been so clear and against expectation that I'm willing to live with that possibility.

The above test is obviously impractical and time consuming, So most of the time, I have to make do with a reduced version, which introduced unreliability - such is life.

It doesn't follow that what works best for me will work best for anyone else.

FWIW, my view is that by finding out the different results between sighted, blind comparison and ABX tests we can then make a more informed decision about our hifi and what we need.  The wine industry has not ignored blind testing like the hifi industry has. Wine tasters and reviewers have not banned blind testing like many hifi forums have. To them it is not a dirty secret to be hidden away. Expensive brands of wine still live next to supermarket branded wines and people are able and indeed encouraged to enjoy both.

So why not do the same with hifi? Let people know the different results between sighted and the blind tests. Let people know that when sighted a £100 USB cable can sound better than a £5 one in a sighted test. But in a blind test no one can hear a difference. Then people can make up their own minds as to which one to buy. I do not believe for one minute the hifi industry will then collapse. Just as the wine industry has not collapsed. There is a place for both a £100 and a £5 USB cable.

Fact is that the arguments against blind tests are more about the results they give than the test procedure itself. That some here will accept a positive result and not a negative result from the same procedure shows how people are repared to manipulate blind testing and ignore proper sceintific process. I also see that the various theories against blind testing have not been accompanied by any evidence or experimentation. The theories also ignore the sheer number of blind tests already conducted in different places, different equipment and different people which have consistently produced the saem results.

Indeed it is odd how so many audiophiles, as bright and well educated as many clearly are, behave. As Wapiti, head fi forum member said "It is striking how in a hobby based on the creations of science, science is readily and routinely both ignored and held highly suspect."