Head-Fi.org › Forums › Equipment Forums › Sound Science › What are the arguments against double blind tests (incl. ABX)?
New Posts  All Forums:Forum Nav:

What are the arguments against double blind tests (incl. ABX)? - Page 8

post #106 of 209
Thread Starter 
Quote:
I think that we can agree that positive ABX results are more relevant, because they do prove that there is a difference that can be discerned? Are there objections to positive ABX results?
Quote:
Originally Posted by Mischa23v View Post

No because if you fail the test even though you were supposed to pass, it’s a huge flaw. Don’t you agree?

I'm sorry, I'm not sure I understand you answer: do you claim that successful ABX tests do not show with certainty that there is an audible difference?
post #107 of 209


 

Quote:
Originally Posted by jumblejumble View Post

 

If they do that, they're not scientists. This would be cherry picking or confirmation bias. However, what is the evidence that ABX doesn't work? All that is necessary to disprove the theory that it does work, is one instance of reliable scientific testing that reveals a difference not found in ABX can be reliably detected through a different method.

 

I've already given my examples, relating to countless tasting DBTs earlier in this thread. And in previous similar threads, I've even posted links to some specific results.

I class myself as an observer, not a scientist. Observers make their observations and scientists explain the results. Yes that's a cop out in a way, but here's the thing: most of the people on my side of the debate aren't that bothered about the science bit. Yes it's kinda interesting, but not so interesting as to spend the considerable effort to get truly scientific evidence. That's the part we expect scientists to play because that's what floats their boat and also they're better skilled at doing it. If that's not a good enough answer for the Sound Science forum, then I can understand that, but shame, because the debate won't get progressed very far. There are some on my side who can also explain themselves better scientifically, and who are doing a better job of it than me in this thread - but they are very much in the minority.  

 

I should also point out that there is nothing in DB ABX methodology that inherently restricts us to short term tests. There is nothing stopping someone from listening to a whole album on A, again on B, and then again on X. Or even spending a week with A, a week with B and then a week with X. The ABX method itself is not at fault here if we've decided that short term memory is no use, just the specific use of it in these cases.

 

Agreed. My issue is not with DBT or ABX in principle. It's with the use of structured, formal, fast switching tests for comparing certain kinds of sound (and taste) differences. As already explained in my earlier posts - they weren't that long ago. I just don't seem to be getting my points across here.


Finally, I have yet to see an instance of a subjectivist, when presented with a positive result from an ABX test (eg speakers perhaps), rejecting the findings due to inherent unreliability in the tests.

 

I've already said that speakers and headphones are easily differentiated in ABX tests, primarily because they have a non-flat frequency response and relatively high distortions, which are the type of sound differences that are  easily identified in fast switching tests. 

 

Quote:
Originally Posted by Prog Rock Man View Post


Can you evidence scientists shooting down arguments against blind testing in flames and only remembering the evidence that makes then right? What is the overwhelming observational evidence that there are problems with blind testing and what applications are you referring to?

 

Yes, the whole point of my earlier "prediction" is that it's happening right now. Actually, this particular thread is pretty civilized compared to similar ones in the past.  

 

You have made some very strong remarks against science which I think you need to evidence them.

 

I haven't said anything at all against science. Only that some scientists don't behave very scientifically, by not taking the time to properly investigate all sides of the evidence.

 

One of the complete fails of the audio industry is to submit their claims for proper independent study, by scientists. For example, no cable maker has ever put their cables to be tested at a university engineering department to validate their claims about enhanced sound quality.

 

True. And the reason that they get away with it is because most of their customers don't really care about the science bit.

However, I did start a thread once about Acuity and two cable manufacturers, where exactly this was intended. Sadly, after the hype, the initial results were disappointingly amateurish, at least as far as the sound scientists here were concerned. That group still publishes results I think.

 

There is no flaw in a properly conducted blind test. It is a case that many audiophiles do not know what to do with the results of those tests, especially since they are contrary to what they expected/hoped for.

 

You're just following my prediction again.

BTW, I'm not trying to say that one side is all good and the other all bad. Just that some types of tests are unreliable for certain types of sound (and taste) differences.

 

 


 

 

post #108 of 209
Quote:
Originally Posted by skamp View Post


I think we can agree that ABX tests prove only one thing, and only when they're successful: that there was a difference to be heard. A failed test doesn't really prove anything indeed. While I failed ABX tests with higher bitrate Ogg Vorbis files, I might succeed later on, when I have better training.
As the thread starter, let me go back to my main point: a failed ABX test is not proof of anything, but when I see claims of a difference, I expect measurements, and with claims of audible differences, I expect ABX logs. Do you find that unreasonable? I mean, I need some kind of proof, because more often than not, I will not be able to perceive claimed differences (like "256kbps AAC sounds like crap"). Do you have anything better than ABX tests that would provide such proof?


I am sorry, but that is nonsense. Blind tests that are positive show an audible difference but blind tests that are negative do not show an inaudible difference?! confused_face_2.gif

 

For you to dismiss one set of results and accept another from exactly the same test procedure is very bad science indeed.

post #109 of 209

 

Nope I only have problem with them when they fail, not when they pass.

 

Quote:

Originally Posted by skamp View Post


I'm sorry, I'm not sure I understand you answer: do you claim that successful ABX tests do not show with certainty that there is an audible difference?


 

post #110 of 209

"I think we can agree that ABX tests prove only one thing, and only when they're successful: that there was a difference to be heard. A failed test doesn't really prove anything indeed. While I failed ABX tests with higher bitrate Ogg Vorbis files, I might succeed later on, when I have better training."

 

yes sir, that's the flaw that I wanted to point out. I agree.

 

"Do you find that unreasonable? I mean, I need some kind of proof, because more often than not, I will not be able to perceive claimed differences (like "256kbps AAC sounds like crap"). Do you have anything better than ABX tests that would provide such proof?"

 

I find that to be very reasonable. However, I do not know of any other tests that might yield better results. Maybe someone else in this thread can share their thoughts on this matter.

 

 

 

 

 

 

Quote:

Originally Posted by skamp View Post


I think we can agree that ABX tests prove only one thing, and only when they're successful: that there was a difference to be heard. A failed test doesn't really prove anything indeed. While I failed ABX tests with higher bitrate Ogg Vorbis files, I might succeed later on, when I have better training.
As the thread starter, let me go back to my main point: a failed ABX test is not proof of anything, but when I see claims of a difference, I expect measurements, and with claims of audible differences, I expect ABX logs. Do you find that unreasonable? I mean, I need some kind of proof, because more often than not, I will not be able to perceive claimed differences (like "256kbps AAC sounds like crap"). Do you have anything better than ABX tests that would provide such proof?


 

post #111 of 209
Thread Starter 
Quote:
Originally Posted by Prog Rock Man View Post

I am sorry, but that is nonsense. Blind tests that are positive show an audible difference but blind tests that are negative do not show an inaudible difference?!

I went a little too fast. A failed ABX test proves the person being tested did not hear a difference at that particular time, in that particular setting (if they were of good faith). But it doesn't prove that there isn't a difference to be heard. Someone else might hear it. While some people have proven to be able to ABX 320kbps MP3s with so-called "killer" samples, I'm not sure I would succeed as well, because I have very little training, and don't know what to listen for. My failing the test wouldn't prove much, while their success would undeniably prove there is an audible difference.

In other words: a successful ABX test proves someone (not anyone) can hear a difference, while a failed ABX test does NOT prove no-one can hear a difference. Does that satisfy you?

Edit: my failing a test, while not proving that there isn't, generally speaking, an audible difference, would be valuable information to me, as far as spending money in equipment or choosing a bitrate for lossy encoding go, for instance. And since I'm a pretty normal guy, with normal hearing, basic training and arguably decent gear, I choose to take into account ABX results from other normal people with normal hearing, basic training and decent gear, because I can't just buy everything, see for myself, and return everything I've bought that didn't result in a positive outcome.
Edited by skamp - 3/16/12 at 10:20am
post #112 of 209
Quote:
Originally Posted by Prog Rock Man View Post



I am sorry, but that is nonsense. Blind tests that are positive show an audible difference but blind tests that are negative do not show an inaudible difference?! confused_face_2.gif

 

For you to dismiss one set of results and accept another from exactly the same test procedure is very bad science indeed.



In a way he is right.. you cannot really prove a negative in this way. All you can say is that the test did not show the positive, and the null hypothesis is supported. Even with a hundred thousand iterations with the same results, all we can say is that we have increasing evidence supporting the null hypothesis, but there is always that sliver of uncertainty. But now we are into semantics. 


Edited by liamstrain - 3/16/12 at 10:10am
post #113 of 209
Quote:
Originally Posted by Mischa23v View Post


Nope I only have problem with them when they fail, not when they pass.

 

 

Fine, please now example actual blind tests which were not passed and show what your problem was with that test. You have a theory, I think you should now start to evidence it.

post #114 of 209
Quote:
Originally Posted by skamp View Post


I went a little too fast. A failed ABX test proves the person being tested did not hear a difference at that particular time, in that particular setting (if they were of good faith). But it doesn't prove that there isn't a difference to be heard. Someone else might hear it. While some people have proven to be able to ABX 320kbps MP3s with so-called "killer" samples, I'm not sure I would succeed as well, because I have very little training, and don't know what to listen for. My failing the test wouldn't prove much, while their success would undeniably prove there is an audible difference.
In other words: a successful ABX test proves someone (not anyone) can hear a difference, while a failed ABX test does NOT prove no-one can hear a difference. Does that satisfy you?


By that logic a passed blind test is no guaranteee of audibility and so such results are also worthless.Yes one failed blind test will not prove no one can hear a difference. But it also does not rule out that no one can hear a difference. So how many failed blind tests will you need before you would be prepared to accept a result of there is no audible difference?

 

post #115 of 209
Thread Starter 
Quote:
Originally Posted by TheAttorney View Post

Only that some scientists don't behave very scientifically, by not taking the time to properly investigate all sides of the evidence.

We're not scientists, but I believe that's what we're doing by debating the merits of DBT.
Quote:
Originally Posted by TheAttorney View Post

some types of tests are unreliable for certain types of sound (and taste) differences

Do you agree that successful ABX tests are reliable, and that sighted tests are unreliable? And again, do you have anything better in mind?
post #116 of 209
Thread Starter 
Quote:
Originally Posted by Prog Rock Man View Post

By that logic a passed blind test is no guaranteee of audibility and so such results are also worthless.

I disagree. They guarantee that there IS an audible, tangible difference, if only to one person. If someone hears it, I might hear it as well. It's useful information.
Quote:
Originally Posted by Prog Rock Man View Post

So how many failed blind tests will you need before you would be prepared to accept a result of there is no audible difference?
I have to gauge the likelihood that I will either pass or fail the test, based on how many passed the test, how many people failed the test, who those people were and how applicable their testing conditions are to mine. Yes, if there is no successful test, and only failed ones, I will assume that I would fail it as well. If we're talking about a product, I won't buy it, and if we're talking about the bitrate of a perceptual lossy codec, I will use it.
post #117 of 209

Quote:

Originally Posted by liamstrain View Post

In a way he is right.. you cannot really prove a negative in this way.


For that reason, ABX testing is mainly a tool for proving that a difference does exist, i.e. the burden of proof should be on those who claim there is an audible difference, and they can use reliable positive results (which do not even have to be in a majority) to back it up.

The "blindness" of the test is a necessary condition to exclude biases that are already proven to exist.

 

post #118 of 209
Quote:
Originally Posted by skamp View Post


I disagree. They guarantee that there IS an audible, tangible difference, if only to one person. If someone hears it, I might hear it as well. It's useful information.
I have to gauge the likelihood that I will either pass or fail the test, based on how many passed the test, how many people failed the test, who those people were and how applicable their testing conditions are to mine. Yes, if there is no successful test, and only failed ones, I will assume that I would fail it as well. If we're talking about a product, I won't buy it, and if we're talking about the bitrate of a perceptual lossy codec, I will use it.


If one person hearing a difference is a guarantee of audibility, then using the same logic one person not hearing a difference becomes a guarantee of inaudibility. Both claims are in fact logical fallicies. If it is useful to know that one person heard a difference, it must also be useful information that someone else did not hear a difference, correct or not? Then what if lots and lots of people have not heard a difference and no one has, that must also be useful information, correct or not?

 

My evidence regarding blind testing is here for failed tests

 

http://www.head-fi.org/t/486598/testing-audiophile-claims-and-myths

 

and here for passed tests

 

http://www.head-fi.org/t/513481/are-blind-tests-bogus-examples-of-blind-tests-with-positive-results

 

So you can read both and judge for yourself what hifi products affect audibility and what do not.

 

Please also, with reference to specific tests show me how the positive ones are more valid than the negative ones in terms of their results.

 


Edited by Prog Rock Man - 3/16/12 at 11:13am
post #119 of 209
Thread Starter 
Is there any other type of tests that would undeniably prove the existence of audible differences? One that DBT/ABX detractors would approve of, hopefully?
post #120 of 209

I agree, that’s the point that I have been trying to make form the beginning.


 

Quote:
Originally Posted by stv014 View Post

Quote:


For that reason, ABX testing is mainly a tool for proving that a difference does exist, i.e. the burden of proof should be on those who claim there is an audible difference, and they can use reliable positive results (which do not even have to be in a majority) to back it up.

The "blindness" of the test is a necessary condition to exclude biases that are already proven to exist.

 



 

New Posts  All Forums:Forum Nav:
  Return Home
  Back to Forum: Sound Science
Head-Fi.org › Forums › Equipment Forums › Sound Science › What are the arguments against double blind tests (incl. ABX)?