Head-Fi.org › Forums › Equipment Forums › Sound Science › What are the arguments against double blind tests (incl. ABX)?
New Posts  All Forums:Forum Nav:

What are the arguments against double blind tests (incl. ABX)? - Page 5

post #61 of 209
Quote:
Originally Posted by Mischa23v View Post

Kay I’m going to make this extremely easy for everyone, the reason that I am aware of are:

  1. People can have bad memory and they aren’t aware of it. Or haven’t slept well and aren’t really focused, or have something on their mind that is preventing them from focusing or interfering with their memory processing.
  2. People can have ADD or any other type of diseases or disabilities that would make it even harder for them to spot difference using their memory (they might not even be aware of it).
  3. Using small number of people would increases chances of failure and using a large number of people would still increase chance of failure because there is a million reasons why a test like this could fail.
  4. People could have bad hearing and they are just not really aware of it.
  5. Even by specially picking people to minimize memory data loss and increase memory data processing you still have a higher than usual failure chance in these kind of tests.
  6. Using people who have interest in finding out results might also add bias.
  7. A test like this is great when people notice differences, but tells you nothing when people can’t. Because there is a million different reasons why this test could fail.
  8. Forcing people to depend on their short term memory just greatly increases chances of failure that it results in questionable findings if test fails.
  9. Human are just plain bad at remembering complex things and unfortunately tests that push boundaries of human memory by trying to force them to notice slight differences are unconvincing.

 

Bah I ran out of time I’ll come back and post other stuff later.

 

 

  1. All the more reason to use fast switching, which reduces the reliance on memory compared to long tests. Also all the more reason to use a large sample size
  2. All the more reason to use fast switching, which reduces the reliance on memory compared to long tests. Also all the more reason to use a large sample size
  3. It won't increase the chance of failure at all. Assume, just as an example, that 50% of the time our memory fails and 50% of the time it's correct as a rule. With one person, they'll either get it right or wrong, with 50% chance. With 10 people, on average 5 will get it right and 5 will get it wrong. It's the exact same chance. When spread out over a large enough sample size, you home in on that exact percentage, and reduce the likelihood that a lucky spree of failures or successes skews the results (as would especially be the case with one person)
  4. All the more reason to use a large sample size
  5. Higher than usual compared to what? What's the "usual" test?
  6. Not if the people are interested in finding a positive result, which is why it's so important to ABX test audiophiles that think the gear is different. Their bias cannot affect the outcome of an ABX test. Though it can very much affect the outcome of a sighted test, or even a normal preference-based blind test if they know when switches take place
  7. It tells you something when people can't. It tells you that they didn't hear a difference in that specific context. It could be due to memory or any other problems, but spread out over a large enough sample size those problems specific to that single test are made less relevant. Negative results cannot say for certain that a difference is inaudible, but enough of them compared to the number of positives can be used as evidence to support a hypothesis that the difference is inaudible
  8. Short term memory is much more accurate than long term memory, especially when it comes to sensory stimuli like sound
  9. How is an ABX test complex at all? For the participants, it's simple. Here's a sound. Here's another sound. Are they the same? The issue with memory has nothing to do with complexity, simply the physical limits of our sensory memory (which is long enough for fast switching, until you prove otherwise)
post #62 of 209

I would like to re-engage with this issue of ABX testing relying on memory, and therefore being unreliable.

 

Let's be clear here: although we are talking about "arguments against ABX testing", we are talking about them in the context of audio. ABX testing is used to test claims that, say, a cable can improve sound quality. The null hypothesis in this case is that there is no audible difference. Hopefully, we all agree on those statements. And I think we all agree that an ABX test on Coke and Pepsi would yield great results, showing that everyone can taste a difference between them.

 

Now, let's think about it a bit. Imagine a scenario where someone has said "the sound quality is better with cable X". Obviously at the time they say it, they're saying it based on memory. But maybe it is a statement based on the memory of knowing it was better at the time. So, it's not that they're remembering the two different sounds, comparing them, and then deciding that brand X was better - they're remembering that at the time of listening, while they could actually hear the music, that they knew it was better, right then at that time. Fair enough. Maybe they're listening to it right now, one cable leading to either ear, and saying it's better. This, supposedly, eliminates memory from the equation completely.

 

However. Think about listening to music. How much of listening to it does not involve your memory? Put some on now and get 30 seconds into the song. Sound good? Sure! Great! Now, what is it that sounds good? Is it what you're listening to right now? Now? There is no "now" when it comes to listening to music. "Now" is an infinitely short amount of time, and at any given instant, any snapshot in time, you are hearing nothing - your eardrum is not in motion. Sound is a change in air pressure over time. In order to experience sound, memory is absolutely necessary. You cannot hear without it. So absolutely everything you ever hear is reliant on short term memory. To be a bit less picky, imagine that at any given moment you are hearing a kind of snapshot of the sound. It's probably more accurate as your brain is busy perceiving it. However, it's an instant snapshot of the sound and it's certainly not what we imagine when we think about music. It would probably just sound like an annoying noise if you stretched it out.

 

What I'm saying, in my long-winded way, is that arguing against ABX because it relies on memory is probably a dead end, because your whole experience of sound, everything you ever think about the sound you're hearing, is already in the past. Sound is memory.

post #63 of 209
Thread Starter 
Btw, even with one sound going to one ear and another going to the other ear, I don't think you can focus on both sounds at the same time. Just like your sight, you're gonna have to focus on one sound, then on the other, and make a comparison based on your short-term memory. Not only that, but AFAIK you can only focus on one specific part of the sound at a time. I guess our brain is to blame?
post #64 of 209
Quote:
Originally Posted by skamp View Post

Btw, even with one sound going to one ear and another going to the other ear, I don't think you can focus on both sounds at the same time. Just like your sight, you're gonna have to focus on one sound, then on the other, and make a comparison based on your short-term memory. Not only that, but AFAIK you can only focus on one specific part of the sound at a time. I guess our brain is to blame?


Yes, our brains suck. Which begs the question: If we can't hear it in an ABX test even though theoretically we should be able to, does it matter? The difference must be small enough that we wouldn't notice it during casual listening either, or at least not realize that it's different.

post #65 of 209

 

Yes you’re absolutely correct, but main problem is that people will say that the difference is so obvious it’s easy for you to notice it. No one will complain about you noticing a new chair in room for example or a new hairdo.

 

Quote:

Originally Posted by skamp View Post

Mischa, I'm still not sure where you're getting at. Surely when people claim system B sounds better than system A, they're relying on their memory, don't you agree?


 

post #66 of 209

correct, except with he part.

 

Quote:
Originally Posted by liamstrain View Post

I think his point is not falling in line with the subjectivists, but rather the opposite, - I think he is saying that even DBT and A/B/X is still too subjective and flawed, and more objective testing methods are needed. 



 

post #67 of 209


you're on my top 3 ignore list and I'm not going to read anything you type so don't bother. if you want to know why then:

 

Quote:
Originally Posted by Head Injury View Post

Oh, and I sort of didn't read the rest of your post because I lost all respect for you. But if you assure me there's something good in there, I'll do it wink.gif

 

 


 

 

post #68 of 209

I wish people would stop putting down our brains all the time. Our brains are the greatest, most wonderful things in our known universe. Capable of the most incredible feats of detection, like differences between, erm, cables for example. And they are brilliant enough to be forever compensating for all sorts of things to make our life more bearable.

It’s this automatic compensation ability that also makes them incredibly stupid, capable of the most appalling errors of detection, like experts failing to spot the difference between whisky and cognac, when the two products are clearly made from completely different components.

 

Short term, fast switching tests are great for differentiating between certain kinds of sound differences, particularly relating to frequency and gross distortion, as you get with headphones/loudspeakers. And with gross volume (say 3db+).  The taste analogy here would be sweet/sour/sharp etc. So Coke and Pepsi can be largely differentiated in a DBT because one is sweeter than the other. And even dumbos can successfully DBT a high tannin red wine v a white wine.

 

But as soon as you take away those gross differences, then long term tests start taking the lead. Like whisky/cognac, or the same-label wine in 2 different vintages.  Here, the short term test actually works against you, because the brain is trying to make two similar things identical, or is confusing itself by thinking that the repeats of the same thing are actually different. Which quickly brings in stress, which is the end.

 

Back to sound. Cables and DACs, and to a lesser extent, amps, have ruler flat frequency responses, negligible distortion and, when level matched, are all very similar in one sense. Similar enough to completely confuse our brains in a DBT. But they have different kind of differences, which are harder to pin down, and are more easily expressed in terms of “sound stage”, “space around the images” etc.  

 

So, the case for DBT is that it removes expectation bias. This is undeniable.

 

And the case against DBT (of the fast switch variety) is that it is very unreliable at differentiating certain kinds of subtle differences. It’s not the fault of the DBT, or of sighted tests. In BOTH cases, it’s our minds that fail. No test is perfect, but judging from personal experience only, I maintain that I can overcome the limitations of sighted long term tests more successfully than I can overcome the subtlety crippling limitations of DBT.  

 

There is another case against DBT, which is most likely to crop up shortly.

post #69 of 209

I really like your posts so far, thank you for posting your concerns.  Well the main problem is that memory isn’t really a snap shot, it doesn’t work that way. Without going into too much detail memory is more like a description that your brain writes and retrieves later.  Furthermore, this ability memory in which brain writes a description of past varies considerably among humans.

 

In the brave heart movie, there is actually an airplane landing in the movie. I would say maybe 10% of people who watched movie actually was shocked about it. Most people didn’t even notice airplane because who cares it’s just an airplane (even though there shouldn’t have been airplanes at time). So when they were asked later about it, it wasn’t really important at time so their brains didn’t record it in memory. But most people who are told about it and watch it again go “what the hell yes I remember that how did I forget that or not notice it at time”

 

Some people can remember everything in their lives and have almost what they call a perfect memory, but for most human even if one second passes, the best their brain can do is retrieve about 50% to 60% of a description of a complex snapshot.

So the big question is, how can you tell if you actually noticed a change but you just can’t remember it? because your brain didn’t think it was important at time. Since:

  1. It’s not an accurate snapshot.
  2. You’re expecting to hear the something most of time because it’s same audio file. So what your brain does is focus on stuff that it didn’t record and ignores stuff that it already recorded because it feels that stuff already recorded isn’t as important.
  3. Even short term human memory is actually bad in most humans.

 

 

Quote:
Originally Posted by jumblejumble View Post

I would like to re-engage with this issue of ABX testing relying on memory, and therefore being unreliable.

 

Let's be clear here: although we are talking about "arguments against ABX testing", we are talking about them in the context of audio. ABX testing is used to test claims that, say, a cable can improve sound quality. The null hypothesis in this case is that there is no audible difference. Hopefully, we all agree on those statements. And I think we all agree that an ABX test on Coke and Pepsi would yield great results, showing that everyone can taste a difference between them.

 

Now, let's think about it a bit. Imagine a scenario where someone has said "the sound quality is better with cable X". Obviously at the time they say it, they're saying it based on memory. But maybe it is a statement based on the memory of knowing it was better at the time. So, it's not that they're remembering the two different sounds, comparing them, and then deciding that brand X was better - they're remembering that at the time of listening, while they could actually hear the music, that they knew it was better, right then at that time. Fair enough. Maybe they're listening to it right now, one cable leading to either ear, and saying it's better. This, supposedly, eliminates memory from the equation completely.

 

However. Think about listening to music. How much of listening to it does not involve your memory? Put some on now and get 30 seconds into the song. Sound good? Sure! Great! Now, what is it that sounds good? Is it what you're listening to right now? Now? There is no "now" when it comes to listening to music. "Now" is an infinitely short amount of time, and at any given instant, any snapshot in time, you are hearing nothing - your eardrum is not in motion. Sound is a change in air pressure over time. In order to experience sound, memory is absolutely necessary. You cannot hear without it. So absolutely everything you ever hear is reliant on short term memory. To be a bit less picky, imagine that at any given moment you are hearing a kind of snapshot of the sound. It's probably more accurate as your brain is busy perceiving it. However, it's an instant snapshot of the sound and it's certainly not what we imagine when we think about music. It would probably just sound like an annoying noise if you stretched it out.

 

What I'm saying, in my long-winded way, is that arguing against ABX because it relies on memory is probably a dead end, because your whole experience of sound, everything you ever think about the sound you're hearing, is already in the past. Sound is memory.



 

post #70 of 209

All I’m saying is that out brain is limited, and because it’s limited we can’t expect our brain to remember and notice slight difference we hear between 2 audio files when there is maybe a 10% to 20% gain. Maybe hearing from one ear and then the other is just as bad, but still I’m saying that study is flawed and even though the results show that there is no difference. We can’t be absolutely certain of that.


 

Quote:
Originally Posted by skamp View Post

Btw, even with one sound going to one ear and another going to the other ear, I don't think you can focus on both sounds at the same time. Just like your sight, you're gonna have to focus on one sound, then on the other, and make a comparison based on your short-term memory. Not only that, but AFAIK you can only focus on one specific part of the sound at a time. I guess our brain is to blame?


 

post #71 of 209
Thread Starter 
Quote:
Originally Posted by TheAttorney View Post

And the case against DBT (of the fast switch variety) is that it is very unreliable at differentiating certain kinds of subtle differences.
I can tell from my experience in ABXing Ogg Vorbis that it's not true. The differences that I detected where very subtle, and I detected them reliably (like 15 times out of 16).
Quote:
Originally Posted by TheAttorney View Post

I maintain that I can overcome the limitations of sighted long term tests more successfully than I can overcome the subtlety crippling limitations of DBT.

I highly doubt that overcoming sighted bias is more easy than running an ABX test. TBH I'm under the impression that ABX detractors are just too scared of failing the test. I'm not scared because I have successfully passed ABX tests, when there was indeed a difference to be heard, however subtle. I have no problem with being unable to hear a difference.
post #72 of 209
Quote:
Originally Posted by TheAttorney View Post

I wish people would stop putting down our brains all the time. Our brains are the greatest, most wonderful things in our known universe. Capable of the most incredible feats of detection, like differences between, erm, cables for example. 

...

 

Back to sound. Cables and DACs, and to a lesser extent, amps, have ruler flat frequency responses, negligible distortion and, when level matched, are all very similar in one sense. Similar enough to completely confuse our brains in a DBT. But they have different kind of differences, which are harder to pin down, and are more easily expressed in terms of “sound stage”, “space around the images” etc.  


 

There has not been shown to actually be a difference in cables. So it is no surprise that they are not detected in ABX testing. This is not a limitation of ABX testing, it is a flaw in the assertion that there is an audible difference to be detected. Dacs and Amps are a different beast, and for different reasons. But detecting subtleties in presentation (and sound-stage) is actually something ABX testing is quite good at showing. When they are there to be heard. 

 

 

 

Quote:
There is another case against DBT, which is most likely to crop up shortly.

 

 

Curious now. 

 

 

post #73 of 209
Quote:
Originally Posted by Mischa23v View Post

you're on my top 3 ignore list and I'm not going to read anything you type so don't bother. if you want to know why then:


Why, because I point out flaws in your logic? If you ignored everyone who did that, you wouldn't see many posts down here.

post #74 of 209

(I cannot get the original post to quote, so have copied and pasted instead)

 

Reply to Mischa23v

 

 

"You didn’t even bother to read what everyone wrote here; you could care less. You state your opinions like they were facts. But it doesn’t matter right? This isn’t the sound science thread right? I’m pretty sure after reading your post that this is the unscientific sound thread.

 

This is where I got my information from http://www.head-fi.org/t/486598/testing-audiophile-claims-and-myths Read through the first post and you will see no connection between how quickly switching was done and the result. Instead there is total consistency between the type of test, either ABX or blind comparison and the result.

 

Quote:
Originally Posted by Prog Rock Man View Post

It does not affect the results of ABX or blind comparison tests whether you listen to a system you are familiar with, not familiar with, with short pieces of music or long, in your own house or elsewhere. I have read loads of ABX and blind comparison tests and whatever the situation of the tests you get the same result. With ABX no one can tell the difference, with blind comparison the brand name, price etc no longer has an affect on the result. Cheap, no brand is just as likely to win as expensive high end brand.

 

 


let me get this straight, the people who argue against you are retards, and the people who are with you are awesome. Just when you thought his post couldn’t get any more scientific.

 

No not at all, I promise you that is not true.


 

Quote:
Originally Posted by Prog Rock Man View Post

Those who argue against ABX and blind comparison tests are those who do not like the results of such tests. They tend to have a vested interest in hifi and either work in the industry or have expensive kit. Those who are in favour like the results, usually because they can save money by not constantly upgrading and buying expensive kit. I personally enjoy my music more now as I know the kit I have is in reality as good as it gets.

 

I think that for years now people have been misinterpreting the results of blind testing. They think that it suggests some people are deaf and others have golden ears and that sources, amps, DACs, cables can have as big an impact on sound (if not more) as speakers. Speakers are the one part of the hifi chain that does pass blind testing, where people really can hear differences when blinded.

 

 


 

Are you even sure you’re on the right thread?

 

Yes

 

Quote:

Originally Posted by Prog Rock Man View Post

 

What blind testing really shows us is that sight and knowledge of brands and prices has a profound impact on perceived sound quality, that there is no such thing as golden ears, so long as your source etc is built to a decent working ability it will do as well as anything else and speakers is where you concentrate your efforts and money.

 

 

 

 

I should add that as well as speakers, codecs, bit rates and a few amps have also passed blind tests. CDPs, DACs, cables, most amps have not.

post #75 of 209
Quote:
Originally Posted by Mischa23v View Post

All I’m saying is that out brain is limited, and because it’s limited we can’t expect our brain to remember and notice slight difference we hear between 2 audio files when there is maybe a 10% to 20% gain. Maybe hearing from one ear and then the other is just as bad, but still I’m saying that study is flawed and even though the results show that there is no difference. We can’t be absolutely certain of that.


 



 




If our audio memory is that bad, how can we manage to audition hifi products? How do you know if any hifi product is different from any other equivalent? I think that we do have a pretty good audio memory since for example we can recognise different voices over the phone and I can remember the difference between the same music played on my car stereo, on the radio or with my hifi, because the differences are clear.

 

If you then reduce the differences it is not so much they are harder to remember, it is more of a case that they are harder to hear, to a point that there is no audible difference. So if you supposedly cannot remember the difference between two pieces of music played on different hifi during a blind test, I think that instead you are finding that in fact there is no audible difference. 

New Posts  All Forums:Forum Nav:
  Return Home
  Back to Forum: Sound Science
Head-Fi.org › Forums › Equipment Forums › Sound Science › What are the arguments against double blind tests (incl. ABX)?