What are the arguments against double blind tests (incl. ABX)? - Page 4

Quote:

Originally Posted by Mischa23v 

If any scientist declares that he/she the results of a test depending on human memory is absolute fact. Almost every engineer and scientist out there would laugh at him or her and would probably think very little of them. Yet, people on this site keep insisting that using short term memory or long term memory is still a valid scientific approach. I really have nothing else to add except “whatever”.

No tests absolutely prove facts, and certainly relying on memory has limitations.  But I don't see how you can claim that any tests using memory do not use a valid scientific approach.  (This is not to mention experiments about memory itself, but of course that's not what you're talking about.)  There are limitations with many different approaches, yet with the right design and controls, you can still get some valuable data.  When you work with memory, you probably want to use short-term memory and control the environment to ensure there's no potato-chip eating distractions nearby.

What other kind of comparative listening experiment do you propose?  Having a somebody listen to A in one ear and B in the other ear and express a preference for one--then swapping, and controlling for order effects, and so on?  Make somebody listen to A and B several times each without saying which is which, and score each sample based on different parameters?

You don’t seem to understand what I’m saying. First of all you can tell difference (when it’s obvious) except when it’s really close then switching back and forth makes it even more difficult because your depending on memory. There are other method’s you can do an abx tests that don’t depend on your memory. Like listening to an audio using your left and right ear separately at the same time.  There are also way more methods you can do abx tests that don’t depend on memory. Another exmaple is using technology and computers to mimic hearing of human beings.  

No I wouldn’t say that because memory is unreliable, not consisted, not accurate which is pretty much the anti-characterization of what scientific testing should be about.  

Quote:

Originally Posted by skamp 

If you can't tell the difference over the extremely short timespan of an ABX test (rapidly switching back and forth), then you can't tell the difference period. If you're right, what could possibly validate the claims that something makes a difference (if you can even tell in an ABX test)?
In other words, wouldn't you say that an ABX test (which suffers the least from memory-induced limitations) is superior than any other kind of comparison?

Switching back and forth *decreases* the reliance on memory - it offers a much closer direct comparison experience. I agree other methods are more rigorous (especially measurement/instrument testing), but this works within the known and controllable limitations. I do not think left/right ear comparisons are that useful, especially since stereo data relies on *both*. You can duplicate mono signals, but that does not approximate actual listening conditions at all. And obviously, you would have to double the number of test iterations to compensate for left/right ear hearing differences. (and honestly, if you have the same mono track playing at the same volume, are you going to be able to tell which ear heard the 2nd order harmonics? I don't think so - but that's something else worth testing).

It does not affect the results of ABX or blind comparison tests whether you listen to a system you are familiar with, not familiar with, with short pieces of music or long, in your own house or elsewhere. I have read loads of ABX and blind comparison tests and whatever the situation of the tests you get the same result. With ABX no one can tell the difference, with blind comparison the brand name, price etc no longer has an affect on the result. Cheap, no brand is just as likely to win as expensive high end brand.

Those who argue against ABX and blind comparison tests are those who do not like the results of such tests. They tend to have a vested interest in hifi and either work in the industry or have expensive kit. Those who are in favour like the results, usually because they can save money by not constantly upgrading and buying expensive kit. I personally enjoy my music more now as I know the kit I have is in reality as good as it gets.

I think that for years now people have been misinterpreting the results of blind testing. They think that it suggests some people are deaf and others have golden ears and that sources, amps, DACs, cables can have as big an impact on sound (if not more) as speakers. Speakers are the one part of the hifi chain that does pass blind testing, where people really can hear differences when blinded.

What blind testing really shows us is that sight and knowledge of brands and prices has a profound impact on perceived sound quality, that there is no such thing as golden ears, so long as your source etc is built to a decent working ability it will do as well as anything else and speakers is where you concentrate your efforts and money.

You didn’t even bother to read what everyone wrote here; you could care less. You state your opinions like they were facts. But it doesn’t matter right? This isn’t the sound science thread right? I’m pretty sure after reading your post that this is the unscientific sound thread.

Quote:

Originally Posted by Prog Rock Man

It does not affect the results of ABX or blind comparison tests whether you listen to a system you are familiar with, not familiar with, with short pieces of music or long, in your own house or elsewhere. I have read loads of ABX and blind comparison tests and whatever the situation of the tests you get the same result. With ABX no one can tell the difference, with blind comparison the brand name, price etc no longer has an affect on the result. Cheap, no brand is just as likely to win as expensive high end brand.

 

 

let me get this straight, the people who argue against you are retards, and the people who are with you are awesome. Just when you thought his post couldn’t get any more scientific.

 

Quote:

Originally Posted by Prog Rock Man

Those who argue against ABX and blind comparison tests are those who do not like the results of such tests. They tend to have a vested interest in hifi and either work in the industry or have expensive kit. Those who are in favour like the results, usually because they can save money by not constantly upgrading and buying expensive kit. I personally enjoy my music more now as I know the kit I have is in reality as good as it gets.

 

I think that for years now people have been misinterpreting the results of blind testing. They think that it suggests some people are deaf and others have golden ears and that sources, amps, DACs, cables can have as big an impact on sound (if not more) as speakers. Speakers are the one part of the hifi chain that does pass blind testing, where people really can hear differences when blinded.

 

 

 

Are you even sure you’re on the right thread?

 

Quote:

Originally Posted by Prog Rock Man

 

What blind testing really shows us is that sight and knowledge of brands and prices has a profound impact on perceived sound quality, that there is no such thing as golden ears, so long as your source etc is built to a decent working ability it will do as well as anything else and speakers is where you concentrate your efforts and money.

 

 

 

Mischa - what he has written, fundamentally agrees with what you have written. Read what he actually says. 

There should be no argument against double blind testing, but you can make an argument agains the classic double blind A/B test that a lot of people use.  The common test is to just to try component X against component Y and see who can tell a differernce in perceived quality.

The biggest flaw I see in that test process is that there is no control to see what (or if) the participants are capable of discerning.  If you are postulating that X component colors the sound, then you need to have a control where a component DOES color the sound in an agreed upon way.  Either by measurement (equalization or hardware), or agreed upon by a panel.

In statistical measurement and studies (something I do for a living) the results of a double blind test without a control would never be considered.  When you are testing for things where the measurement device, in this case the ear and brain, are variable, you absolutely must establish a variabilty metric before the results carry any meaning.  In the field this is called a MSA, Measurement Systems Analysis.  In the case of several test subjects performing double blind AB testing, I would establish a guage variability score for each one using controlled & measured changes to the sound.  Much like an audiologist does when they measure your hearing and capability in understanding the sounds that make up language.

In other words, you have to prove that the listeners are capable of detecing change, and at what precision they can detect change, before you can score results of them testing between a & b components.

Kay I’m going to make this extremely easy for everyone, the reason that I am aware of are:

1. People can have bad memory and they aren’t aware of it. Or haven’t slept well and aren’t really focused, or have something on their mind that is preventing them from focusing or interfering with their memory processing.
2. People can have ADD or any other type of diseases or disabilities that would make it even harder for them to spot difference using their memory (they might not even be aware of it).
3. Using small number of people would increases chances of failure and using a large number of people would still increase chance of failure because there is a million reasons why a test like this could fail.
4. People could have bad hearing and they are just not really aware of it.
5. Even by specially picking people to minimize memory data loss and increase memory data processing you still have a higher than usual failure chance in these kind of tests.
6. Using people who have interest in finding out results might also add bias.
7. A test like this is great when people notice differences, but tells you nothing when people can’t. Because there is a million different reasons why this test could fail.
8. Forcing people to depend on their short term memory just greatly increases chances of failure that it results in questionable findings if test fails.
9. Human are just plain bad at remembering complex things and unfortunately tests that push boundaries of human memory by trying to force them to notice slight differences are unconvincing.

Bah I ran out of time I’ll come back and post other stuff later.

Quote:

Originally Posted by skamp 


I didn't read it that way. How did you read it?

You are correct, I was misreading what Mischa was saying. What ProgRockMan has written fundamentally agrees with what most other people here have been saying. 

very interesting, thank you for sharing.
 

Quote:

Originally Posted by TWIFOSP 

There should be no argument against double blind testing, but you can make an argument agains the classic double blind A/B test that a lot of people use.  The common test is to just to try component X against component Y and see who can tell a differernce in perceived quality.

 

The biggest flaw I see in that test process is that there is no control to see what (or if) the participants are capable of discerning.  If you are postulating that X component colors the sound, then you need to have a control where a component DOES color the sound in an agreed upon way.  Either by measurement (equalization or hardware), or agreed upon by a panel.

 

In statistical measurement and studies (something I do for a living) the results of a double blind test without a control would never be considered.  When you are testing for things where the measurement device, in this case the ear and brain, are variable, you absolutely must establish a variabilty metric before the results carry any meaning.  In the field this is called a MSA, Measurement Systems Analysis.  In the case of several test subjects performing double blind AB testing, I would establish a guage variability score for each one using controlled & measured changes to the sound.  Much like an audiologist does when they measure your hearing and capability in understanding the sounds that make up language.

 

In other words, you have to prove that the listeners are capable of detecing change, and at what precision they can detect change, before you can score results of them testing between a & b components.

 

Consider me subscribed!

Frankly, I am rather skeptical of the DBT process.................YMMV.

I think his point is not falling in line with the subjectivists, but rather the opposite, - I think he is saying that even DBT and A/B/X is still too subjective and flawed, and more objective testing methods are needed.