Head-Fi.org › Forums › Equipment Forums › Sound Science › What are the arguments against double blind tests (incl. ABX)?
New Posts  All Forums:Forum Nav:

What are the arguments against double blind tests (incl. ABX)? - Page 6

post #76 of 209

Dear Prog Rock Man,     

       I’m terribly sorry if I misunderstood your earlier post. Thank you for taking time to reply to my post.

 

However, I would like to ask you this question. How can you be 100% sure that the reason people cannot tell difference between 2 cables is because they cannot hear it and not for other reasons? What is the scientific proof that you have that proves that’s it’s a hearing limitation and not any other reason? Not a memory limitation for example? Why’re you so certain? I’m just wondering if you’re aware of some facts that I might not be. Thanks and again I apologize if I misunderstood you first time.

 

Quote:

Originally Posted by Prog Rock Man View Post

(I cannot get the original post to quote, so have copied and pasted instead)

 

Reply to Mischa23v

 

 

"You didn’t even bother to read what everyone wrote here; you could care less. You state your opinions like they were facts. But it doesn’t matter right? This isn’t the sound science thread right? I’m pretty sure after reading your post that this is the unscientific sound thread.

 

This is where I got my information from http://www.head-fi.org/t/486598/testing-audiophile-claims-and-myths Read through the first post and you will see no connection between how quickly switching was done and the result. Instead there is total consistency between the type of test, either ABX or blind comparison and the result.

 


let me get this straight, the people who argue against you are retards, and the people who are with you are awesome. Just when you thought his post couldn’t get any more scientific.

 

No not at all, I promise you that is not true.


 


 

Are you even sure you’re on the right thread?

 

Yes

 

 

I should add that as well as speakers, codecs, bit rates and a few amps have also passed blind tests. CDPs, DACs, cables, most amps have not.



 

 

post #77 of 209

 

 

Quote:
How can you be 100% sure that the reason people cannot tell difference between 2 cables is because they cannot hear it and not for other reasons? What is the scientific proof that you have that proves that’s it’s a hearing limitation and not any other reason? Not a memory limitation for example? Why’re you so certain?

 

If I can throw in my $.02. 

 

Short answer, we cannot be 100% sure. But in the end, if no change is detectable, it doesn't matter why not. 

 

Long answer. If ABX or DBT were the *only* tests available and no other measures were at all in play, you might have a point. But they are not in a vacuum. We have reams of theory, mathematics, electrical measurements, human audiotory experience, etc. which also support the inaudibility. Memory may well be a factor, but we can infer from our other observations and experience, that in this case, there was not a difference either significant enough to be heard, or there at all.

 

A well run ABX or DBT is one piece of data in a whole mess of other data. Not proof of anything in and of itself. And by the same token, you would need to provide your own data to show that memory was a more significant factor, in order to overturn the other historical and testable information that build our current understanding of how and why we hear what we do. 


Edited by liamstrain - 3/15/12 at 3:14pm
post #78 of 209

Things are not black and white, it’s not either good or bad, we have a degree of things we can remember and notice and things we can’t notice. I’m talking about very very slight difference that are very hard to notice, not things that you can easily spot like telling difference between different voices during a phone call. The more difficult the task is the harder it is for you to tell the difference. I think we can both agree on that right? I mean if you have to tell difference between 2 young twin boys it’s going to be hard right?

 

You don’t seem to understand I agree that there is no audible difference. I’m just saying that the test is flawed because it’s hard to hear and notice the differences (because they’re very slight differences). Furthermore, having to notice the difference using your short term memory makes things even harder. On an already hard test  Can you prove to me that the test isn’t too difficult for people to pass and that’s why they failed?  Don't you sometimes forget what you're about to say even though your in the middle of saying it?

 


 

Quote:
Originally Posted by Prog Rock Man View Post




If our audio memory is that bad, how can we manage to audition hifi products? How do you know if any hifi product is different from any other equivalent? I think that we do have a pretty good audio memory since for example we can recognise different voices over the phone and I can remember the difference between the same music played on my car stereo, on the radio or with my hifi, because the differences are clear.

 

If you then reduce the differences it is not so much they are harder to remember, it is more of a case that they are harder to hear, to a point that there is no audible difference. So if you supposedly cannot remember the difference between two pieces of music played on different hifi during a blind test, I think that instead you are finding that in fact there is no audible difference. 



 

 

post #79 of 209

Oh no I don’t believe that there is a difference, or that will notice a difference if we do other tests. I’m just pointing out a major flaw in abx and blind tests in the way they’re being conducted.

 

"if no change is detectable, it doesn't matter why not", if the reason that it isn't detectable is because of a problem in the way the test is being conducted then it does matter why.

 

Because that in itself proves that even though people cannot tell the difference, it’s not because they cannot hear it.  Assume that I’m right that people’s memory is so bad that they cannot recall hearing the differences in subtle changes. Then arguably they might be able to tell the difference if they listen to it using 2 different ears at the same time. Not to mention the absolute joy a lot of people will feel, by knowing that they were right the whole time. Assuming no other tests are available of course.

 

 

Quote:

Originally Posted by liamstrain View Post

 

 

 

If I can throw in my $.02. 

 

Short answer, we cannot be 100% sure. But in the end, if no change is detectable, it doesn't matter why not. 

 

Long answer. If ABX or DBT were the *only* tests available and no other measures were at all in play, you might have a point. But they are not in a vacuum. We have reams of theory, mathematics, electrical measurements, human audiotory experience, etc. which also support the inaudibility. Memory may well be a factor, but we can infer from our other observations and experience, that in this case, there was not a difference either significant enough to be heard, or there at all.

 

A well run ABX or DBT is one piece of data in a whole mess of other data. Not proof of anything in and of itself. And by the same token, you would need to provide your own data to show that memory was a more significant factor, in order to overturn the other historical and testable information that build our current understanding of how and why we hear what we do. 



 

post #80 of 209

Just to make it clear to everyone, I’m not saying that there is a difference between cables or claiming anything. I’m just pointing out a major flaw in the arguments against abx and blind tests. Because that’s what this thread is about right?

 

My point is that even though people cannot tell the difference, it could be reasons other than human hearing limitations because of the way the test is being conducted (hearing loss, bad memory, etc). Thus, even though the test shows that people cannot tell the difference. We cannot be 100% sure that it’s because of a human hearing limitation and not one of other things that might cause people to not be able to notice differences. Also, because of the way the test is being conducted, we’re making an already difficult task even harder by using short term memory as a means to test if people can notice subtle differences.

That’s pretty much it.

 

Oh and because we cannot measure or detect if a person is suited to take this test, the chances of failure are higher than chances of success in a test like this. Because let’s be honest the things that can affect human hearing or concentration, or short term memory are almost infinite. Human aren’t perfect, and the diseases and disabilities and circumstances like environment a person lives in, the mood he or she is in, there are just way too many things the can result in people failing this test. Thus when people fail to notice changes in the test, it’s useless because the test actually tells us absolutely nothing important. We have no idea why they failed ot notice changes.

 

 


Edited by Mischa23v - 3/15/12 at 4:15pm
post #81 of 209

While you have presented a theoretical flaw in abx testing. You need to present evidence that the memory issues you raise, actually would in practice, invalidate the tests or even have a measurable effect on the results. There is a lot of evidence to support these testing methods as useful within their known limitations. You need more than your theory and assertions to overcome their use, or to redefine their scope.

 

It is an interesting argument. But you have not demonstrated anything that would indicate we need to adjust our methodology. Do you have data, for instance about what threshold of audibility is that would make abx less reliable? 

 

 

 

Quote:
Also, because of the way the test is being conducted, we’re making an already difficult task even harder by using short term memory as a means to test if people can notice subtle differences.

 

 

No, as has been stated before, our experience has been that abx makes the differences easier to identify (by providing immediate contrast in stereo under realistic listening conditions - short term memory not-withstanding). Not harder. Especially with large enough samples to provide a measure of error correction (as most good DBT/ABX data should be).

 

You have stated repeatedly otherwise, do you have experience or data to back that assertion up? 

 

 

 

Quote:
We cannot be 100% sure that it’s because of a human hearing limitation and not one of other things that might cause people to not be able to notice differences.

 

 

No, we can never be 100% sure of anything. But I'm pretty comfortable with statistical significance and high percentages of confidence. Especially when corroborated by different methodologies and data sets. 


Edited by liamstrain - 3/15/12 at 7:45pm
post #82 of 209

Just stumbled across this thread, and here's some food for thought.

 

Although all these quantitative measurements and tests seem to prove that cables do not make audible differences in a technical sense, therefore implying that there is no point in buying aftermarket cables, if there actually are perceivable differences for the believers, whether it be placebo, bias, or purchase justification, wouldn't that mean that there are technically audible differences and the purchases were justified since the buyers had heard the improvement, ignoring the scientifically proven fact that there actually were none? For example, a person truly believed the room became hotter, and he could feel it in his skin, but the room actually did not, of course the room temperature scientifically did not change, but to the person, it actually did. Yes, it might have been caused by some other factor, but if we keep it simple, we can conclude that there actually was a change, at least to the said person. Therefore, both sides are correct: no, there is no audible difference, but yes at the same time because a difference to the person was heard.

 

What do you think? (Obviously all you wonderful one-dimensional people will shoot my post down, but the rest of you? Think about it.)

 

 

My argument against DBT:

 

It's not necessary. The ones who don't believe in cables will find DBT useful for scientific experiments. Those who do believe in cables, are not required to prove to anyone anything, it's their enjoyment and their ears. Why go through all the trouble just for the sake of seeing others fall? Let them be with their ears, and you go enjoy yourself with your stock cable. I don't see the point in arguing; there is no argument...


Edited by Girls Generation - 3/16/12 at 12:51am
post #83 of 209
Quote:
Originally Posted by Girls Generation View Post

Just stumbled across this thread, and here's some food for thought.

 

Although all these quantitative measurements and tests seem to prove that cables do not make audible differences in a technical sense, therefore implying that there is no point in buying aftermarket cables, if there actually are perceivable differences for the believers, whether it be placebo, bias, or purchase justification, wouldn't that mean that there are technically audible differences and the purchases were justified since the buyers had heard the improvement, ignoring the scientifically proven fact that there actually were none? For example, a person truly believed the room became hotter, and he could feel it in his skin, but the room actually did not, of course the room temperature scientifically did not change, but to the person, it actually did. Yes, it might have been caused by some other factor, but if we keep it simple, we can conclude that there actually was a change, at least to the said person. Therefore, both sides are correct: no, there is no audible difference, but yes at the same time because a difference to the person was heard.

 

What do you think? (Obviously all you wonderful one-dimensional people will shoot my post down, but the rest of you? Think about it.)

 

 

My argument against DBT:

 

It's not necessary. The ones who don't believe in cables will find DBT useful for scientific experiments. Those who do believe in cables, are not required to prove to anyone anything, it's their enjoyment and their ears. Why go through all the trouble just for the sake of seeing others fall? Let them be with their ears, and you go enjoy yourself with your stock cable. I don't see the point in arguing; there is no argument...


If you're willing to spend money on an illusion, you're welcome to it.

 

What most of us take issue with is when those who hear the illusory differences between cables start to claim that the differences are very much real, and applicable to others. If one is to accept that what they perceive matters more than measurements and fact, they must also accept that their perceptions are their own and only their own. They cannot be applied to others, and therefore should not be used to sell products. Which means no more reviews (the act of reviewing a cable is itself unbelievable to me), and no more "You should buy this cable because it really opens up the sound". This should all be replaced with "I bought cable X and it changed how I perceive my audio, but I cannot explain why so proceed with caution."

 

Audiophiles rarely do this. You occasionally get the true subjectivist like Steve Eddy who actually knows what he's talking about and will argue that cables don't make a real difference, even though he makes, sells, and uses fancy cables. Mostly, you just get pseudo-objectivists who say things like "I hear it, therefore it is real. And here is a bunch of unrelated sciencey-sounding nonsense that I'm going to claim support my beliefs, while I ignore all evidence against and continue to spout my own perceptions as universal yet unsubstantiated objective truth." I'm paraphrasing.

 

It always saddens me when someone claims a scientific pursuit is not necessary just because they disagree with its results.

 

P.S. It is 4 a.m. and I am so lucid right now it's like I can see through walls. I think there's two crickets making love on the roof of my neighbor's house, and they cries they are making hurt me. I'm going to bed.

post #84 of 209
Thread Starter 
Quote:
Originally Posted by Head Injury View Post

What most of us take issue with is when those who hear the illusory differences between cables start to claim that the differences are very much real, and applicable to others. […]

No more "You should buy this cable because it really opens up the sound". […]

Mostly, you just get pseudo-objectivists who say things like "I hear it, therefore it is real. And here is a bunch of unrelated sciencey-sounding nonsense that I'm going to claim support my beliefs, while I ignore all evidence against and continue to spout my own perceptions as universal yet unsubstantiated objective truth."

This.

I have zero problem with people enjoying their placebo. I enjoy mine myself.

I enjoy the fact that the Wolfson WM8740 DAC and the amplifier in my FiiO E7 are high quality (measurably so - where are the measurements for cables?), even though I'm pretty sure I couldn't ABX the Wolfson DAC against the Realtek DAC in my laptop. Honestly, I don't even want to try.

I also enjoy my high bitrate lossy files when on the go, even though I know for a fact (from ABXing) that I can't tell the difference from much lower bitrates. I like to know that I'm listening to what I consider to be a high fidelity sound system.

What I have a problem with is people (and worse, corporations, who actually have a stake in it) trying to get others to spend unreasonable amounts of money in unnecessary equipment, while providing non-sensical proof laced with pseudo-science to back their claims, either because they really believe in it, or worse, because they're flat out lying.

Feel free to claim you perceive a difference. Your perception is no less real than mine. Just don't claim there is an actual difference without providing measurements, and don't claim that you actually hear a difference without providing ABX logs. Well, ideally, anyway. I know this isn't HA :-/
Edited by skamp - 3/16/12 at 2:38am
post #85 of 209
Quote:
Originally Posted by Girls Generation View Post

Although all these quantitative measurements and tests seem to prove that cables do not make audible differences in a technical sense, therefore implying that there is no point in buying aftermarket cables, if there actually are perceivable differences for the believers, whether it be placebo, bias, or purchase justification, wouldn't that mean that there are technically audible differences and the purchases were justified since the buyers had heard the improvement, ignoring the scientifically proven fact that there actually were none?


Absolutely, except for the part that I've put in bold. "Audible" is the wrong word here, as they're not detecting these differences with their ears. My best guess at the explanation for what the placebo effect is doing in this case (and I use the word "placebo" here with no negative connotations at all) is that the additional factors like confirmation bias, self-justification, price, aesthetics, and so on, are creating an effect in the brain similar to (but, I would hope, weaker than) psychedelic drugs. LSD users often report that "music sounds better" when they're tripping. Their perception of the sound is most definitely causing them more pleasure than it usually would, and that's with no change in equipment, or their ears, at all. So, we know that the sound going in is the same, but they're enjoying it more. Maybe the chemical and electrical processes in the brain that create our experience of sound are firing at a higher level of effectiveness in both the spaced-out hippy and in the audiophile with his expensive, chunky cables. Again, no negative connotations are to be taken from this comparison please.

 

At that point, we really can't argue that it doesn't sound better - it does! It's just that it's not an "audible" difference in the strictest sense. It can't be measured. Others can be in the same room and not hear any difference. We need help from our brain, which can alter our perception in a way that we find to be positive.

 

What ABX does, is tell us whether the difference we are experiencing is due to the sound being produced, or other factors. It can't, however, tell us that we were wrong to enjoy one sound more than another.

 

post #86 of 209
Quote:
Originally Posted by Mischa23v View Post

Just to make it clear to everyone, I’m not saying that there is a difference between cables or claiming anything. I’m just pointing out a major flaw in the arguments against abx and blind tests. Because that’s what this thread is about right?

 

My point is that even though people cannot tell the difference, it could be reasons other than human hearing limitations because of the way the test is being conducted (hearing loss, bad memory, etc). Thus, even though the test shows that people cannot tell the difference. We cannot be 100% sure that it’s because of a human hearing limitation and not one of other things that might cause people to not be able to notice differences. Also, because of the way the test is being conducted, we’re making an already difficult task even harder by using short term memory as a means to test if people can notice subtle differences.

That’s pretty much it.

 

Oh and because we cannot measure or detect if a person is suited to take this test, the chances of failure are higher than chances of success in a test like this. Because let’s be honest the things that can affect human hearing or concentration, or short term memory are almost infinite. Human aren’t perfect, and the diseases and disabilities and circumstances like environment a person lives in, the mood he or she is in, there are just way too many things the can result in people failing this test. Thus when people fail to notice changes in the test, it’s useless because the test actually tells us absolutely nothing important. We have no idea why they failed ot notice changes.

 

 




The reason why I do not think out audible memory is that bad is because of the following.

 

How come people can remember the 'differences' when the test is done sighted?

 

How come we can remember differences such as the difference between a radio, your car stereo and your own hifi?

 

How come we remember voices over the phone?

 

Indeed how are we able to audition hifi and tell any differences at all if it was not for memory?

 

 

Lets say it is memory, then ABX is not a fail for showing that up, it is a success. It has discovered we have a short term memory loss when it comes to sound quality differences. We should not dismiss that as a fail, we should then fit that into what we know about hearing and sound quality.

 

Lets say it is not memory, then again ABX is a success as it shows there is a point where differences get so small that they are now in reality inaudible. So any differences we hear when sighted are down to other senses.

 

With that knowledge you can then make buying decisions based on what is really audible and what is not. I for one will chose to buy cheap cables and not worry about my DAC or amp and concentrate on my headphones. Others will do it differently.

 

 

post #87 of 209

Quote:

Originally Posted by liamstrain View Post

 

Quote:
There is another case against DBT, which is most likely to crop up shortly.

 

Curious now.

 

 

OK, so it's nothing special, but it is totally predictable:

 

One argument against DBT/ABX testing is that any such arguments are AUTOMATICALLY and IMMEDIATELY shot down in flames, such that the arguments against ABX testing never get to be properly investigated by the very people skilled enough to do anything about it. There is overwhelming observational evidence that there are problems somewhere within such tests in certain applications, but the scientists are so busy being right that they barely stop for 10 seconds to question the possible flaws.

 

The scientists simply choose to remember all the evidence that makes them be right, and ignore any contrary evidence. Yes we all tend to do that. but this thread I think  is about "the evidence against ABX testing ...and explore what that may be", rather than "the evidence against ABX testing and immediately shoot down any suggestion".

 

Back to the taste analogy, because it's something we can all equally relate to: Sometime in the next 3 weeks, there will be a news article along the lines of: "wine buyers can't tell the difference between a £5 and £50 bottle of wine". The reporter will then triumphantly conclude that there is absolutely no point in people spending more than £5 for a bottle of wine - because DBT doesn't lie. It never seems to occur to them that this is the millionth time there's been a result like this, and the million example why such tests are unreliable. And 3 weeks later, it will happen again. Just stop occasionally and consider the evidence from other angles.

 

My take on short term memory in tests is...

 

The argument against short term memory in ABX tests is that it can actively work against you. For example, when listening to B, you suddenly notice a slight Cymbal strike that you didn't notice with A. From then on, you will always hear that cymbal with either A or B. Some people would deduce that there is no difference between the two and that it's just their stupid brains randomly noticing things that were always there. But just as likely, B was indeed superior because it's extra detail retrieval "pulled out" that cymbal from the mix. But once heard, your memory simply won't let you forget it, so of course you hear it in A as well, because the cymbal was always there, A just had it slightly more buried in the mix.

 

Just for balance, I think that short term memory is much better than long term at spotting differences in frequency response, volume and gross distortion. But that's not what this thread is about.  

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

   

 

 


 

 

post #88 of 209
Quote:
The scientists simply choose to remember all the evidence that makes them be right, and ignore any contrary evidence.

 

If they do that, they're not scientists. This would be cherry picking or confirmation bias. However, what is the evidence that ABX doesn't work? All that is necessary to disprove the theory that it does work, is one instance of reliable scientific testing that reveals a difference not found in ABX can be reliably detected through a different method.

 

Quote:
Some people would deduce that there is no difference between the two and that it's just their stupid brains randomly noticing things that were always there. But just as likely, B was indeed superior because it's extra detail retrieval "pulled out" that cymbal from the mix. But once heard, your memory simply won't let you forget it, so of course you hear it in A as well, because the cymbal was always there, A just had it slightly more buried in the mix.

 

If it's more buried in the mix, then you'd be able to spot that in ABX testing by listening to a loop.

 

I should also point out that there is nothing in DB ABX methodology that inherently restricts us to short term tests. There is nothing stopping someone from listening to a whole album on A, again on B, and then again on X. Or even spending a week with A, a week with B and then a week with X. The ABX method itself is not at fault here if we've decided that short term memory is no use, just the specific use of it in these cases.

 

I also think it's worth reiterating that this ABX discussion should be taken in the context that it is usually used to test claims of seriously detectable differences. These differences usually are reported as being easily detectable when switching between the equipment with full knowledge of which is which, without saying that short term memory makes these sighted tests invalid.

 

Finally, I have yet to see an instance of a subjectivist, when presented with a positive result from an ABX test (eg speakers perhaps), rejecting the findings due to inherent unreliability in the tests.


Edited by jumblejumble - 3/16/12 at 5:46am
post #89 of 209
Quote:
Originally Posted by TheAttorney View Post

Quote:

 

OK, so it's nothing special, but it is totally predictable:

 

One argument against DBT/ABX testing is that any such arguments are AUTOMATICALLY and IMMEDIATELY shot down in flames, such that the arguments against ABX testing never get to be properly investigated by the very people skilled enough to do anything about it. There is overwhelming observational evidence that there are problems somewhere within such tests in certain applications, but the scientists are so busy being right that they barely stop for 10 seconds to question the possible flaws.

 

The scientists simply choose to remember all the evidence that makes them be right, and ignore any contrary evidence. Yes we all tend to do that. but this thread I think  is about "the evidence against ABX testing ...and explore what that may be", rather than "the evidence against ABX testing and immediately shoot down any suggestion".

 

.........

 


 

 


Can you evidence scientists shooting down arguments against blind testing in flames and only remembering the evidence that makes then right? What is the overwhelming observational evidence that there are problems with blind testing and what applications are you referring to? You have made some very strong remarks against science which I think you need to evidence them.

 

One of the complete fails of the audio industry is to submit their claims for proper independent study, by scientists. For example, no cable maker has ever put their cables to be tested at a university engineering department to validate their claims about enhanced sound quality.

 

There is no flaw in a properly conducted blind test. It is a case that many audiophiles do not know what to do with the results of those tests, especially since they are contrary to what they expected/hoped for.

 

 

 

post #90 of 209

 

I’m really enjoying reading your posts so far, they’re insightful, open minded, and very interesting. Thank you for taking part in this discussion.  Your one of the few people here that really add a lot to this discussion, and one of the people that I look forward to reading from.

 

Unfortunately, I don’t have any data relative to an audio test. However, I do have data that I personally believe can be used and is useful in demonstrating that at least the tests can be conducted in a better fashion. However, I’m not really allowed to share this publicly, but if you send me your email I can send it to you as a protected pdf file. I would really appreciate if you don’t post it on the net though. (As soon as I’m back to work)

 

It was a short term memory experiment conducted on children (ages 7 to 12), to try and see if they can find better ways in which children can retain information. The most relevant part in the experiment was when the children were shown a circle with slices shaped like well a pizza. The circle had 4 colors and 12 slices.  One of each color has its position changed in the circle.

 

The children were shown first picture A with circle and colors arranged in certain way for 5 seconds, and then shown picture B which had a slight different color arrangement for 5 seconds. Finally the children were shown picture X which was either A or B and were asked to identify if picture X matched picture A or B.  What I liked about this experiment was:

 

  1. At the end of experiment they showed the children all of the pictures together and asked them to see whether they can identify if picture X matched A or B. one of the tests was so hard that none of children could solve it even when pictures were right in front of them.
  2. Some of the children just couldn’t identify which picture matched X because well their either were color blind maybe, or didn’t understand the question, or didn’t want to cooperate. They were removed from the experiment.
  3. They started using more pictures later on and more colors, and they were able to measure when it was normal for the children to notice the differences and when it was extremely hard to impossible for the children to remember or be able to notice the differences. They were also able to tell you whether the children failed because there was too much information being presented so they can’t remember it all, or because the colors were blending together and it was too difficult to notice the difference.
  4. When they had 6 pictures up and asked the children to try to identify which picture matched picture X, none of children were able to identify which picture matched X. When they used 2 pictures the rate of success was around 76% with girls having a higher success rate. When they used 3 pictures, the success rate dropped to 47% if I recall correctly with girls having a higher rate of success than boys ( I have read the document it a month ago, so the numbers might not exactly be accurate).
  5. They added marks later on, arranged the colors in certain way, and used different shapes, from bright to dark to see if that would make it easier for the children to remember the difference or notice them, which resulted in a 23% increased success rate.

 

The difference between this test and most of abx tests that I’ve come across when it comes to sound. Is that when the children failed, they were able to accurately tell you why they failed and even readjust the test so they can accomplish their goal. They were able to identify and measure to a very accurate point, and tell you the success rate of children, in any point in experiment.

 

When the children failed they knew why they failed. Last but not least, they were able to remove the children who would have failed for other reasons, reasons that shouldn’t be taken into account when giving a test like this. For example a child who is blind shouldn’t even be participating in this experiment.

 

However, most of these audio abx tests pretty much don’t really take anything into account if people fail. They don’t really bother in researching why people might have failed. Most of these abx audio tests really only are interested in the results when people pass the test. That is why I’m claiming that when an audio abx tests fails, you learn absolutely nothing of importance. You have no idea why the test failed. Because no measure have been taken to see why the test has failed.

 

Thank you again for taking part in this discussion.

 

 

Quote:
Originally Posted by liamstrain View Post

While you have presented a theoretical flaw in abx testing. You need to present evidence that the memory issues you raise, actually would in practice, invalidate the tests or even have a measurable effect on the results. There is a lot of evidence to support these testing methods as useful within their known limitations. You need more than your theory and assertions to overcome their use, or to redefine their scope.

 

It is an interesting argument. But you have not demonstrated anything that would indicate we need to adjust our methodology. Do you have data, for instance about what threshold of audibility is that would make abx less reliable? 

 

 

 

 

 

No, as has been stated before, our experience has been that abx makes the differences easier to identify (by providing immediate contrast in stereo under realistic listening conditions - short term memory not-withstanding). Not harder. Especially with large enough samples to provide a measure of error correction (as most good DBT/ABX data should be).

 

You have stated repeatedly otherwise, do you have experience or data to back that assertion up? 

 

 

 

 

 

No, we can never be 100% sure of anything. But I'm pretty comfortable with statistical significance and high percentages of confidence. Especially when corroborated by different methodologies and data sets. 



 

 

New Posts  All Forums:Forum Nav:
  Return Home
  Back to Forum: Sound Science
Head-Fi.org › Forums › Equipment Forums › Sound Science › What are the arguments against double blind tests (incl. ABX)?