Head-Fi.org › Forums › Equipment Forums › Sound Science › My approach to interpreting ABX and Blind listening tests
New Posts  All Forums:Forum Nav:

My approach to interpreting ABX and Blind listening tests

post #1 of 3
Thread Starter 
Let me just make a point that I think is important.

If a listener compares $20 Radio Shack speakers with $10,000 B&W speakers and finds that the B&W's sound WAY BETTER, unblinded, the two speakers may actually sound different. On the other hand, that listener's positive expectations of the $10,000 speakers vs. $20 speakers (especially after forking over $10,000 of his hard earned cash) might influence his perception ("the placebo effect"). This might have caused him to PERCEIVE that his $10,000 speakers sound better than his $20 speakers because he EXPECTS them to and would feel really foolish and embarrassed if they actually sounded the same. So which is it? Are the B&W's really better, or is the listener protecting himself from cognitive dissonance (the psychology term)?

To TEST this, we would do a blind ABx comparision between the $20 RS and $10,000 B&W's. Assuming the ABX comparison is well-conducted (i.e. blinded, level-matched, large population, etc. etc.), if most blinded listeners correctly identify and favor the B&W's, then the B&W's probably do sound better in actuality. While the listener's unblinded glowing impression of the B&W's might have been influenced by his expectations for speakers he paid $10,000 for, at least we know that the B&W's truly SOUND better as well.

In contrast, supposed this same listener compares $1000 fancy speaker cables with $5 Radio Shack zip cord in unblinded fashion. He again reports that the $1000 cablesound WAY BETTER. Could it be that the $1000 cable truly sounds way better? Yes! Could it also be that the listener's positive expectations for a $1000 cable plus the fact that he forked over $1000 instead of $5 also cause him to PERCEIVE that the $1000 cable sounds way better? Yes!

Once again, we could sort this out with a blinded ABx test. This time, let's say that in an ABx test with many trials, 5 out of 10 listeners could identify and prefer the $1000 cable in 65% of trials after long 1 hour listening tests, and this was significant at the p<0.05 level. We could conclude that there probably is an audible difference between the tested cables! However, we would want to verify that levels were matched precisely, and there was no deviations from the blinding, etc. Even though this test doesn't specifically test the magnitude of the differences between the tested cables, one could probably conclude that the differences are very small - after all, if people could only choose the $1000 cable 65% of the time, and only half the listeners could do this, how much of a difference would a large population find?

Now, let's say that in a large ABx test with that $1000 cable vs. $5 cable, none of the 10 listeners could reliably choose the $1000 cable when blinded beyond that which we would expect from random guessing. Does this mean there is NO difference between cables? Not necessarily. Perhaps we were using very lousy speakers or a low-quality source or the subjects were given 1 second total listening time, etc. But if there are NO genuine and reasonable criticisms, then the difference between the cables is probably either non-existent or so small that is non-detectable by the ABX test used.

So then how would we explain the amazing difference initially PERCEIVED by the listener when he was not blinded? The explanation would be that it came from his positive expectations for a cable he spent $1000 for and NOT from actual differences in the sound quality of the cable itself.

This is how I would approach ABX tests, and this is generally how I think any controlled trial or comparison should be interpreted. Thanks.
post #2 of 3
I would like to point out a flaw in your reasoning: a product(say headphone, dac, or amp) with a higher price tag isn't always preferred over lower priced ones.

I'll use myself as an example, I've came across of of headphones(Edition 8/9) and amps(Opera Consonance, HR-2), that are much have a much higher price tag than other some of my other gear, but I much preferred the cheaper ones while staying level matched.

As a result, I believe that a higher price tag does play some part but not always.
post #3 of 3
Thread Starter 
Quote:
Originally Posted by moonboy403 View Post
I would like to point out a flaw in your reasoning: a product(say headphone, dac, or amp) with a higher price tag isn't always preferred over lower priced ones.

I'll use myself as an example, I've came across of of headphones(Edition 8/9) and amps(Opera Consonance, HR-2), that are much have a much higher price tag than other some of my other gear, but I much preferred the cheaper ones while staying level matched.

As a result, I believe that a higher price tag does play some part but not always.
Would you be so kind as to re-read what I wrote again? Based on your response, I'm not sure if you're following it.

I used the example of "high price" to illustrate influence of an expectations on perceptions. For many people, seeing a high price tag makes them think the speaker will sound better. However, I could have used ANYTHING as an example. It could have been the color of the cabinet, the size of the woofer, whatever - anything that could "bias" that particular listener. Does this make sense? Do you understand how a blinded comparison can "control" for these types of "biases?"
New Posts  All Forums:Forum Nav:
  Return Home
  Back to Forum: Sound Science
Head-Fi.org › Forums › Equipment Forums › Sound Science › My approach to interpreting ABX and Blind listening tests