Let me just make a point that I think is important.
If a listener compares $20 Radio Shack speakers with $10,000 B&W speakers and finds that the B&W's sound WAY BETTER, unblinded, the two speakers may actually sound different. On the other hand, that listener's positive expectations of the $10,000 speakers vs. $20 speakers (especially after forking over $10,000 of his hard earned cash) might influence his perception ("the placebo effect"). This might have caused him to PERCEIVE that his $10,000 speakers sound better than his $20 speakers because he EXPECTS them to and would feel really foolish and embarrassed if they actually sounded the same. So which is it? Are the B&W's really better, or is the listener protecting himself from cognitive dissonance (the psychology term)?
To TEST this, we would do a blind ABx comparision between the $20 RS and $10,000 B&W's. Assuming the ABX comparison is well-conducted (i.e. blinded, level-matched, large population, etc. etc.), if most blinded listeners correctly identify and favor the B&W's, then the B&W's probably do sound better in actuality. While the listener's unblinded glowing impression of the B&W's might have been influenced by his expectations for speakers he paid $10,000 for, at least we know that the B&W's truly SOUND better as well.
In contrast, supposed this same listener compares $1000 fancy speaker cables with $5 Radio Shack zip cord in unblinded fashion. He again reports that the $1000 cablesound WAY BETTER. Could it be that the $1000 cable truly sounds way better? Yes! Could it also be that the listener's positive expectations for a $1000 cable plus the fact that he forked over $1000 instead of $5 also cause him to PERCEIVE that the $1000 cable sounds way better? Yes!
Once again, we could sort this out with a blinded ABx test. This time, let's say that in an ABx test with many trials, 5 out of 10 listeners could identify and prefer the $1000 cable in 65% of trials after long 1 hour listening tests, and this was significant at the p<0.05 level. We could conclude that there probably is an audible difference between the tested cables! However, we would want to verify that levels were matched precisely, and there was no deviations from the blinding, etc. Even though this test doesn't specifically test the magnitude of the differences between the tested cables, one could probably conclude that the differences are very small - after all, if people could only choose the $1000 cable 65% of the time, and only half the listeners could do this, how much of a difference would a large population find?
Now, let's say that in a large ABx test with that $1000 cable vs. $5 cable, none of the 10 listeners could reliably choose the $1000 cable when blinded beyond that which we would expect from random guessing. Does this mean there is NO difference between cables? Not necessarily. Perhaps we were using very lousy speakers or a low-quality source or the subjects were given 1 second total listening time, etc. But if there are NO genuine and reasonable criticisms, then the difference between the cables is probably either non-existent or so small that is non-detectable by the ABX test used.
So then how would we explain the amazing difference initially PERCEIVED by the listener when he was not blinded? The explanation would be that it came from his positive expectations for a cable he spent $1000 for and NOT from actual differences in the sound quality of the cable itself.
This is how I would approach ABX tests, and this is generally how I think any controlled trial or comparison should be interpreted. Thanks.
If a listener compares $20 Radio Shack speakers with $10,000 B&W speakers and finds that the B&W's sound WAY BETTER, unblinded, the two speakers may actually sound different. On the other hand, that listener's positive expectations of the $10,000 speakers vs. $20 speakers (especially after forking over $10,000 of his hard earned cash) might influence his perception ("the placebo effect"). This might have caused him to PERCEIVE that his $10,000 speakers sound better than his $20 speakers because he EXPECTS them to and would feel really foolish and embarrassed if they actually sounded the same. So which is it? Are the B&W's really better, or is the listener protecting himself from cognitive dissonance (the psychology term)?
To TEST this, we would do a blind ABx comparision between the $20 RS and $10,000 B&W's. Assuming the ABX comparison is well-conducted (i.e. blinded, level-matched, large population, etc. etc.), if most blinded listeners correctly identify and favor the B&W's, then the B&W's probably do sound better in actuality. While the listener's unblinded glowing impression of the B&W's might have been influenced by his expectations for speakers he paid $10,000 for, at least we know that the B&W's truly SOUND better as well.
In contrast, supposed this same listener compares $1000 fancy speaker cables with $5 Radio Shack zip cord in unblinded fashion. He again reports that the $1000 cablesound WAY BETTER. Could it be that the $1000 cable truly sounds way better? Yes! Could it also be that the listener's positive expectations for a $1000 cable plus the fact that he forked over $1000 instead of $5 also cause him to PERCEIVE that the $1000 cable sounds way better? Yes!
Once again, we could sort this out with a blinded ABx test. This time, let's say that in an ABx test with many trials, 5 out of 10 listeners could identify and prefer the $1000 cable in 65% of trials after long 1 hour listening tests, and this was significant at the p<0.05 level. We could conclude that there probably is an audible difference between the tested cables! However, we would want to verify that levels were matched precisely, and there was no deviations from the blinding, etc. Even though this test doesn't specifically test the magnitude of the differences between the tested cables, one could probably conclude that the differences are very small - after all, if people could only choose the $1000 cable 65% of the time, and only half the listeners could do this, how much of a difference would a large population find?
Now, let's say that in a large ABx test with that $1000 cable vs. $5 cable, none of the 10 listeners could reliably choose the $1000 cable when blinded beyond that which we would expect from random guessing. Does this mean there is NO difference between cables? Not necessarily. Perhaps we were using very lousy speakers or a low-quality source or the subjects were given 1 second total listening time, etc. But if there are NO genuine and reasonable criticisms, then the difference between the cables is probably either non-existent or so small that is non-detectable by the ABX test used.
So then how would we explain the amazing difference initially PERCEIVED by the listener when he was not blinded? The explanation would be that it came from his positive expectations for a cable he spent $1000 for and NOT from actual differences in the sound quality of the cable itself.
This is how I would approach ABX tests, and this is generally how I think any controlled trial or comparison should be interpreted. Thanks.
