A/B tests are useful, but be careful trying to work out exactly what such comparisons "prove." There's a lot of logic dimensions to the epistemology here. Stereophile just ran a column which points out a few of these thorny points. ("Audio, Meet Science," March 2011 issue) They used these points to frame apologist arguments in favor of their particular brand of subjectivism, but their misuse of these arguments doesn't negate such truth as may be inherent in them.
For example, you can't prove a negative by studying examples. That is to say, let's we took RCA cable A and RCA cable B and devised a proper method for a valid comparison of the sonic impact they had on audio signals passing through them. Now we allow 1,000 listeners to each conduct an A/B/X blind listening test. Let's further hypothesize that none of these 1,000 listeners could tell cable A from cable B at a level any better than chance. While we have shown that 1,000 listeners can't tell the difference, this does NOT prove that a sonic difference does not exist. Stereophile is correct in stating this.
Now while this is LOGICALLY true, philosophically true, mathematically true, it should be noted that in addition to philosophical considerations, life as it is lived has certain practical nuances. So, while the test cannot show that no sonic differences exist, would YOU spend $600 on a product whose sonic benefits a large number of listeners couldn't hear, and not one single listener that tried the A/B/X test was able to hear a difference?
So, just because you can't hear a difference doesn't prove that no difference exists. But if it sounds the same to you, this might influence your spending!
There are also well documented placebo effects that can be observed in hearing based tests- the famous one took place under the auspices of the National Research Council Canada. They do a lot of audio related research, in part to find ways to design better sounding gear to help Canadian manufacturers. They had a large listening room with a curtain at one end. A number of listeners were seated in the room -young, old, men, women, musicians, golden-ear audiophiles, regular Joes. The curtain pulls back to show a bookshelf speaker on a stand. It has a red cabinet and black grille cloth. A number of musical selections and pink noise are played. The curtain closes. In a short time, the curtain opens again- this time there is a speaker on a stand with a blue cabinet. The same musical selections and pink noise are played. The curtain closes. The audience is asked to write down their impressions of differences between the two speakers. These are collected and the majority of listeners find that the first speaker had a "warmer, musical" sound while the second speaker had "less bass and a clearer treble." Actually, however, there was only ONE SPEAKER used. Stage lights were shown on the speaker that was housed in a MATTE WHITE cabinet- red filtered light made it look like it had a red cabinet, blue filters made it seem a different speaker, with a blue cabinet. The differences heard were entirely in the minds of the listeners. The color of the speaker influenced the way the listeners judged the sound. This is like placebo effect in medicine- the listeners didn't IMAGINE sonic differences- their brains CREATED different audio perceptions for them. They actually heard these differences, although the differences they heard were based on changes in their perception rather than being related to some physical property of the actual transducers. The SOUND was same from both red and blue speakers- but that sound was HEARD differently.
It's important to remember that hearing takes place in the BRAIN, not the ears. There's more that goes into a persons' perception of subtle audio qualities than just sound! For example, I swear that amplifiers with thick front panels can sound better to me, in spite of myself. That's why BLIND testing is a better gauge of what, if any, differences a listener can perceive. I will be doing blind tests before long, I am building the little relay boxes for this and writing a little code, at some point I will start using these gizmos. (IF I can make them work! Hah, it's been a long time since I wrote any software aside from some simple shell script type stuff, javascript and kindergarten-level PHP.)
