That is an interesting topic. Back when I was using a minidisc, I did some A/Bs where I consistently PREFERRED the sound of ATRAC3 to the original CD source. It sounded warmer to me. This is a lossy codec, but one which is far more advanced psychoacoustically than something like mp3.
Here's an interesting clip from that same worthless document. (It's worth reading.)
|Besides the danger of listening to equipment instead of music, the next most fundamental challenge to useful evaluation is overcoming the amazing human ability to adapt.
• We are astonishingly capable of “seeing” through distortion. We (generally) don’t feel our clothes, yet we are sensitive to even a single rain drop falling on our clothing. We can wear all colors of sun-glasses and yet still see that the sky is blue. If we use yellow goggles while skiing on a cloudy day,
when we take them off the snow looks purple. The “solution” isn’t to get out the yellow paint to fix the snow, the solution is to allow ourselves time to re-calibrate our references. Once we have adjusted to a colored (distorted) reference, we can be fooled into thinking reality is wrong.
Have you ever been given a cassette tape and you didn’t know if it was Dolby encoded or not? You probably pushed the Dolby button on and off, while you were playing the tape, in an attempt to decide which way was correct. Odds are that both positions sounded wrong. One way sounded too bright and the other sounded too dull. In this artificial context one is faced with two conflicting references, both of which make the other sound wrong. A common response is to wish there were a middle position, even though one of the existing positions is absolutely correct and the other is absolutely wrong.
This is an example of how an instant comparison can be a highly deceptive selling technique and not part of a trustworthy evaluation methodology. Whoever controls the switch can sell whatever they want. This also applies to a lone individual doing an “evaluation” by themselves. Just because a second party isn’t involved doesn’t prevent someone from “selling” themselves whichever component first grabs their attention, whichever one got the good review, whichever one has an attractive story.
• Another simple opportunity for deception (including self-deception) is the A/B phenomenon: The second time a piece of music is played, the listener is bound to notice something that wasn’t noticed the first time-even with familiar music. This perception feeds directly into the value system which dictates that more information is our most commanding priority. If you want to sell something, always play it second.
There are ways around this pitfall: Go back to “A”. No matter which is better, going back to “A” will be a surprise. Since the step from “A” to “B” included the “novelty factor” in addition to the real difference, the step back to “A” will be surprisingly different from the original step to “B”, simply because the novelty factor has disappeared. “A” will seem to be better than when played the first time. Continuing on to play “B” a second time, without the benefit of the novelty factor, then reveals its truer relationship to “A”. After an initial A/B/A/B, it is possible to move to “C” and “D” with far less confusion.
• It can be easier to evaluate three products instead of the apparently simpler task of evaluating only two. Even without the deception of an instant A/B, any A/B is subject to a certain amount of the effect described with the cassette tape example-the truth is perceived as somewhere between the two.
If two of the three products are relatively similar, probably (but not always) different models from the same manufacturer, then it is quite easy to establish an absolute hierarchy between the two products.
When a third and different product is compared to a similar pair, it becomes a comparison between a line and a point, instead of just between two points. It becomes much easier to establish a hierarchy: that the third product is preferable to either member of the pair, inferior to either, or somewhere in-between.
• There is almost no way back to the “garden” of complete innocence. It requires great awareness and careful methodology to attain anything like the direct vision available to those who cannot be distracted by misleading details. This view flies in the face of those who declare that people have to learn what is good sound, go to lots of live concerts and study the technology. Bull!
The only thing that needs to be learned is how not to be misled by the incredibly deceptive process of listening to equipment. People hear real sounds all day long. None of these real sounds has the added layers of distortion which exist in every audio system. Whether or not we have ever heard a particular singer or instrument, we can recognize whether more or less “extra stuff” is in the way.
• As for comparisons where there seem to be only “insignificant” differences between components, this is usually proof of a faulty context and/or methodology. This is most obvious in the discussion of ABX testing.
In an ABX set-up, the listener does not know whether or not there has been any equipment change at all. ABX testing is not a question of how a fixed but blind “A” compares to a fixed but blind “B”. Because there are too many unknowns, the ABX test becomes primarily an opportunity for embarrass-
ment. Context is everything, and the ABX set-up is one very distorted context, much too far removed from the purpose of an audio system. ABX fans believe that a lack of repeatable hierarchy proves there are no valid differences. Others of us believe the same evidence proves that the ABX test is an invalid methodology.
Does all this mean that trustworthy conclusions are impossible? No. It means a balanced perspective is paramount. It’s a little like shopping for advice (which really is more useful than shopping for equipment): If honesty is the sole criterion, you’ll probably end up taking advice from someone honest, but
incompetent. If competency is the sole criterion you get the picture.