I'm fairly new to Head-Fi, having joined recently do add a bit of insight and technical info to a thread on oversampling. My home stomping ground is HydrogenAudio.org - a site devoted to the advancement and understanding of lossy codec performance (among other things) through objective, scientific analysis - with a strong emphasis on burden of proof. I'd like to chime in with a few thoughts, hopefuly without rustling any feathers. At least, that's my intent. Moving forward...
After having read the first few pages on this topic, I've come to the conclusion that folks in this thread, and probably Head-Fi at large, can be loosely divided into 2 camps. The first camp more or less reflects the same objective attitudes and knowledge base as the participants at HydrogenAudio. The second camp seems genuinely passionate about audio and sound quality, but falls short of fully understanding the science behind how to characterize and identify same.
What I'm getting at is that while the spirit of this test approach may have been completely sincere and well-meaning, it fails to appreciate a simple truism in codec testing: a frequency domain graphic of a test sample will never convey the quality of a lossy codec.
The simple and undeniable reason for this is that the human ear to brain connection is not a visual discrimination device, but rather a psychoacoustic transform with its inherent flaws (ie. frequency, temporal and loudness masking) that are fully exploited by lossy codec algorithms. Furthermore, a visual representation of an audio test sample in the frequency domain may completely mask sound quality cues that are easily perceived by the human ear.
The purpose of an audio codec is not to visually match an arbitrary frequency domain representation of the original audio sample. Rather, it is to fool the listener into believing that it sounds the same as the original. Enter the ABX testing methodology.
I won't, nor do I want to attempt to settle what appears to be a heated discussion between two opposing points of view. What I will suggest, though, is that those who believe this to be a fair and objective codec test take their case to HydrogenAudio with a view to gaining additional insight into why this test is truly and completely flawed.