May I change the topic? I love the concept of this thread and would like to make a few observations and then a challenge.
I want to discuss testing (with human listeners) vs. measurement (with equipment).
The food industry provides an interesting model. Food chemistry is certainly advanced, on par with modern audio science and electronics. Nearly every aspect of human tasting has been studied (some new discoveries recently!) and can be measued chemically. But the profit-making food industry uses taste testing, not measurements, to make final marketing decisions, since time after time it has been shown that measurements, no matter how precise, just fall a tad short of predicting human perception. The chemical tests say "difference in saltiness is too small to be tasted" but the consumer panel says one sample is "too salty".
So very careful scientific testing rules their world. Disclosure: I am not a food scientist and don't work in the food industry, nor an audio or electrical engineer working in the audio industry. But I am a pretty good statistician (I won the Journal of The American Statistical Assoication Theory and Methods prize several decades ago) and have read widely on testing via food tasting and testing via audio listening. I believe I am qualified to judge the statistical methods used (there's a doctorate diploma in statistics from a fancy school with my name on it gathering dust in the back of one of my mom's closets, btw).
Blind testing should rule the day here. It doesn't have to be double blind, just blinded from the listener. Typically that would eliminate headphones, but not cables, oversampling DACs, portable amps etc. I suppose we could literally blindfold people and put the cans on their ears, so let's allow HPs back in ... but to start, lets try cables and portable amps.
Consumer food testing uses some clever protocols that bait and switch ... you lie to the tester about which sample he has, etc. This is important. As important as getting the levels right, etc.
At the NJ meet there was a blind test of silver vs copper line-out cables for an iPod. The blind was done well, but no valid statistical protocol was used. Nonetheless, I am convinced most of us heard a difference between the cables. Trying to assign "cool" to silver and "warm" to copper, I mis-identified them. Interesting.
A better test -- a test with a protocol -- would have proven or disproven than I could reliably tell the difference between the cables, i.e., that there was a difference or not (for me). It would then be great fun to measure the standard electrical attributes of these two cables.
I tried to get the group to do a blind test of Predator vs iQube, but no one would take the time. Everyone wanted to listen to the shiny toys. I had prepared a precise "fool 'em" design, and printed sheets to record the answers. No go.
I plan to do a mini-meet at my house in December specifcall inviting two or three NJ-based "subjects" and do this for real.
Shouldn't we organize this ... do it at meets all over ... champion this as the way out of our endless debates ... sure, the pro audio literature does blind tests and reports "no difference" all the time, but they typically use big panels of listeners and rooms with speakers. The variation between people swamps the results. This group needs to do this our way, with our esoteric gear ... person-by-person.
We exist to help each other wring performance out of our gear, and optimize our spending. Vendors are guests only. It's time to put these things to test.
My view of a meet would be a sequence of highly planned, well-publicized shoot outs done scientifically.
Comments? Ideas? Suggestions on how to go forward?
