As I wrote that it did strike me the PWDs et al. are hardly 'top end' I guess, compared to $40,000 dacs
(start of rave - may be TL; DR)
The interesting thing about the 'science' crowd is that it isn't science. As far as I was able to determine during a deeply unsatisfactory year or so combing those forums. Those that proclaimed scientific principles and 'facts' the loudest seemed in general to have no idea that Aristotle's science (deductively based) gave way to Galileo (hey, how about we perform the experiment to check whether our deduction from current theory actually works? Before we say anything?) many centuries ago
A good deal of my scientific work includes measurement of ambiguous stimuli, similar to what we on head-fi use our ears for Reliable metrics with good predictive validity are routinely developed in many scientific fields. That the listening we do as audiophiles might also have real validity seems supported by anecdotal evidence that we can meaningfully share impressions and find we frequently agree. I haven't tried to model this statistically; there would be ethical and other issues to setting up a real study on head-fi. (But a simulation to test assumptions is a real possibility). The other frequent gripe - poor aural memory - is true as far as it goes but ignores real-world fact about how memory actually works (chunking and the role of language).
As a statistician as well as researcher my best guess is the agreement occurs well above chance and with good 'effect size'.
In other words, there is a point to head-fi and its reviews such as yours (HC, Prep et al), Purrin's group, Project86's, Skylab's and others. We're not just a social club after all
(end of rave)
Sorry for the O.T. - er, where was I?
I think it was Einstein that said: "Not everything that counts can be counted, and not everything that can be counted counts".
The reality of science is that the more we learn, the more we learn we don't know. Knowledge generates more questions than it answers. By that logic you can make the argument that we are actually getting dumber, not smarter. Einstein didn't say that. That was me.
Edited by alancohen - 7/5/13 at 10:20am