Originally Posted by powertoold
As I said, it's easy to eliminate the objectivists altogether - have a DBT
That's not easy at all. It is in fact extremely difficult. I know what I'm talking about, I've lead or took part in two double blind tests so far. The first between interconnects, the second between amplifiers.
The first major difficulty is to find a room. Our first test took place in a hifi shop while it was closed, thanks to a salesman who had access to it. The second took place in the listening room of a dealer, who had it separated from the shop. It took more than one year between the listeners being ok to meet and bring a given list of amplifiers, and the day the dealer proposed to host a blind listening test.
The second difficulty is to adjust the amplifiers levels. Same problem (even worse, since there is no volume setting) with CD players. With many amplifiers, this is plainly impossible because the volume is only adjustable by steps of 1 dB, while a precision of 0.1 dB is required for blind comparisions.
The level measurment is extremely difficult in itself. A voltmeter or an oscilloscope are useless, not talking about a sonometer, because none have the required precision, except very high end voltmeters... don't forget that we are measuring alternative voltage, not direct... Standard voltmeters do nothing less than 230 volts in alternative mode (that is, if you don't go too high in frequency above 50 Hz !).
A soundcard can do it, if you feed it with the speaker output, but unless you have strong XLR plugs, the soundcard jacks are likely to cause short-circuits, that can blow the amplifier out, especially expensive ones, that seldom have the basic protections against it.
And don't forget that you must measure the speaker output while the amplifier is loaded, since a low damping factor is likely to cause a voltage drop higher than 0.1 dB when you plug the speakers on !
Third problem, you talk about 13/16 being a success... that's a 1 percent chance of guessing. For many sceptics, the odds for two amplifiers that measure well to sound different is just 1 percent. Guessing or sounding different is thus equally likely ! A proof would require a probability of guessing equal to 1 percent squared, that is 1/10 000, therefore a minimal score of 16/16 !
Then, one isolated success is not a scientific evidence. The result must be reproduced.
Then, if some people fail because a musical sample, or a given speaker model, is not discriminating enough, you get the problem of the meta-analysis : the successes become isolated among failures, they can then be considered as statistical fluctuations, while they were perfectly significant, taken alone !
Double blind listening test are extremely interesting. They are even very interesting to undergo by oneself even if the result is unconclusive.
But they are also extremely difficult to run properly.