There have been serious blind tests where a trained group of listeners have listen to a song and each ten times. During the test the scientists have told the group that they have changed something in setup, like for instance a cable , amp , cd - player or even the speakers.
The group have during the test filled in to a test blanket if they thought that the sound got worser or better in a 10 grade scale with 10 different questions to fill in.
And the result in the end of the day the scientists could distinguish a very interesting pattern from all in the test group just after the second listening to the same track, all taught they could here a significant improvement in different areas vs the first listening and so on..
What the group did not know of was that the scientists hadn't dune anything with the setup, the group had been listening to the same untouched setup with the same track 10 times in a row, but their mindset have tricked them to believe and their brains have focused on different things in the music each time the scientists have told them that they had dune something with the setup.
Yes, these tests have been done. Some equally interesting and of fully blind approach were performed at the London HiFi Circle. What was tested here was the power of belief. Here the procedure itself was very similar, however the administrator claimed to have changed mains cables, when in reality the polarity of one channel was inverted.
Those in the group who were "cable believers" could hear the polarity inversion of one channel with 100% accuracy and remarked how soundstage collapsed and bass became weak. Those in the group who were "cable disbelivers" failed to hear any difference and marked accordingly.
What both sets of tests show is that any blind one in which the subject have both knowledge of what is being tested and expectations is... actually severely biased. As a result, 'blind' in a way fails to remove experimental bias.
Any valid blind test must be blind not only to the identity of the devices/options tested, but also to the nature of the test itself. Otherwise the placebo/nocebo effect introduces a strong bias, which results in a strong randomising effect on the outcome. Differences expected are thus heard whereas non-existent and not expected ones are thus not heard, even when they exist.
We would like to (again) draw the attention to a blind test that shows how it should be implemented and is among a very small list of tests that avoided massive levels of bias in the experiment.
http://www.stereophile.com/content/god-nuances-page-3
Starting at the heading "Expert testimony" we read the description of a really serious experiment in listening, with interesting conclusions.