I can’t believe that no one here addressed the main issue. For a very long time psychology was rejected as science because all their blind tests depended on short term memory. Memory depends on your imagination, your expectations, strength of your memory, etc. Memory can be over written, manipulated, and plain false sometimes, etc.
Pick up a picture puzzle where they have 2 pictures right next to each other, cover the first one and then look at second one and try to find the differences. It is much harder to pin point the differences when you're depending on your memory. If they were right next to each other, the differences are much more evident. When watching HD movies, most people are aware that the picture looks better somehow, but ask them to pin point the differences and most of them will get them wrong, they might be headed in the right direction but most people will guess wrong.
A lot of people will also claim that test subjects are using gear and tracks that they’re familiar with. However, that is much worse. Why do people tell you not to correct your own essay, because no matter how many times you read it you can’t see some of your own spelling mistakes. You will actually read the word correctly even though it is quite obvious to another person that you spelled it wrong. That’s why tests that depend on memory are so unreliable. You already have expectation and stubborn people will never change those expiations. Some people would have already made their decision just by hearing the first track.
I’ll give you another example, with my jh 16. I used the normal cable vs my Piccolo cable in blind test. The difference was much more obvious when I hooked up one end of normal cables to right jh 16 in ear and one end of Piccolo cable to the left jh 16 in ear phones (you can use same trick with 2 different files). But it’s still tricky because the brain processes sound differently to help you pin point where sound is coming from. If you’re surprised by loud sound sometimes you can’t even pin point where it came from because your brain is spending resources to keep you alert in case of danger.
There is nothing wrong with the test, it’s the way it’s being used that people have problem with. Basically what their saying is, you’re taking a 12 year old kid and you’re giving him a test designed for engineering graduates. When he fails you claim that this kid is retarded. However, even though there is nothing wrong with test, its design for engineer graduates. Furthermore, the 12 year old isn’t retarded because the test was too hard.
As an artist who loves to draw and loves to take Photoshop courses, I can tell you that I know a lot of tricks to make an image look better even though those tricks are undetectable. You will clearly see a difference but you can never really tell what it is because it blends in with image. Most of these audio blind tests are testing material with maybe a 10-20% difference. We’re not talking about adding noticing new stuff (like a chair that wasn’t there before), where talking about minor changes. However, these minor changes can sometimes tremendously increase enjoyment or experience.
You cannot claim that people cannot tell the difference between 2 files scientifically when you use something so unreliable like memory as a major factor in a test result that was made harder than it should have been. Even if you cut file up or loop through segments you’re still using your memory. There are at least 4 best sellers from the dean of medicine in Harvard on why using memory in blind test is unreliable and unscientific.
Edited by Mischa23v - 3/9/12 at 4:03am