SoundAndMotion
100+ Head-Fier
- Joined
- Jul 15, 2015
- Posts
- 353
- Likes
- 229
Hi MM,
I often like your posts, not because I often agree, but because your role is that of being skeptical of the “skeptic side”. You don’t fear science, but rather embrace some interesting ideas such as auditory scene analysis, and the need to know *how* to measure something. But for years (and in this thread), you have made some incorrect statements that must be addressed.
First, let’s clear up an important definition: “blind” testing does not relate to the use of the visual system; it means certain *information* about the test is withheld from the subject. Because we often use the visual system to obtain information, and because we often borrow visual terms for lack of more general terms (the sound is “bright” or “colored”), “blind” is used. Note that an ABX test where the subject is blindfolded but your software announces “A is playing” or “B is playing”, even for the X playback, is “sighted”, while an open-eyed lights-on see-everything EXCEPT which is playing is “blind”. I know you already know this, but your arguments about the importance of multisensory integration for perception (my field) don’t extend to “blind testing”. Blind testing in no way inhibits multisensory integration.
Second, trained scientists are like a master carpenter or chef: one expects their outcomes/results to be better than a lay-person, but that is not always so. Some lay people obtain excellent results, while results from “professionals” are often lacking. There is nothing magical about doing it right. As you point out, one must know which pitfalls to avoid, and avoid them. In science, this is true not only for experimental design and execution, but also in critical evaluation of others’ work. Which lead to…
Third, there is no symmetry between sighted and blind tests, as you claim:
No. Of course the quality of any design depends completely upon what question(s) you are hoping to answer. But when testing audibility of differences between {devices, sources, formats,…}, choosing to make the test sighted is always a flaw/weakness. You confound your results with known biases, and therefore complicate interpretation of results by adding variables. Perhaps this results in false positives, but it can also lead to false negatives, depending on exactly what you are testing.
Making the blind/sighted choice is independent of other choices.
Choices of music, duration, time of day, number of trials, etc. can/should be made independent of blind/sighted. You can’t disparage blind testing just because the other choices made are not to your liking.
Also, I would argue that an ABX blind test can’t give you a false negative. It is not designed to give any info on negatives. It can give a true positive, a false positive (with p giving you the liklihood of this) and a “failure to demonstrate the effect”. I know you believe (probably correctly) that many people interpret a test with p>0.05 as a negative result, but that is their problem in interpretation, not a problem with the test. A p>0.05 result is simply an “I don’t know”. The fact that SETI has not received any evidence of life on other planets does NOT show there is none, even though MANY people have participated.
But that does not mean the test should never be done sighted. All tests have flaws, some grave, some negligible. In the design, you have to choose your trade-offs. For example, although choosing to make a test "blind" won’t weaken or damage the results, it may make the test prohibitively expensive, time-consuming, or difficult. I can’t see many people going through the effort of duplicating Harmon’s setup just so they can test two sets of speakers before buying. You may just have to accept the flaw of being sighted.
I know you will keep challenging objectivists who have closed minds, as well you should, but you may want to reevaluate some of your criticisms.
Happy listening!
Cheers,
SAM
I often like your posts, not because I often agree, but because your role is that of being skeptical of the “skeptic side”. You don’t fear science, but rather embrace some interesting ideas such as auditory scene analysis, and the need to know *how* to measure something. But for years (and in this thread), you have made some incorrect statements that must be addressed.
First, let’s clear up an important definition: “blind” testing does not relate to the use of the visual system; it means certain *information* about the test is withheld from the subject. Because we often use the visual system to obtain information, and because we often borrow visual terms for lack of more general terms (the sound is “bright” or “colored”), “blind” is used. Note that an ABX test where the subject is blindfolded but your software announces “A is playing” or “B is playing”, even for the X playback, is “sighted”, while an open-eyed lights-on see-everything EXCEPT which is playing is “blind”. I know you already know this, but your arguments about the importance of multisensory integration for perception (my field) don’t extend to “blind testing”. Blind testing in no way inhibits multisensory integration.
Second, trained scientists are like a master carpenter or chef: one expects their outcomes/results to be better than a lay-person, but that is not always so. Some lay people obtain excellent results, while results from “professionals” are often lacking. There is nothing magical about doing it right. As you point out, one must know which pitfalls to avoid, and avoid them. In science, this is true not only for experimental design and execution, but also in critical evaluation of others’ work. Which lead to…
Third, there is no symmetry between sighted and blind tests, as you claim:
It boils down to two methods of evaluation - both of which can skew the results but in completely opposite directions - sighted towards false positives & blind towards false negatives.
No. Of course the quality of any design depends completely upon what question(s) you are hoping to answer. But when testing audibility of differences between {devices, sources, formats,…}, choosing to make the test sighted is always a flaw/weakness. You confound your results with known biases, and therefore complicate interpretation of results by adding variables. Perhaps this results in false positives, but it can also lead to false negatives, depending on exactly what you are testing.
Making the blind/sighted choice is independent of other choices.
I have never seen blind ABX testing repeated over a week with different music, at different times of the day, etc I have always seen a one shot test of maybe 16 or 20 trials. Maybe repeated a couple of times but that tends to be it.
Choices of music, duration, time of day, number of trials, etc. can/should be made independent of blind/sighted. You can’t disparage blind testing just because the other choices made are not to your liking.
Also, I would argue that an ABX blind test can’t give you a false negative. It is not designed to give any info on negatives. It can give a true positive, a false positive (with p giving you the liklihood of this) and a “failure to demonstrate the effect”. I know you believe (probably correctly) that many people interpret a test with p>0.05 as a negative result, but that is their problem in interpretation, not a problem with the test. A p>0.05 result is simply an “I don’t know”. The fact that SETI has not received any evidence of life on other planets does NOT show there is none, even though MANY people have participated.
But that does not mean the test should never be done sighted. All tests have flaws, some grave, some negligible. In the design, you have to choose your trade-offs. For example, although choosing to make a test "blind" won’t weaken or damage the results, it may make the test prohibitively expensive, time-consuming, or difficult. I can’t see many people going through the effort of duplicating Harmon’s setup just so they can test two sets of speakers before buying. You may just have to accept the flaw of being sighted.
I know you will keep challenging objectivists who have closed minds, as well you should, but you may want to reevaluate some of your criticisms.
Happy listening!
Cheers,
SAM