Quote:
Originally Posted by Nightmare /img/forum/go_quote.gif
..."Difference, no preference" is a safe and ego-soothing response for subjects to give if they don't actually hear a difference. My concern is that wavoman's protocol may invite a glut of these responses, which will teach us nothing.
|
Agreed ... but subjects who give this answer all the time, or most of the time, when presented with the swindle will have smoked themselves out and be eliminated.
BTW, all results will be confidential and the swindles will not embarrass anyone. Privately people will want to know "how they did", and we simply say "you were not able to distinguish these two cables".
The whole point of this experiment is to find out if there are people who can tell the difference between the two cables -- especially in light of the fact that most of us believe there is no difference between in-spec digital cables transmitting S/PDIF, and that spending $500 on such cables is not smart.
Statistical tests and the chance/guessing effect play no real role here. The binomial probability test is a red-herring, and although used in most of the published AES articles, is really meaningless here. Averaging the results of the sample group is pointless. Someone by chance may get a great score -- you have to test him again. So let's do this directly -- focus on the subjects who demonstrate they might be golden ears, or at least can resist the social pressure to find a difference (by seeing what they say when there is no difference -- the swindles).
If this effect exists -- if high-end digital cables sound better to someone -- then there is that someone. And he or she will be able to hear this difference almost all the time. Not some of the time, or "better than chance" ...
most of the time, nearly all the time in fact, or there is no effect. There are no random elements here -- one piece of music, played over and over, through both cables.
I can either read the line on the eye chart, or I can't. OK, sometimes I miss a letter or two, it's fuzzy, that's why the eye doc does not expect you to get them all -- but only on the transition line. So it is with hearing. There might be someone who can hear a difference right on the edge of his perception ... but if this effect exists, then there is someone who can hear it (nearly) all the time.
I want to find that person, and estimate how rare that person is in the population. Play-the-winner experiments do just that. No statistical formulas will be necessary when we happen upon Mr. Goodears, and he aces the test, over-and-over. OK OK, we will use probability just to prove guessing is most likely impossible, but that's a trivial calculation.
Large sample sizes do not eliminate the pollution of the masses, in fact they exacerbate it. You must track individual results, not pool them.
Finally, let's think about A/B/X vs my 1-2-3-4 "prefer" testing.
1-2-3-4 is a refinement (in the technical sense of that word, i.e., it preserves the original choices but presents them in finer detail) of "difference vs no-difference". I am happy with either doing difference vs no-difference testing, or preference testing, but I beleive (I have not proven this experimentally, but I do have a mathematical model that backs me up, however the model has not been verified) that 1-2-3-4 will induce less response bias than "difference vs no-difference". As I said though, I'm OK with "difference vs no-difference" as long as we have swindle.
But A/B/X is NOT a difference vs no-difference test. Not close. A/B/X plays A, plays B, then picks one at random and asks "is it A or B". What the hell does that have to do with preferring A over B, or detecting a difference?
It is unnatural, and to my mind so stupid as to make me question why people keep doing it over and over -- answer: because they can do it double blind with the A/B/X hardware, but who cares. IMO, A/B/X has set back the science of human panel-based audio perception testing and damaged it nearly totally. Think hard, people! Why A/B/X? Why? If you want to know if two things sound different, play them and ask. Yes, this can only be done single blind, but double blind testing is not needed if the subjects are out-of-communication with the testers, especially if the testers are unbiased. With medicines, you HAVE TO do double blind, since the doctors are involved in the treatment. Not the case with audio. Think for yourself -- this is all obvious if you start from scratch with an open mind and don't say "DBT" and "A/B/X" just 'cause everyone else does. Think.
It's not worth arguing over "difference / no-difference" vs "prefer?" testing. I like "prefer?" since I think it reduces response bias, but I can't prove that. If you want to do "difference / no-difference", I'm not against you ... as long as you use some swindles to help stratify the population ... heck, I won't even fight you if you don't use swindles as long as you don't pool the results, as long as you look for winners and then repeat the test with them, etc.
But A/B/X? Nope. Unnatural, and the wrong question is being tested.
