wavoman
Headphoneus Supremus
- Joined
- Jan 19, 2008
- Posts
- 1,873
- Likes
- 45
Quote:
Hey I don't want to fight, but I am afraid you just don't understand modern experimental design, and "play the winner" testing.
You keep missing the point: we are trying to cater for the case where some people can really hear differences, but many can't. We are trying to find the golden ears that can hear the difference if the difference exists.. That is what we are trying to test.
If we can find anyone who can reliably, over and over, in true blind conditions, tell the cables apart, with enough trials that it can't be luck, then there IS a difference in cables -- for that person at least.
The key is this: if the "truth" is that cable differences can be heard by a select few, but not most people, then only the type of design I am talking about will uncover this truth. Testing large crowds will always fail to reject the "no difference" null hypothesis.
Throwing out those who report no difference does NOT make the test worthless. Yea, in the old days, that was called biasing the test, but not when it is done correctly, only when it is done in a non-thinking fashion and you end up with -- by accident, by luck -- people who seem to be able to tell the difference but can't really. That's called the selection fallacy. But we know all about that and can guard against it with proper design and model analysis.
Look, I really know this stuff, please. Trust me, my technique does not invalidate the test. It is correct. OK I know that people hate it when I bring up degrees, but I feel I have no choice here: I have a PhD in Statistics from Yale, AB summa cum laude in Statistics from Princeton, won the American Statistical Associaton Theory and Methods Prize, and the classic multivariate two-sample nonparametric test is named for me. I'm in the textbooks. PM me and I will send you the references, or my name and you can Google.
Here is another way to think about it. I make the claim: "Nobody can bench press 400 pounds, it is impossible". We get a roomful of people off the street, and you know what? None of them can. Then, instead, we assemble an even bigger crowd, we pre-screen people and ask them to bench 300 pounds, and we tell the ones who can't to go home. We play the winners, and indeed, when we increase the weight to 400 pounds a few guys do it and my claim is rejected.
However sensory evaluation is harder, because we cannot verify by watching merely one time that people can tell a difference, unlike lifting weights. But with blind testing, replicated enough times, and with some swindles thrown in for good measure, we isolate the golden ears, and maybe maybe we find exotic cables make a difference for the select few.
Makes sense, no?
Originally Posted by ILikeMusic /img/forum/go_quote.gif And in doing that you're throwing out those who can't hear a difference and keeping those that can, making the results totally worthless. That's the whole point. |
Hey I don't want to fight, but I am afraid you just don't understand modern experimental design, and "play the winner" testing.
You keep missing the point: we are trying to cater for the case where some people can really hear differences, but many can't. We are trying to find the golden ears that can hear the difference if the difference exists.. That is what we are trying to test.
If we can find anyone who can reliably, over and over, in true blind conditions, tell the cables apart, with enough trials that it can't be luck, then there IS a difference in cables -- for that person at least.
The key is this: if the "truth" is that cable differences can be heard by a select few, but not most people, then only the type of design I am talking about will uncover this truth. Testing large crowds will always fail to reject the "no difference" null hypothesis.
Throwing out those who report no difference does NOT make the test worthless. Yea, in the old days, that was called biasing the test, but not when it is done correctly, only when it is done in a non-thinking fashion and you end up with -- by accident, by luck -- people who seem to be able to tell the difference but can't really. That's called the selection fallacy. But we know all about that and can guard against it with proper design and model analysis.
Look, I really know this stuff, please. Trust me, my technique does not invalidate the test. It is correct. OK I know that people hate it when I bring up degrees, but I feel I have no choice here: I have a PhD in Statistics from Yale, AB summa cum laude in Statistics from Princeton, won the American Statistical Associaton Theory and Methods Prize, and the classic multivariate two-sample nonparametric test is named for me. I'm in the textbooks. PM me and I will send you the references, or my name and you can Google.
Here is another way to think about it. I make the claim: "Nobody can bench press 400 pounds, it is impossible". We get a roomful of people off the street, and you know what? None of them can. Then, instead, we assemble an even bigger crowd, we pre-screen people and ask them to bench 300 pounds, and we tell the ones who can't to go home. We play the winners, and indeed, when we increase the weight to 400 pounds a few guys do it and my claim is rejected.
However sensory evaluation is harder, because we cannot verify by watching merely one time that people can tell a difference, unlike lifting weights. But with blind testing, replicated enough times, and with some swindles thrown in for good measure, we isolate the golden ears, and maybe maybe we find exotic cables make a difference for the select few.
Makes sense, no?