Blind cable testing: initial report

[mike1127]

Quote:

Originally Posted by Dane /img/forum/go_quote.gif Oh, you mean that you always hear the "rat" cable before the "snake" cable, in case A and B are fixed and known. Yeah, that would add bias. Good point (if that was indeed what you meant )

I'm not sure if you are being sarcastic. It's what I said in the first post. Our ears and brains are measuring instruments which are subject to "noise." Knowing a cable's identity adds noise to the "measurement."

 

[Dane]

Quote:

Originally Posted by mike1127 /img/forum/go_quote.gif Now let's call ABX a "measurement instrument." You are introducing "noise" into the "measurement" when you allow the listener to know the identity of A and B, by the objectivists own reasoning.

I believe this is wrong if the subject can switch between A, B and X freely before deciding X=A or X=B. However, since your protocol (for practical reasons) doesn't allow you this freedom, yes, I guess not randomizing A and B would add noise, since each of your trial sequences always start with AB.

 

[Dane]

Quote:

Originally Posted by mike1127 /img/forum/go_quote.gif I'm not sure if you are being sarcastic.

I was not.

 

[mike1127]

Quote:

Originally Posted by Dane /img/forum/go_quote.gif I believe this is wrong if the subject can switch between A, B and X freely before deciding X=A or X=B. However, since your protocol (for practical reasons) doesn't allow you this freedom, yes, I guess not randomizing A and B would add noise, since each of your trial sequences always start with AB.

My reasoning is that knowing what A and B are is what adds the "noise" (bias). It doesn't matter, in this case, whether you can switch freely between them. I don't see how that affects the argument.

 

[Dane]

Quote:

Originally Posted by mike1127 /img/forum/go_quote.gif My reasoning is that knowing what A and B are is what adds the "noise" (bias). It doesn't matter, in this case, whether you can switch freely between them. I don't see how that affects the argument.

If you, in each trial sequence, always hear the "poor" cable before the "good" cable, I can see why that could skew the test results. I thought this was what you meant and found it reasonable. It seems though, that you have a different kind of noise in mind, but I'm afraid I don't understand what you mean then.

Edit: Let's move on. For whatever reasons, given your protocol, knowing the identity of A and B adds noise. Accepted.

 

[mike1127]

Quote:

Originally Posted by Dane /img/forum/go_quote.gif If you, in each trial sequence, always hear the "poor" cable before the "good" cable, I can see why that could skew the test results. I thought this was what you meant and found it reasonable. It seems though, that you have a different kind of noise in mind, but I'm afraid I don't understand what you mean then. Edit: Let's move on. For whatever reasons, given your protocol, knowing the identity of A and B adds noise. Accepted.

Well, I'll answer anyway in case someone else cares. By the way, it's not just "my protocol."

Lets say we have two devices X and Y and you listen to them blind, and evaluate each one on a quality scale of 1 to 10. You rate X=5 and Y=6.

Now imagine that your friend comes along and says, "I want you to hear two more devices, W and Z!" Again you'll listen blind. Unknown to you W is actually X and Z is actually Y. This time, before listening to W, your friend says, "This is the most amazing device ever!"

So you listen to W and rate it 8. Previously under conditions in which no one tried to influence you, you rated it 5. Now there is another influence---your friend whipping up excitement.

Your friend also says, "Now listen to Z, but I have to warn you, it is total crap." So you listen to Z and rate it 3. Previously you rated it 6.

So expectations can influence your perception. We all know that. My claim is that you are a more accurate measuring instrument if you don't have other factors influencing your response. The first time you rated X=5 and Y=6 was probably the most accurate indication. Under other conditions, you will change your rating. I refer to those other conditions as "noise" in the measurement.

Knowing the identity of A and B---for instance knowing that A is a cheap cable and B is a beautiful, expensive one---is similar to having a friend whip up your excitement or turning you off.

 

[wavoman]

Quote:

Originally Posted by mike1127 /img/forum/go_quote.gif Wrong. That's why the order of presentation is randomized. It eliminates response bias in the null hypothesis. And in my trials, the order of the first two sub-trials and last two sub-trials are randomized independently. One reason they are independent.

Ouch, you really don't understand. Please, enough already!

Randomization does not eliminate response bias.

I think I now realize that you have not read up on either response bias or statistical power.

Give it up -- you do not understand sensory testing.

You have taken, it seems, an elementary stats course and are repeating the most basic ideas from binomial hypothesis testing. This has so little to do with the problem at hand.

You know I'm a well-known statistician, right? With top rank publications, prizes, tests named after me, etc. Do not say "wrong" to me -- you are not in my league here. You are now embarassing yourself.

I applaud your testing efforts. You need to get help from professional statisticians, however, with sensory testing experience, if you want to analyze them correctly.

 

[mike1127]

Quote:

Originally Posted by wavoman /img/forum/go_quote.gif Give it up -- you do not understand sensory testing.

Why don't you "give it up." What you have been doing is arguing by authority. I have presented my arguments in a logical way, and they could be answered in a logical way. But you choose simply to tell me I'm wrong.

Quote:

You know I'm a well-known statistician, right? With top rank publications, prizes, tests named after me, etc. Do not say "wrong" to me -- you are not in my league here. You are now embarassing yourself.

If you are a well-known statistician, then you should be able to answer the following argument. I would like to see the answer. With math. And no condescension. Got that? No condescension.

Let's take the case of a test with two answers to each trial, A or B. In the null hypothesis, the subject cannot hear the sound, and so is answering randomly, with some bias. Let R_a be the probability that the subject answers A. Let R_b = 1 - R_a.

R_a and R_b are a model of response bias.

If we randomize the "actual answers" of the test, then 50% of the time the actual answer is A and 50% it's B.

Let N be the number of trials.

The subject answers "A" this many times: N * R_a. In that group of answers, we expect that half are right and half are wrong, because the actual answers are randomized. So the subject gets 0.5 * N * R_a right answers.

Likewise, when the subject answers "B", 0.5 * N * R_b are right answers. So the expected number of right answers is

0.5 * N * R_a + 0.5 * N * R_b = 0.5 * N * (R_a + R_b) = 0.5 * N.

In other words, we expect, in the null hypothesis, the subject to get half of them right. Whatever the response bias is.

Now go to it. Tear this apart. But do it with math and without condescension and I might take you seriously.

 

[wavoman]

Mike -- ygpm. No need to drag everyone else in to this.

The issues lie elsewhere, not in the 50-50 nature of the null hypothesis.

Privately we can together dig in to this at great length. Happy to do so.

And I did not intend to be condescending -- I was angry at your naked "wrong".

I apologize for that. One should never email or post when angry.

BTW I really did win the Theory and Methods Award from the American Statistical Association, and the test I invented (and is named after me) is on the Wikipedia page for Multivariate Normal Distribution. I really do have a PhD in Statistics. No BS. And I am not arguing by appeal to authority, just asking you to think deeper, because you should conclude I probably know what I am talking about.

Again, as I said, I applaud your testing. However, I see difficulties in your design and analysis, and (in PMs, not here) let me work harder to explain them to you.

 

[Real Man of Genius]

Why not educate the masses? I would be interested in your discussion.

 

[ezzieyguywuf]

Quote:

Why not educate the masses? I would be interested in your discussion.

+1 to that

Quote:

I really do have a PhD in Statistics. No BS

Don't you usually get a Bachelor of Science (BS) before a PhD? :-D

 

[Pio2001]

Quote:

Originally Posted by wavoman /img/forum/go_quote.gif and the test I invented (and is named after me) is on the Wikipedia page for Multivariate Normal Distribution.

OMG

! Carl Gauss !

 

[royalcrown]

Quote:

Originally Posted by ezzieyguywuf /img/forum/go_quote.gif Don't you usually get a Bachelor of Science (BS) before a PhD? :-D

Not to speak for wavoman, but at least at the college I attend a mathematics degree is a BA, and (I think) a stats major is also a BA. The BA/BS distinction depends on where you go to college, which school you are enrolled in (ArtSci vs B-School vs. Engineering), and your coursework (normally BA's have more rounded liberal arts degrees and BS's are more in-depth with less classes in other areas).

Quote:

Originally Posted by Pio2001 /img/forum/go_quote.gif OMG ! Carl Gauss !

LOL

 

[sohels]

Quote:

Originally Posted by wavoman /img/forum/go_quote.gif The lack of independence is between the answers, not between the first two and the second two listenings. Whether or not you can tell which is the good or bad cable is very related to whether or not you can tell which two samples match. Think this way: if the effect of the cables was so dramatic that everyone could instantly match them, they could also instantly tell you which was which. Now think this way: if the cables were really identical, then nobody could answer either question correctly.

I shall attempt to make this more explicit: In general statistical usage, correlation refers to the departure of two random variables from independence. Hence, If the variables are independent then the correlation must be zero - but as demonstrated above, there exists a strong positive correlation between the answers to question one and two - and thus they cannot be independent.

Is this correct?

 

[wavoman]

sohels-- you are exactly correct

royalcrown -- ezzie was making a joke. A pretty damn clever one. Of coure I meant "no bulls**t", not "I don't have a BS". Yes, my undergraduate degree is in Statistics too, and at the school I went to, they are very strange and call the bachelor's degree an AB, because that is closer to the original Latin, and the school is very stuck up. I used to ignore this and just say I have a BA (since AB sounds like a 2-year "Associates Degree"), but I would hear from a**hole alumni who would say "our dear old alma mater grants AB's, not BA's".

Pio's Gauss joke is also very funny. I feel that old, believe me!

I will try to write a Word doc with all the ins and outs of response bias etc. Mike1127 is making the correct probability calculation, but still cannot claim n=8. If you PM me your email address I will email it to you when I am done. Give me a week or so -- the NJ meet is this Saturday and it will consume my time.

I did not intend for my post to be condescending, only angry, and I apologize again. Condescension is not part of my nature, or if it is, I want to kill it. Anger is different, that I can live with. Although I repeat: one should not email or post when angry. Let me add an "IMHO" in case that last statement appears condescending!