Propose your protocol here

Hello Mike,
Your protocol is mathematically quite difficult to handle.

The possible lists are not equiprobable. There are 2 chances out of 9 to get one of the 56 possible lists of 5xCardas / 3xRadioshack, 5 chances out of 9 to get one of the 70 possible lists of 4xCardas and 4xRadioshack, and 2 chances out of 9 to get one of the 56 possible lists of 3xCardas and 5xRadioshack.

The problem is to calculate the probability that any given list out of the 182 possible ones has a number of matches equal or superior to yours with the one got by the operator.

I'll post my protocol later, as soon as I have some time.

Yes, this is a correct method. You'll have to simulate the 10 little squares, and the drawing of 8 of them.

Don't forget to sum the probability of getting your score with the ones of getting even better score.

I still don't have the time to post my own protocol. But if you happen to read french, I've just posted a good deal of how I perform my own tests here : homecinema-fr.com • Voir le sujet - Le débat objectivistes/subjectivistes

I have a protocol. Take several people who strongly believe that they can hear differences between cables. They should be well-known to the audience you wish to convince (i.e. head-fi, for instance). They should be the most vocal.

Then invite them to a test. These people can use their own equipment (e.g. their own speakers, their own favorite cables, their own living room, their own CD collection), whatever they think they need to hear the difference between cables.

After it's all set up, switch their favorite cable with a Radio Shack cable, unblinded. If they DO hear a difference, then continue. If they do NOT hear a difference, you're done. Pack your bags and find another person.

Now, since they can hear a difference using their own conditions (i.e. room, CD's, equipment, and length of testing), the ONLY thing you change now is their knowledge of which cable is which.

Do exactly the same thing that allowed them to hear the difference previously. Switch the cables when the listener wants them switched, just as before, except place a blanket over the cables or cover them up so the listener can't see them.

Then do a pre-determined number of trials based on the expected alpha and beta desired. Then tally up the score.

Then repeat this experiment with everyone else who "believes." Definitely send an invitation to Noel Lee to participate.

One refinement I would suggest. Ask the believers to put up $50 of their own money, more if they are really vocal .

If they succeed they get the money back and big kudos, if not use the $50 for expenses...

I don't follow your explanation at all. A sighted portion is absolutely necessary. It acts as a positive control. A positive control demonstrates that your test is actually capable of showing what it needs to. Without a positive control, a listener who is unable to differentiate between two cables could object and say that the equipment/room/whatever masked his ability to hear differences between the cables. However, if he WAS able to hear differences before blinding (positive control) but WASN'T able to hear them after blinding (and doing nothing else to change the conditions), then we can say with great confidence once bias was removed (listener knowledge of cable identity), the differences between tested cables were not audible to the listener.

It would be a perfectly legitimate objection for a listener to say "the system/setup I listened to did not have sufficient resolution for me to hear the difference between cables. Cables have audible differences. I just couldn't hear them on this stupid system." It sounds ridiculous, but I know people will say this. HOWEVER, if you demonstrate to the listener that he can hear differences between cables (before all you do is cover up the identity of them), he cannot make this objection!

"smoothness" and "microdynamics" mean little if undefined and without a calibrated benchmark (a system arbitrarily defined as "5" for example).

That still doesn't make sense. If during the sighted test, people perceive MNOP differences (which aren't actually present and are a result of the placebo effect), there is absolutely nothing stopping them from hearing QRS differences (true audible differences in cables, should they exist), during the blinded portion. Listeners are not robots! You don't program them to hear just MNOP differences and nothing else afterwards. If you blind them and there are now true ABC or DEF audible differences, they will report them.

What you really mean is: if we are only interested in finding true differences between cables, we cannot permit any methodology that allows hallucinated differences to be counted as true differences. This is why we do BLIND listening tests. Blinding eliminates the ability of listener hallucinations (based on placebo effect) to affect the proper identification of truly audible differences.

A completely separate issue is whether or not our test apparatus (speakers, amp, cd's, etc., and of course the listener) is even capable of detecting differences between cables. If a blinded listener can correctly differentiate between two cables beyond guessing, then who cares if he can differentiate them unblinded. However, if a blinded listener can't tell the difference, then he would simply claim that the test apparatus is flawed and not of sufficient resolution to allow him to differentiate anyway (i.e. claim false negative). Once again, this is why it is imperative that you have a positive control - some way of verifying that the listener is able to discern differences between cables. The concept of a positive control is very basic and fundamental to experimental design. Perhaps you could read about it? If you have a better suggestion for a positive control, I'd like to know.

Your typical "believer" (once who believes cables sound different) has his cable hooked up in the living room and he knows the identify of it (after all, he bought the cable). It is under these circumstances that he has convinced himself that his cable affects sound quality. Therefore, it is only natural that he be allowed to compare his cable to a cheapo Radio Shack cable under identical conditions. The only thing we change is the BLINDING and nothing else.

Part of good experimental design is anticipating the objections your audience and reviewers will have and addressing them in your methodology or analysis before the objections can even be made. If you don't, a simple objection like the common one I pointed out will cause a large group of people to dismiss your results based on such a criticism. If you're not interested in designing a good experiment that will pass muster among people who capable of interpreting your experiment, then why do one? Just do your own test at home, using whatever methods you want, forget about statistics, and come to your own conclusion!

This is not necessarily true, and I am unaware of an evidence that requires this to be true.

[QUOTE[ If cable A is audibly brighter than B under the right conditions, but the test subject has a strong unconscious expectation that cable B is brighter, then that subject won't be able to perceive the real differences, because he will likely hallucinate that cable B is brighter.[/QUOTE]

Again, this is not necessarily true. A subject who expects cable B to brighter, yet hears that cable A is brighter, could certainly say "Well, I was convinced B was brighter, but A is definitely brighter to me." If differences among cables are as great as people claim they are, then they should have NO PROBLEM detecting them, REGARDLESS of what expectations they had going in.

Sure. If the subject indicates he can hear a difference under sighted conditions, he demonstrates the he has perceived R *AND/OR* H, to use your example. On the other hand, if the subject hears NO difference on the sighted test, then he has demonstrated that he can perceive NEITHER R *NOR* H. If a listener can perceive NEITHER R NOR H, then he is useless to thet study because we need people who can perceive R. Make sense? This is why we need the sighted test as a positive control.

The purpose of the BLINDED condition is to test if R is present. If a blinded listener can reliably hear a difference between two cables, he has detected R (but NOT H) by definition. If the BLINDED listener cannot reliably hear a difference, then he cannot detect R. Since the blinded listener passed the positive control and was thus able to hear R AND/OR H, but proved unable to hear R, we can conclude that he is only able to hear H, the hallucinated differences.

It's pretty simple and straightforward.

Your argument is that knowledge of which cable is which during a listening test will invoke hallucinated perceived differences that will later mask the listener's ablity to identify REAL differences during the blinded test. I personally see very little merit in this statement.

To be frank, if it takes you weeks to hear differences between cables, then chances are, you wouldn't notice the difference in a 1 hour listening session. I don't know about you, but I don't listen to music more than an hour a day, maybe two if I'm bored. If during my usual listening session I wouldn't hear the difference between cheapo cable A and expensive cable B (because it takes me hours or weeks to be able to tell reliably), why on earth would I spend more money on expensive cable B?????