Let's Prove The Null Hypothesis

[wavoman]

Quote:

Originally Posted by upstateguy /img/forum/go_quote.gif NY meet in Queens next month...our source situation is under control. I am as curious as anyone else to see if I can tell the difference between cables. Snake oil salesmen have been making a fortune for thousands of years. Don't you want to see for yourself?

OK Eric we'll do it. I don't know the NY movers and shakers so I just have to get there early and get table space. You and I at least will SBT digital coax cables. Let's do USB cables too, with your source we can manage that.

I did not correctly describe the NJ meet. SACD Man did in fact do SBT testing of mini2mini cables, using an iPod Line Out + iCube and good phones (701s I believe). The A/B was copper vs silver mini2mini, both high-end (built by him). The test was done the way I like -- plenty of samples, plenty of time. I was reliably able to tell the two apart, but I had no preference, and in fact mis-identified the cables according to the usual folklore: silver is cold/analytical, copper is warm. I called the silver one the copper one (i.e. I thought it was "warm and lush"), and of course visa-versa. Still, there was an audible difference, no doubt. And I could, with very little error, classify the samples correctly into two camps.

But there were no swindle comparisons, and no organized records kept. We'll do that in NY, yah?

"Honey, can I borrow the Denons?". How do I tell my wife that I'm spending my 60th Birthday weekend with headphone guys instead of her (I don't need another ex ... although neither make very good pancakes).

 

[jonathanjong]

Yea, I get it that within subjects designs are more powerful and get rid of some annoying individual difference effects, but (a) if the samples are large enough, and the allocation is random, the individual differences should even out, and (b) others in this thread have raised issues with within-subjects designs in the context of audio testing.

I think it's worth doing on a random sample of the general population, not just audiophiles or alleged audiophiles. I mean, not knowing whether I have golden ears or not, there's no reason for me to assume that I'm any different from the masses. Most consumers are in my position.

 

[jonathanjong]

Quote:

Originally Posted by wavoman /img/forum/go_quote.gif "Honey, can I borrow the Denons?". How do I tell my wife that I'm spending my 60th Birthday weekend with headphone guys instead of her (I don't need another ex ... although neither make very good pancakes).

AH HAH! But do they use metal or wood?

 

[Nocturnal310]

I have ears and i can hear the difference.

isnt that a proof enuff?

 

[ILikeMusic]

Quote:

If this were fine wine vs cheap wine, the masses would get it wrong and wine experts would get it right. This has been proven in blind tests.

But I thought that blind tests don't mean anything?

I think you are confusing two different kinds of tests. If one is testing food or televisions then you are specifically looking for a subjective response and thus you might care about things like preference, but if one is looking for a simple objective answer to the question of whether a subject can detect a difference or not then preference doesn't matter and asking about it will only needlessly confuse the results. The questions 'which cola do you prefer' and 'can you hear a difference between copper and silver conductors' are two completely different things.


Quote:

I prefer A I prefer B I do not detect a difference I detect a difference but have no preference

In this example the subject could simply use answer four and never really demonstrate whether they could detect a difference or not, rendering the test utterly pointless towards the end of proving whether a difference can really be resolved. This seems so obvious I wonder if you meant to word answer four a little differently?


Quote:

The A/B was copper vs silver mini2mini, both high-end (built by him). The test was done the way I like -- plenty of samples, plenty of time. I was reliably able to tell the two apart, but I had no preference, and in fact mis-identified the cables according to the usual folklore: silver is cold/analytical, copper is warm. I called the silver one the copper one (i.e. I thought it was "warm and lush"), and of course visa-versa. Still, there was an audible difference, no doubt. And I could, with very little error, classify the samples correctly into two camps.

I want to be sure that I understand you correctly. Are you saying that the only difference between the test cables was whether the conductors were made of copper or silver, and that you could identify them based on this difference alone?

.

 

[upstateguy]

Quote:

Originally Posted by Nocturnal310 /img/forum/go_quote.gif I have ears and i can hear the difference. isnt that a proof enuff?

Hi Nocturnal

Here's my take on this:

You have ears,

You think you can hear a difference,

You may have some nice cables,

You expect to hear a difference,

You believe you will hear a difference,

So you hear a difference.

Now, if you were to sit down for a few minutes, at Wavo's setup, with your cables (or his) and your headphones, and found you could reliably hear differences between cables, I think you could be confident that the differences you report hearing are not imaginary or placebo. Until then, you really just don't know, do you?

On the other hand, there will be those who may not wish to know because they have too much invested in their particular belief system and are fearful that they may not actually be able to hear the differences they have claimed. These people will continue along blindly believing. It's a free country and you can do that. No one will think the worse of you for choosing not to know.

Now, there is no reason why someone shouldn't, couldn't, or wouldn't use a nicer cable even if there is no sonic advantage that they can hear.

I have a 1M glass optical cable running from one of my computer rigs to my Stello and a much longer Dayton plastic optical cable running from an identical rig further away, into the same Stello. I can't tell them apart, but the glass cable looks nicer, has a fabric like covering material instead of plastic and I like it....

USG

 

[PhilS]

Quote:

Originally Posted by jonathanjong /img/forum/go_quote.gif I think it's worth doing on a random sample of the general population, not just audiophiles or alleged audiophiles. I mean, not knowing whether I have golden ears or not, there's no reason for me to assume that I'm any different from the masses. Most consumers are in my position.

I'm curious about this. Wouldn't it be better to use people who have some experience with listening to music on decent quality systems? As an analogy, if you were trying to determine whether two wines tasted different, would you not want the test population to consist of people who had some experience drinking wine? I don't drink wine, for example, and I think I would have a very difficult time telling the difference between two wines that someone else who drinks wine regularly (even if he does not claim to have a "golden palate") could readily distinguish. Or does this skew the result somehow?

 

[PhilS]

Quote:

Originally Posted by upstateguy /img/forum/go_quote.gif Hi Nocturnal Here's my take on this: You have ears, You think you can hear a difference, You may have some nice cables, You expect to hear a difference, You believe you will hear a difference, So you hear a difference. Now, if you were to sit down for a few minutes, at Wavo's setup, with your cables (or his) and your headphones, and found you could reliably hear differences between cables, I think you could be confident that the differences you report hearing are not imaginary or placebo. Until then, you really just don't know, do you?

USG, let me ask you this.

I have ears.

I have a really nice home system.

I expect to hear a difference between my home system and my car radio.

I believe I do hear a difference.

Would you say I really don't know if one sounds better than the other until I do a bind test?

If your response is that the magnitude of the difference is greater with a car radio and a home system, I would ask you which differences are of such a magnitude that you think someone has to do blind test before they can really know whether there is a difference.

I suppose you would use alleged differences between cables as one example. But what about the difference between amps? The difference between an MP3 player and a good home system? The difference between CD players? The difference between a headphone that is burnt in and one that is not?

In your view, does someone have to do a blind test in all of these instances to really know whether something sounds different?

I guess my point is that there are differences or types of differences that can certainly be questioned and those that cannot (i.e., the difference between a car radio and a home system) and people might reasonably drawn the line in different places as to what is "obvious,"

Quote:

Originally Posted by upstateguy /img/forum/go_quote.gif On the other hand, there will be those who may not wish to know because they have too much invested in their particular belief system and are fearful that they may not actually be able to hear the differences they have claimed. These people will continue along blindly believing. It's a free country and you can do that. No one will think the worse of you for choosing not to know.

On the other hand, there will be those who are pretty intelligent people who are skeptical of snake oil, claimed audible differences, etc., who have heard an audible difference between certain components or whatever (sometimes by accident) who determine that the difference they hear is obvious enough to them and/or significant enough to them that it is not worth conducting a blind test -- with all that is entailed in doing a proper one -- to see if what their senses tell them is correct.

If I find a new Chinese restaurant and I like their Kung Pau chicken better than I remembered at the previous one I went to, I don't conduct a blind test to see if it really is better before I make it my new favorite Chinese restaurant.

In other words, some people make judgments based on the best information reasonably available to them, and considering matters of expediency, efficiency, and/or practicality, and not because they "may not wish to know."

And their belief is not "blind," it's just not supported to the full extent that you would like it to be supported it it were you (or at least so you claim).

And as far as "No one will think the worse of you for choosing not to know," yes they will. This forum (and not just this thread) is filled with veiled insults about how such people believe in aliens, fall for "psychobabble," etc.

 

[wavoman]

Quote:

Originally Posted by ILikeMusic /img/forum/go_quote.gif But I thought that blind tests don't mean anything? ...I think you are confusing two different kinds of tests. If one is testing food or televisions then you are specifically looking for a subjective response and thus you might care about things like preference, but if one is looking for a simple objective answer to the question of whether a subject can detect a difference or not then preference doesn't matter.

In a word: nope.

Of course I believe in blind tests, that's what this is all about. Wine, food, audio ... it is about avoiding repsonse bias.

You are simply wrong about stressing a difference vs a preference. Years of research in sensory testing has shown this. Asking for a preference can, in some situations, avoid a type of bias that asking for a difference introduces.

Google "statistical tests sensory evaluation" and do some research if you are so inclined.

We want people who are not serious to take the cop-out answer #4. We might drop them out of the study as things continue ... the experimental design caters for this. That's the whole point.

Anyway there is no reason why a 4-choice refinement of your 2-choice question can hurt in any way. No one with an IQ above 80 would be confused by my 4 questions.

 

[ILikeMusic]

Quote:

I guess my point is that there are differences or types of differences that can certainly be questioned and those that cannot (i.e., the difference between a car radio and a home system) and people might reasonably drawn the line in different places as to what is "obvious,"

Yes, that's true in general, but one has to consider the reasonability of where the line is drawn. When someone says that their home system sounds better than their car system there's usually not a lot of criticism because it's quite possibly the case. When someone claims to be able to casually differentiate between 320 kbps and lossless the claim is a good deal less likely since that ability is rather uncommon, but it does remain possible nonetheless as a select few can positively demonstrate the capability. However, as one approaches the realm of physical impossibility (such as claiming that they can hear a difference based solely on the type of conductor) then the game kind of changes and at that point outright disbelief is quite legitimate and bound to occur.

.

 

[ILikeMusic]

Quote:

Originally Posted by wavoman /img/forum/go_quote.gif We want people who are not serious to take the cop-out answer #4. We might drop them out of the study as things continue ... the experimental design caters for this. That's the whole point.

And in doing that you're throwing out a group of respondents who can't identify a difference, skewing the results. That's the whole point.

Anyway, no point in going back and forth regarding who does and does not understand the basic premise here, best to let the viewers decide at this point. Thanks for the discussion.

.

 

[wavoman]

Quote:

Originally Posted by ILikeMusic /img/forum/go_quote.gif I want to be sure that I understand you correctly. Are you saying that the only difference between the test cables was whether the conductors were made of copper or silver, and that you could identify them based on this difference alone?

No, they were totally different cables that SACD Man sells. He said one was primarily silver and one was primarily copper, but surely there were lots of other differences as well ... had to be, since I could tell them apart (so would you have been able to, I'm sure). Test was totally blind; I could not see the two cables.

Worth trying to repeat, and we will.

 

[ILikeMusic]

Quote:

He said one was primarily silver and one was primarily copper, but surely there were lots of other differences as well ... had to be, since I could tell them apart

Whew... glad to hear you say that...

Cheers,

 

[upstateguy]

Quote:

Originally Posted by PhilS /img/forum/go_quote.gif USG, let me ask you this.....

Phil, you are a professional arguer so I will decline being drawn into one. You are entitled to your beliefs and your preferences and I'll not argue with you about them.

If, however, you become curious as to whether or not you are able to hear subtle differences between cables, spend a few minutes with us at the next NY meet.

Jude will be there, why not you too?

USG

 

[jonathanjong]

Quote:

Originally Posted by PhilS /img/forum/go_quote.gif I'm curious about this. Wouldn't it be better to use people who have some experience with listening to music on decent quality systems? As an analogy, if you were trying to determine whether two wines tasted different, would you not want the test population to consist of people who had some experience drinking wine? I don't drink wine, for example, and I think I would have a very difficult time telling the difference between two wines that someone else who drinks wine regularly (even if he does not claim to have a "golden palate") could readily distinguish. Or does this skew the result somehow?

Depends on what you're interested in. A lot of people here want to know if there "really is" a difference between systems. I'm not, really. I want to know if people can tell a difference between two systems. I don't care if a difference exists if it's only discernible by special groups of people. But perhaps that makes puts me in the minority here.