Let's Prove The Null Hypothesis

[wavoman]

Quote:

Originally Posted by ILikeMusic /img/forum/go_quote.gif And in doing that you're throwing out those who can't hear a difference and keeping those that can, making the results totally worthless. That's the whole point.

Hey I don't want to fight, but I am afraid you just don't understand modern experimental design, and "play the winner" testing.

You keep missing the point: we are trying to cater for the case where some people can really hear differences, but many can't. We are trying to find the golden ears that can hear the difference if the difference exists.. That is what we are trying to test.

If we can find anyone who can reliably, over and over, in true blind conditions, tell the cables apart, with enough trials that it can't be luck, then there IS a difference in cables -- for that person at least.

The key is this: if the "truth" is that cable differences can be heard by a select few, but not most people, then only the type of design I am talking about will uncover this truth. Testing large crowds will always fail to reject the "no difference" null hypothesis.

Throwing out those who report no difference does NOT make the test worthless. Yea, in the old days, that was called biasing the test, but not when it is done correctly, only when it is done in a non-thinking fashion and you end up with -- by accident, by luck -- people who seem to be able to tell the difference but can't really. That's called the selection fallacy. But we know all about that and can guard against it with proper design and model analysis.

Look, I really know this stuff, please. Trust me, my technique does not invalidate the test. It is correct. OK I know that people hate it when I bring up degrees, but I feel I have no choice here: I have a PhD in Statistics from Yale, AB summa cum laude in Statistics from Princeton, won the American Statistical Associaton Theory and Methods Prize, and the classic multivariate two-sample nonparametric test is named for me. I'm in the textbooks. PM me and I will send you the references, or my name and you can Google.

Here is another way to think about it. I make the claim: "Nobody can bench press 400 pounds, it is impossible". We get a roomful of people off the street, and you know what? None of them can. Then, instead, we assemble an even bigger crowd, we pre-screen people and ask them to bench 300 pounds, and we tell the ones who can't to go home. We play the winners, and indeed, when we increase the weight to 400 pounds a few guys do it and my claim is rejected.

However sensory evaluation is harder, because we cannot verify by watching merely one time that people can tell a difference, unlike lifting weights. But with blind testing, replicated enough times, and with some swindles thrown in for good measure, we isolate the golden ears, and maybe maybe we find exotic cables make a difference for the select few.

Makes sense, no?

 

[ILikeMusic]

Yes, I understand and I am not saying that pre-qualification in itself disqualifies the results, as long as you're not looking for results that are representative of the general public (and in this case indeed we are not.) But bringing it back to the realm of audio equipment specifically, most of the (properly conducted) tests I've seen do involve some pre-qualification in that the subjects are usually aficionados, be it test groups analyzing encoders (who have a lot of experience in doing so) or other similar educated listeners, and even these results seem to be summarily dismissed as having yet other flaws: test too long, too short, lighting not right, etc., moving the goalposts out so that (conveniently) no adequate test can ever be developed. This is what I thought you were attempting to do but I now realize otherwise.

If you concur that it's possible to construct some test to positively determine whether it is humanly possible to hear, say, a difference based solely on conductor material alone then we are in complete agreement in our desire to see this done. But I'm not holding my breath, and even if accomplished the results would probably be rejected by some even if completed to the satisfaction of someone with a PhD in Statistics from Yale, AB summa cum laude in Statistics from Princeton, and who won the American Statistical Associaton Theory and Methods Prize, and for whom the classic multivariate two-sample nonparametric test is named.

My skepticism lies only with the 'I hear it, I believe it, and no test can prove otherwise' crowd, and I assume someone with your level of expertise would as well.

 

[wavoman]

Quote:

Originally Posted by ILikeMusic /img/forum/go_quote.gif Yes, I understand and I am not saying that pre-qualification in itself disqualifies the results, as long as you're not looking for results that are representative of the general public ... If you concur that it's possible to construct some test to positively determine whether it is humanly possible to hear, say, a difference based solely on conductor material alone then we are in complete agreement in our desire to see this done.

We agree. First we'd like to test actual, commercial, expensive high-end cables against very cheap ones, and we will labor hard to put the test above reproach from all angles -- comfort, duration, statistical theory, and so forth. This is not easy, but we will try.

I have an exotic, well-broken in, retail > $1000 digital S/PDIF cable that I paid dearly for on eBay exactly for this test. I will pit it against a 75-ohm video cable thrown in the box from the last cheap DVD player someone buys. We will have a well-respected DAC, and HP amp, plus HPs. The only variation will be the connection from the source to the DAC. We will try 16/44.1 and 24/96 if possible.

The test will be single blind. It will have false comparisons/swindles, and will evaluate each individual as a sample-size one block.

Hope to do this at the March NYC meet, although it's just Upstate's idea, not officially blessed by anyone.

 

[upstateguy]

Quote:

Originally Posted by ILikeMusic /img/forum/go_quote.gif Yes, that's true in general, but one has to consider the reasonability of where the line is drawn. When someone says that their home system sounds better than their car system there's usually not a lot of criticism because it's quite possibly the case. When someone claims to be able to casually differentiate between 320 kbps and lossless the claim is a good deal less likely since that ability is rather uncommon, but it does remain possible nonetheless as a select few can positively demonstrate the capability. However, as one approaches the realm of physical impossibility (such as claiming that they can hear a difference based solely on the type of conductor) then the game kind of changes and at that point outright disbelief is quite legitimate and bound to occur. .

Quite well said.

USG

 

[ILikeMusic]

Quote:

Originally Posted by wavoman /img/forum/go_quote.gif I have an exotic, well-broken in, retail > $1000 digital S/PDIF cable that I paid dearly for on eBay exactly for this test.

Well I will say that you are dedicated!

But I'll also suggest that a little applied engineering might also answer the question at a lower cost.

 

[wavoman]

Quote:

Originally Posted by jonathanjong /img/forum/go_quote.gif AH HAH! But do they use metal or wood?

Ex-wives and wood? Let's not "touch" that subject.

OK -- back to statistical sampling -- your position makes sense. You want to know if a general population effect exists, so your (standard) test is correct. You know about between-subject variation, but don't care. You control for it with size of test group. Granted.

BUT it is better to control for it by making each subject a block, so-called sample-size one, and track replicated trials within that block. The sample population will self-classify itself if the difference exists only for some.

I have a different goal -- I want to know if this difference is audible to anybody out there, even one person. So I use "play the winner" sampling, and adjust the tests accordingly. My sample-size one blocking is automatic.

You will learn more than I would, but you do a lot more testing -- you can isolate the individual (if they exist) that can hear a difference, as well as learn something about the population at large. I can only do the first, but that's all I care about. The crowd can listen to MP3s on iPods with the free buds Apple supplies.

 

[PhilS]

Quote:

Originally Posted by upstateguy /img/forum/go_quote.gif Phil, you are a professional arguer so I will decline being drawn into one.

I understand. But I will continue to point our when I think your statements, or those of others, are imprecise, misleading or judgmental (for lack of a better word). We all have different types of expertise (or purported expertise) and different experiences that we bring to this discussion. As I indicated previously, I'm not only interested in the subject matter, but also in the way the discussion is conducted by each side. But feel free not to comment, and I won't take your lack of response to anything I say as any type of admission, just as my lack of comment regarding certain statements made by others that I find quite disagreeable should not be construed as an admission on my part.

 

[wavoman]

Quote:

Originally Posted by ILikeMusic /img/forum/go_quote.gif Well I will say that you are dedicated! But I'll also suggest that a little applied engineering might also answer the question at a lower cost.

I didn't pay retail! And I now see I lied ... the retail was "only" $975 according to the eBay listing. Actually some web sites say $800 and others say $595 ... who knows or cares. Generally auctions for $300, and I paid a tad less (way more than I would for my own cable, however).

The point is I needed an exotic cable, broken in, expensive.

Here is what I saw on eBay:

Marigo Apparition 5.1 Digital Cable 1.5 meter. This is Marigo's flagship digital cable and one of the best every made. Cost $975.00 new.

Verified reviews, etc. using Google.

marigocable

I bought this just to have something broken in and exotic for these tests.

Applied engineering would answer an important, but different question! Anyway I would still have to buy the cable to test it, no? Nobody was going to lend it to me, not to prove to them that they wasted money!

The final answer lies in blind testing, I really believe this. I know many don't, on both sides of the debate!

You know what people spend on golf clubs? Critical CD listening is my #1 hobby/passion and very important to my mental health (as is all recreation, no?). Not much money really to settle (for me, if no one else) a question that bugs me.

 

[PhilS]

Quote:

Originally Posted by upstateguy /img/forum/go_quote.gif Quite well said. USG

Yes, well said, but fallacious, as the entire statement assumes several facts that are not in evidence, as we lawyers like to say. For example, "it's quite possibly the case," "that ability is rather uncommon," and "approaches the realm of physical impossibility" are all assumptions. It's always easy to assume the entire argument away, but it doesn't really aid the analysis.

 

[upstateguy]

Quote:

Originally Posted by wavoman /img/forum/go_quote.gif We agree. First we'd like to test actual, commercial, expensive high-end cables against very cheap ones, and we will labor hard to put the test above reproach from all angles -- comfort, duration, statistical theory, and so forth. This is not easy, but we will try. I have an exotic, well-broken in, retail > $1000 digital S/PDIF cable that I paid dearly for on eBay exactly for this test. I will pit it against a 75-ohm video cable thrown in the box from the last cheap DVD player someone buys. We will have a well-respected DAC, and HP amp, plus HPs. The only variation will be the connection from the source to the DAC. We will try 16/44.1 and 24/96 if possible. The test will be single blind. It will have false comparisons/swindles, and will evaluate each individual as a sample-size one block. Hope to do this at the March NYC meet, although it's just Upstate's idea, not officially blessed by anyone.

Chill Waveperson, this is only our first foray.... nothing has to be perfect the first time. There are endless meets to attend and endless data to collect along the way....

I find it hard to imagine anyone interested in spending big bucks on cables not coming over to see if they can actually hear any differences. It may not influence their purchase, and that is not our purpose, but for the most part this is a "Show Me" group. We are all from Missouri when it comes to actually hearing it for ourselves... That's why we go to meets.... to hear stuff....

Remember waiting to listen to Ray's switch box?

USG

Oh BTW, single strand Ag wire interconnects

 

[upstateguy]

Quote:

Originally Posted by PhilS /img/forum/go_quote.gif Yes, well said, but fallacious, as the entire statement assumes several facts that are not in evidence, as we lawyers like to say. For example, "it's quite possibly the case," "that ability is rather uncommon," and "approaches the realm of physical impossibility" are all assumptions. It's always easy to assume the entire argument away, but it doesn't really aid the analysis.

Phill the biggest assumptions are the ones you are making.....

 

[PhilS]

Like the pic.

 

[ILikeMusic]

Quote:

Originally Posted by wavoman /img/forum/go_quote.gif The point is I needed an exotic cable, broken in, expensive.

A 'broken in' cable..? OK, I'm not even going to go there...

 

[JaZZ]

Quote:

Originally Posted by upstateguy /img/forum/go_quote.gif Oh BTW, single strand Ag wire interconnects

Pretty!

.

 

[wavoman]

Quote:

Originally Posted by ILikeMusic /img/forum/go_quote.gif A 'broken in' cable..? OK, I'm not even going to go there...

Hey, I don't really believe this stuff is audible ... but if you don't "break in" the cable then the cable-believers say "your test is not valid since the cable was not broken in".

Someone makes an automated cable break-in box.

For the record: I do believe you can hear differences in some analog cables. I have myself.

I also believe that cheap, out-of-spec digital cables can in theory degrade sound even if they correctly transmit binary data because the timing can be screwed up (especially by ferrite beads on USB cables), an edge transition can be missed and a bit flipped ... although I doubt this is audible.

By I think we will not find anyone who can tell a $20 75-ohm RCA video cable used for digital audio (S/PDIF) from a $500 "high end digital" one. And maybe this will still be true if we use a $2 video cable (where the terminations are not likely to even be truly 75 ohms). In the presence of RFI/EMI maybe some cheap cables could have problems with long cable runs, but I don't believe we have these issues with normal home listening rooms.

However I hold none of these opinions strongly, and I think I can help run a single-blind test without tipping off the subjects.