Head-Fi.org › Forums › Equipment Forums › Sound Science › Testing audiophile claims and myths
New Posts  All Forums:Forum Nav:

Testing audiophile claims and myths - Page 114

post #1696 of 3037
Thank God. The disregard for probability theory was frankly insulting. I would suggest to these folks that they read "The Drunkard's Walk", it is among the most approachable explanations of probably and Bayes law that I have yet found. And, it's way cheaper than the cables...

I question if a meta study of all this anecdotal crap, interpreted properly couldn't generate real conclusions. I have the code for a Bayesian meta study. I just don't have the time to mine the data.
post #1697 of 3037
oops... stupid phone
Edited by scootsit - 12/10/12 at 7:50am
post #1698 of 3037
sorry
Edited by scootsit - 12/10/12 at 7:51am
post #1699 of 3037
"5/5 still has a 3.1% probability of guessing and 4/5 of 18.8%. You should have done (at least) 10 trials per cable and identified the cable 9 times." Please explain.
post #1700 of 3037
Quote:
Originally Posted by Yahzi View Post

"5/5 still has a 3.1% probability of guessing and 4/5 of 18.8%. You should have done (at least) 10 trials per cable and identified the cable 9 times." Please explain.

Under the hypothesis that there is no difference and selections go for either with 1/2 probability...

Probability of 5/5 correct is (1/2) ^ 5 = 0.03125.
Probability of 4/5 or more correct is 6 * (1/2) ^ 5 = 0.1875.

For the 4/5, the 6 comes from there being 6 different experimental outcomes where 4/5 or more are correct: C C C C I, C C C I C, C C I C C, C I C C C, I C C C C, C C C C C, where C denotes a correct guess and I is incorrect. And if guessing blindly, each outcome out of 32 is equally likely. Just look up the binomial cdf instead of doing the above counting.

In other words, with a low number of trials, there is still a nontrivial probability that blind guessing would produce the same results, so we can't make any good conclusions from the results.


The bigger issue is probably the methodology of the experiment in the first place. If the cable swaps are single blind, that's no good because of experimenter bias. Who knows what cues can be picked up just by looking at the guy doing the swapping. See Clever Hans, for example.
Edited by mikeaj - 12/10/12 at 8:30am
post #1701 of 3037
Quote:
Originally Posted by mikeaj View Post

Quote:
Originally Posted by Yahzi View Post

"5/5 still has a 3.1% probability of guessing and 4/5 of 18.8%. You should have done (at least) 10 trials per cable and identified the cable 9 times." Please explain.

Under the hypothesis that there is no difference and selections go for either with 1/2 probability...

Probability of 5/5 correct is (1/2) ^ 5 = 0.03125.
Probability of 4/5 or more correct is 6 * (1/2) ^ 5 = 0.1875.

For the 4/5, the 6 comes from there being 6 different experimental outcomes where 4/5 or more are correct: C C C C I, C C C I C, C C I C C, C I C C C, I C C C C, C C C C C, where C denotes a correct guess and I is incorrect. And if guessing blindly, each outcome out of 32 is equally likely. Just look up the binomial cdf instead of doing the above counting.

In other words, with a low number of trials, there is still a nontrivial probability that blind guessing would produce the same results, so we can't make any good conclusions from the results.


The bigger issue is probably the methodology of the experiment in the first place. If the cable swaps are single blind, that's no good because of experimenter bias. Who knows what cues can be picked up just by looking at the guy doing the swapping. See Clever Hans, for example.


I'm thinking that the improvements/detractions should be cumulative if cables do didley over squat. What if you recorded a song with different cables, then basically mixed a level matched ABX in the middle of the track, and phrased the test question very carefully, so that there was zero expectation bias. Something like, please listen to the attached track, and explain your perceptions of the differences in sound quality throughout the track.

 

I guess my thought is this. Let's pretend for a second that cables can alter sound quality ever so slightly. We're talking close to $2000 per cable in the above post (I just quoted the last post, I'm talking about the one Yahzi posted). I question what system really is so stellar that $2000 is best spent on a cable?! I mean, I have never had that kind of dough laying around, but let's pretend for a second that you do. $2000 is a Mitchell or Roksan turntable, it's a pair of incredible line amps, it's a pair of spectacular speakers, it's the better part of a wall of amazing vintage gear, it's an incredible preamp. It's a lot of damn money to spend, even on a single component. So, I guess to me, I could see spending $2000 on cables, if your system is ~$100,000...maybe. Because only then could the $2000 not be spent any better on system upgrades.

 

At the same time, $2000 is also ~200 CDs, or ~100 LPs, so is the hardware upgrade really enhance the enjoyment that much over so much software?

 

I guess I could see spending $2000 on cables if you have the $100,000 system of your dreams, and every CD/LP ever pressed, and you got money to blow...

 

Well, I guess it doesn't matter because I'll never be in that position.

post #1702 of 3037
"For the 4/5, the 6 comes from there being 6 different experimental outcomes where 4/5 or more are correct: C C C C I, C C C I C, C C I C C, C I C C C, I C C C C, C C C C C, where C denotes a correct guess and I is incorrect. And if guessing blindly, each outcome out of 32 is equally likely. Just look up the binomial cdf instead of doing the above counting. In other words, with a low number of trials, there is still a nontrivial probability that blind guessing would produce the same results, so we can't make any good conclusions from the results." I'm feeling especially dumb right now because I don't quite understand what you've written. :) PS How do you quote on this forum? The normal quote code isn't working for me here.
Edited by Yahzi - 12/10/12 at 10:19am
post #1703 of 3037

See http://en.wikipedia.org/wiki/Permutation http://en.wikipedia.org/wiki/Binomial_distribution

To quote, just press the quote button and remove the parts you don't want to quote.


Edited by xnor - 12/10/12 at 11:47am
post #1704 of 3037

I like music more than math.

post #1705 of 3037
Quote:
Originally Posted by scootsit View Post


I'm thinking that the improvements/detractions should be cumulative if cables do didley over squat. What if you recorded a song with different cables, then basically mixed a level matched ABX in the middle of the track, and phrased the test question very carefully, so that there was zero expectation bias. Something like, please listen to the attached track, and explain your perceptions of the differences in sound quality throughout the track.

 

***********************

 

This would not be a very sensitive test.

For a good test, the listener should be familiar with both A and B before starting the test. The listener should be able to select A or B or X at will.  In sensitive tests the listener  is often unable to identify just what the differences are that allowed him to make to correct selection.

post #1706 of 3037

So besides the number of tests for each cable, what else was flawed in the experiment?

post #1707 of 3037
Quote:
Originally Posted by Yahzi View Post

So besides the number of tests for each cable, what else was flawed in the experiment?

 

The report is very vague about a lot of experimental details.  How long were listeners allowed to listen?  What environment, etc.?  Usually this means that best practices were not at all followed.  It seems like this was probably done single blind.

 

Now think about possible ways the listener can get clues (consciously or not) for which cable is being used, other than by the sound:  looking at the body language of the person doing the swapping (if that person knows which is which), listening for differences in how the cables sound when being plugged in and out and dropped on the floor, and so on.  I mentioned single-blind studies and Clever Hans earlier.  

 

On the flip side, certain different testing conditions may make things legitimately easier to distinguish.  For example, maybe a procedure that allows listeners to swap instantaneously would help them out.  Or a better sound system, or possibly higher volume, lower ambient noise, etc.

 

If you're testing differences in sound quality, of course it's better to eliminate any other differences you can, like those above.  Unless somebody is very careful with those kinds of things, they usually aren't handled properly.

post #1708 of 3037

Any specific questions I should ask concerning the experiment? Okay, I''ll ask about the length of time taken between cable swops and where it was conducted. Anything else?  I'll report back here with his answers.

post #1709 of 3037
Those are more minor details. It's better to know more about the listening and testing procedure overall, but the biggest questions are about how the swapping is done.

Did the person who swapped know which cable was which? Were the cables directly physically handled during swapping?**

**To get around these things, I think one way to do it would be to use a switchbox with the cables already wired up, where the person handling the switchbox for the tests does not know which switch position selects which cable (somebody else set it up). Even then, that is not ideal. Best to keep the person switching and the whole mechanism out of sight and sound.
Edited by mikeaj - 12/11/12 at 7:59am
post #1710 of 3037

Am I missing something?  Shouldn't $2000 cables sound amazingly clean and open at first listen and to all listeners?  Do you really have to work this hard to hear a difference after spending that much money?

New Posts  All Forums:Forum Nav:
  Return Home
  Back to Forum: Sound Science
Head-Fi.org › Forums › Equipment Forums › Sound Science › Testing audiophile claims and myths