[wavoman]

Quote:

Originally Posted by Pio2001 /img/forum/go_quote.gif ...most common error ...is to present individual results, then pick among them the most significant and presenting them as successful, because they are above the threshold. ... the question of averaging across the subjects was a very difficult one. On one hand, if we sum the results of everybody, we can prove a difference otherwise unseen, thanks to the statistical weight of all the answers, but on the other hand, since the listeners are rather untrained, the probability is high that one or two listeners score well and not the others. Summing the answers leads then to a failure while these listeners actually hear the difference ... That's why I rather first define a target probability of false positive,... Then I require for any listener that he scores a modest success, like 6/6, in order to be allowed to proceed to the real ABX....

Picking the top scorers and claiming significance is of course the most elementary of mistakes, we agree! (Usually called "the selection fallacy"). But it is also the foundation of more sophisticated "play the winner" designs, which I see you are using ... via your pre-tests!

I think that the selection fallacy is so dreaded that people stay away from "play the winner", which is a shame since it is so powerful.

If you added swindle comparisons your protocols become even stronger. The standard A/B/X question does not cater to that, but the simple "difference", or "preference" question (I like "prefererence" better) does, and the answers quickly weed out the weak subjects.

I see no reason to ever compute a statistical threshold on the group total result ... it has no meaning. These tests are replicated "sample size one" designs, where each subject is a block.

BTW, there is no need to be so precise with your false positive ("guessing") probabilities ... as long as you go forward with additional randomized tests on the selected subjects, the threshold set to allow subjects to go on to the next stage does not need to be based on a probablility calculation.

 

[rimrocks]

Excellent work Wavoman. I just hope you get some data after all this effort.
Totally agree on the futility of the standard A/B/X tests, having participated at both ends. Pressure is always there, and it does affect the results. Also, trying to get pro-audio guys to do blind A/B tests is like pulling teeth - and I can't really fault them for that.

 

[wavoman]

Quote:

Originally Posted by rimrocks /img/forum/go_quote.gif Excellent work Wavoman. I just hope you get some data after all this effort. Totally agree on the futility of the standard A/B/X tests, having participated at both ends. Pressure is always there, and it does affect the results. Also, trying to get pro-audio guys to do blind A/B tests is like pulling teeth - and I can't really fault them for that.

Thank you -- I am deeply honored by your comments. You can hide behind "rimrocks", but we know you are the inventor of dts (I can hear the Jurassic dinosaur footsteps in my head still today) and one of the leading audio scientists in the world.

Pio and others have made a fantastic suggestion that I want to adopt: Do a non-blind, fully-revealed comparison of A and B for each subject to start. Let them look and listen. As much as they want. This has two important benefits:

(1) Ss get comfortable and relax. They fully know what the blind trials will be about. You even tell them some will be swindles.

(2) If after the non-blind intro test they say "hey, I can't hear a difference", then don't bother with the blind tests.

We hope to do the first trials at the next NYC meet March 21. I am hoping to have back my modded Pioneer CDP with the Audio Praise Vanity Board, so I can play DVD-A's and/or SACDs and capture a hi-res 2-channel LPCM output feed on to the S/PDIF cables we are testing, and to have my modded Benchmark DAC-1 (with the S/PDIF input BNC replaced by a true 75 ohm RCA jack) on the other side of the cable (and then balanced analog to the QES balanced HP amp, and balanced HPs, beyer 880/600/headphile cable and Senn 600/Zu cable). [We also have the SE HP out of the DAC-1].

This way the quality of the source and the amp (and the HPs) are above question, and there are no other cables or parts in the digital chain. I keep losing eBay auctions for "high end" balanced analog cables, so I may have to use a studio-quality Belden or Mogami with Neutrik golds, but that won't satisfy the audiophile cable snobs, so I will keep bidding!

 

[rimrocks]

Quote:

Originally Posted by wavoman /img/forum/go_quote.gif Pio and others have made a fantastic suggestion that I want to adopt: Do a non-blind, fully-revealed comparison of A and B for each subject to start. Let them look and listen. As much as they want.

Flattery does work!
I also think this is a great idea. The less pressure the better. The only thing missing would be some pointers from the experimenter (you!) to the subject on what they should be listening for. Perhaps this is the new definition of a DBT?

 

[Pio2001]

I don't remember where, but I recently read a study where it was shown that trained listeners performed much better than untrained listeners, and that four or five sessions were enough for untrained listeners to score as high as trained ones.

 

[olblueyez]

Quote:

Originally Posted by upstateguy /img/forum/go_quote.gif Let's prove the Null Hypothesis: I'm searching for: 1- A body of evidence to support the proposition that cables sound differently from each other. 2- A body of evidence to support the proposition that exotic power cords, make a difference. 3- A body of evidence to support the proposition that all Amps do not sound the same. Of course, manufacturer's data must be excluded for obvious reasons. I'm looking for any evidence, collected by any methodology. What I'm not looking for is an "I hear a difference" show of hands. Let's find the body of evidence! USG

Why would anyone care if you handicap your system for the purposes of believing in writings and charts and graphs instead of listening experience to evaluate audio equipment? How about if you do the testing since you are the one who requires this testing to know what you hear. You purposely block experience from knowledgeable people to then turn around and say "Prove it to me"? Im sorry if I sound harsh but I just dont understand what it is that you want? Proof that you, yourself can benefit from the experience of other people? Equal efforts on your part to do some experimenting with cables will give you the knowledge you seek.

One suggestion that is on topic and is acredited to Wavoman. Let people see what cables you are using, inform them of everthing going on and let them spend as much time as possible with each cable. While doing this keep all the test subjests separate (no communication, whispering, chatting) and have them write the results and when you are done you can see if the experiences have any continuity. Just a thought.

 

[Pio2001]

Body of evidence refer to blind listening tests, not graphs or charts. And in order to blind test, one must hear a difference. That's why many listeners, who don't hear any difference between cables or electronics on their systems, ask for results from other people, who hear something and can try blinded.

Comparing the sighted experiences of many isolated listeners is not a good body of evidence because the knowledge of the system under test can have a similar influence on several listeners. And group effect can show up even without verbal communication. That's why blind tests are better suited for this kind of question.

It is perfectly possible to run a blind test like you suggest, with as much time as anyone need, and just one opinion to give. In fact, at least one blind test have been run this way (Kiang, hifiwigwam.com). The results were disastrous (from the point of view of the significance of the results, because from the point of view of psychological illusions, it was a brillant success !). It confirms that untrained listeners are not the way to go for success in blind tests.

By the way, I found back the study about trained listeners vs untrained ones. It was from Sean Olive. Three first articles in his blog : Audio Musings by Sean Olive

 

[milkweg]

The below says it all for me. Anyone who thinks a power cord makes an audible difference is in cuckoo land as far as I am concerned.

http://www.theaudiocritic.com/plog/

Electronic Signal Paths Do Not Have a Personality!
I keep forgetting that my newer readers outnumber the old-timers and that some of the basic truths about audio that are old hat to me and to the regulars are new and fresh to the recent arrivals. Here is something, therefore, worth repeating for the nth time.
Every low-distortion electronic signal path sounds like every other. The equipment reviewers who hear differences in soundstaging, front-to-back depth, image height, separation of instruments, etc., etc., between this and that preamplifier, CD player, or power amplifier are totally delusional. Such differences belong strictly to the domain of loudspeakers. Depending on the wave-launch characteristics, polar pattern, or power response of the loudspeaker (those are overlapping concepts), the stereo presentation of the program material can vary greatly. It cannot vary as a result of the properties of a normal (i.e., low-distortion) electronic signal path. The only exception I can think of would be totally inadequate channel separation (less than, say, 30 dB) between the left and right channels of a stereo device, which is hardly ever the case—and certainly not when high-end components are being discussed by said reviewers.
Beware, therefore, of electronic audio components with a personality. If they have a personality, they are either defective or the brainchild of a reviewer without accountability.

A Fraud that Anyone with Common Sense Can See Through
Longtime readers of The Audio Critic are fully aware that many of high-end audio’s articles of faith are bogus. Most of these fraudulent pronouncements about cables, tubes, vinyl, etc., require a little bit of engineering science to refute. A typical example is the absurd practice of biwiring, whose futility is made obvious by the superposition principle, a law of physics not known to everyone (see under downloadable Sample Articles, “The Ten Biggest Lies in Audio,” on this website).
There is one particular audio fraud, however, that requires no science but just ordinary common sense to see through. I’m talking about power cords—yes, those short lengths of flexible insulated cable that go between your wall outlet and your audio gear. The big lie is that they hugely affect the sound. The ads tell you that if you pay $499 or $995 or some such insane amount for a specially designed super cord, you will get a bigger soundstage, better transients, tighter bass, smoother highs, etc., etc., blah, blah, blah. Amazingly, quite a few nonthinking audiophiles with deep pockets buy these fantasy cords.
Now, just think about it. The AC current comes into your residence over miles and miles of wire. After it enters the walls of your house or apartment, it again traverses huge lengths of BX cable or similar wiring. After it comes out of your wall outlet through the power cord and enters your amplifier or other equipment, it again goes through a maze of wiring before activating the devices that affect the sound. So, tell me, how does the electricity know where the nondescript lo-fi wiring stops and the super wire—just six feet of it—starts and leaves off? Does the current say, hey, I’m coming out of the wall now, the next six feet are crucial? Come on. The power cord represents an infinitesimal fraction of the AC current’s total path. Even if the wire in the power cord were so much better, it would have to be stretched all the way back to the power station to make a difference! It’s pure bull on the face of it; no science needed. The fact is that any power cord rated to handle domestic AC voltages and currents is as good as any other. The power cord that came with your amplifier or receiver will give you optimum performance. If you have to buy an extra one, just make sure it’s thick enough in gauge for heavy-duty equipment. If you pay more than a few bucks, you’ve been had. (Besides, as I’ve stated a number times before, your audio circuits don’t know and don’t care what’s on the AC side of the power transformer. What they’re interested in is the DC voltages they need. But that’s engineering science…)

Paradoxes and Ironies of the Audio World:
The Doctor Zaius Syndrome
By Peter Aczel Editor
(Abbreviated and edited version of a still timely article first published ten years ago.)
When the truth is so terrible that admitting it would surely make the whole system crumble, ape logic demands denial and coverup.
Have you ever seen that marvelous 1967 science-fiction movie The Planet of the Apes? If you have, you will recall that it depicts a planet of the future where English-speaking anthropoid apes are the rulers and humans are speechless beasts of burden, enslaved by the apes and despised as a totally inferior species. The apes have horses and guns but no real technology. Doctor Zaius, the subtle and highly articulate orangutan who is this society’s “Minister of Science and Defender of the Faith” (he is played by the great Maurice Evans), knows something the other apes do not: that humans in a past era possessed not only speech but superior technology, flying machines, powerful weapons, and so forth, all of which served only to bring about their eventual downfall and reduce them to their present condition. Doctor Zaius fervently believes that any knowledge of this truth about humans would totally destabilize the society of apes and result in the end of their world. The ape dogma he fanatically protects, even though he knows better, is a blatant denial and coverup of the actual history of the vanished human civilization and a paean to the eternal superiority of the ape.
I won’t give away the rest of the plot to those of our readers who haven’t seen the movie and may want to, but doesn’t Doctor Zaius resemble certain key figures in the high-end audio community? He knows the truth but it’s bad for the establishment. The system would come crashing down if the truth were revealed. To pick an obvious example, consider John Atkinson, the subtle and highly articulate editor of Stereophile. Don’t you think he knows? Of course he knows. But if he admitted that $3000-a-pair speaker cable is a shameless rip-off or that a $7000 amplifier sounds no different from a $1400 one, the edifice of high-end audio would begin to totter—or so he thinks (and may quite possibly be right). Consequently, he spouts convoluted scriptural arguments and epistemological sophistries, just like Doctor Zaius, in order to pervert the obvious, uncomplicated, devastating truth.
There is a perfect illustration of this process in the August 1994 issue of Stereophile, where Zaius-Atkinson once again bashes blind listening tests in an “As We See It” editorial. Such tests are of course considered extremely threatening by a publication that reports night-and-day differences in sound which absolutely nobody can hear when the levels are matched and the brand names concealed. He brings up all kinds of intricate flaws and drawbacks that may very well exist in some blind tests but turns his back on the large number of blind tests in which all of his objections have been anticipated and eliminated and which nevertheless yield a no-difference result every time. He knows very well, for example, that no one has ever, ever proved a consistently audible difference between two amplifiers having high input impedance, low output impedance, and low distortion, when operated at matched levels and not clipped—but like Doctor Zaius he conceals that knowledge. He’d rather collect rare case histories of screwed-up blind tests than deal with the vast body of correctly managed blind tests that undermine the Stereophile agenda. (Just for the record, I’ll state for the nth time that there are only two unbreakable rules in blind testing: matched levels and no peeking at the nameplates. To eliminate “stress,” take a week or a month for each test, send everybody else out of the room, operate the switch yourself at all times, switch only twice a day—whatever. The results will still be the same.)
Our columnist Tom Nousaine, in a recent conversation with me, stated his belief that any longtime audio reviewer who has tested hundreds of different audio components over the years knows exactly what the truth is about soundalikes because it is utterly impossible to escape that truth after so much hands-on experience. It asserts itself loud and clear, again and again. Therefore, he argued, the audio journalists who invariably report important sonic differences are most likely a bunch of hypocrites, i.e., exhibit the Doctor Zaius Syndrome. I was strongly inclined to agree with him…
Why do I even bother to tell you all this?
All of our readers who have been with us for more than just one or two issues are aware of my enormous frustration on the subject of scientific truth in audio. The very idea of a Doctor Zaius Syndrome, even it’s only a parody, suggests the existence of antiscience in audio as a tradition, not just a momentary aberration—and a tradition it is, going back to the early 1970s, at the very least. In the late ’40s and throughout the ’50s and ’60s, whatever the most highly qualified and experienced engineers said about audio was the accepted truth. Then came postmodern irrationalism, post-Watergate anomie, fortune tellers in high places, pyramid power, Jesus-haired record-store clerks as self-proclaimed audio experts, untutored high-end journals, pooh-poohing of engineering societies, derision of degreed academics—the B.S. era of audio (and I don’t mean Bachelor of Science). Today, the melancholy truth is that tweako cultism has become mainstream audio, at least above a certain price range, and engineering facts are regarded as disturbingly radical or at least eccentric. The scientific audio community has been marginalized.
I despair at this point of a journalistic solution. Even if The Audio Critic increased its circulation by a factor of 50 overnight—I’m being deliberately absurd— it might still be too late for the message. The cultists have been too deeply indoctrinated and too long. The pimply-faced kid in the Bon Jovi T-shirt who tried to sell you AudioQuest Sorbothane Feet (the bigger kind) in your local audio salon is not going to change his belief system. Not in this antirationalist age and culture.
I can think of only one effective remedy. Many years ago, long before our younger readers became interested in audio, the Federal Trade Commission put an end to fraudulent power-output claims in amplifiers. Today, the power-output specification must take the form of “200 watts continuous power into 8 ohms from 20 Hz to 20 kHz at less than 0.25% total harmonic distortion.” Before then, the same amplifier could have claimed 800 watts because it could produce that for 2 milliseconds at 1 kHz into 2 ohms with 10% distortion. What if the FTC suddenly became interested in audio cable advertising, for example? That chattering sound you hear comes from the teeth of cable vendors at the mere mention of the possibility. And that low, rumbling sound you hear is Doctor Zaius growling, “That's heresy!”
Anyone out there whose nephew or brother-in-law is a young, crusading, Ralph-Nader-like employee of the FTC? Get him interested!

 

[PhilS]

Quote:

Originally Posted by milkweg /img/forum/go_quote.gif The below says it all for me. Anyone who thinks a power cord makes an audible difference is in cuckoo land as far as I am concerned.

The above says it all for me. Anyone who has to express his opinions or offer information in the manner that you choose to do so is just an ass, as far as I am concerned. On the other hand, that may be too harsh. Perhaps you just need therapy.

 

[milkweg]

If being an ass is what it takes to drum some common sense into your heads then so be it.

 

[PhilS]

Quote:

Originally Posted by milkweg /img/forum/go_quote.gif If being an ass is what it takes to drum some common sense into your heads then so be it.

Priceless. Just wanted to capture this in case you decide to edit it later.

 

[olblueyez]

Quote:

Originally Posted by milkweg /img/forum/go_quote.gif If being an ass is what it takes to drum some common sense into your heads then so be it.

Let all of us know when you get us straightened out now, ya hear? Want to start a pool to see how many teeth Milkweed has in his head? I really think I have seen it all now, I finally got to the end of the internet.

 

[milkweg]

Well, I already got you to remove that ridiculous quote you had about the HD650 sounding veiled was all due to the cable so it must be working.

 

[olblueyez]

Quote:

Originally Posted by milkweg /img/forum/go_quote.gif Well, I already got you to remove that ridiculous quote you had about the HD650 sounding veiled was all due to the cable so it must be working.

Here is a clue for you, I altered my signature back and fourth for about a week until I got it the way I want it. It had nothing to do with you, never did and never will. If you knew the difference between breast meat and a chicken necks then you would know the cable on the 650 has a great impact on the sound of the 650. If you have any doubts about this then I recommend you ask some other members who have tried an after market cable. They have experience that makes you and your imagined opinions completely moot. Experienced members are a great resource for everyone and you do nothing but attempt to belittle these people. Very lame dude.

BTW: You are not fun and you certainly are not funny, just lame.

 

[ILikeMusic]

Quote:

Originally Posted by olblueyez /img/forum/go_quote.gif If you knew the difference between breast meat and a chicken neck then you would know the cable on the 650 has a great impact on the sound of the 650.

Priceless. Just wanted to capture this in case you decide to edit it later.