Help with a philosophical problem with blind testing cables

[PhilS]

Quote:

Originally Posted by odigg /img/forum/go_quote.gif That was actually my rationale when I originally stated that people should blind test amps, dacs, sources, etc. before cables. If a person cannot have a positive result between those devices, then how are they going to have a positive result with cables?

I think what you mean is that people should blind test amps, dacs, etc. that measure differently and see if there is a positive result before testing to see if they can hear differences with cables. The italicized language is the key.

 

[odigg]

Quote:

Originally Posted by PhilS /img/forum/go_quote.gif I think what you mean is that people should blind test amps, dacs, etc. that measure differently and see if there is a positive result before testing to see if they can hear differences with cables. The italicized language is the key.

Yes, you are correct. Thankfully, it's quite easy to find equipment where the measured differences between equipment are greater than the measured differences between cables.

And, as always, the equipment should meet certain characteristics (flat frequency response, no noise, etc).

 

[SmellyGas]

Quote:

Originally Posted by royalcrown /img/forum/go_quote.gif Take the case of amplifiers and DACs: saying that differences between DACs and amps are inaudible is a very bold claim. Therefore, a negative result would be far more significant than with cables. Many people claim that cables make small differences that reveal themselves over time. However, not many people claim that amplifiers make small subtle differences that reveal themselves over time. Therefore, a negative result would be of much interest, because whatever the testing methodology used, it would have to be absurdly terrible and horrifically flawed to obscure what would presumably be a very large effect size.

Power amplifiers (for loudspeakers) have already been tested blind. At least 20 independently conducted and separately published trials have been performed, mostly in peer-reviewed journals, encompassing cheapo amps to the high-end exotic stuff, on various loudspeakers and ancillary equipment, included listeners who insisted there were amplifier differences, and NOT ONE has been able to demonstrate an audible difference as long as the equipment was used within specification (i.e. not driven to distortion). The handful of studies that showed a weak ability for individuals to identify differences among amps were actually the result of improperly applied statistic analysis - once corrected, there were no differences. Nousaine in the Journal of AES has a good abbreviated review of the "data" so far.

 

[PhilS]

Quote:

Originally Posted by SmellyGas /img/forum/go_quote.gif Power amplifiers (for loudspeakers) have already been tested blind. At least 20 independently conducted and separately published trials have been performed, mostly in peer-reviewed journals, encompassing cheapo amps to the high-end exotic stuff, on various loudspeakers and ancillary equipment, included listeners who insisted there were amplifier differences, and NOT ONE has been able to demonstrate an audible difference as long as the equipment was used within specification (i.e. not driven to distortion). The handful of studies that showed a weak ability for individuals to identify differences among amps were actually the result of improperly applied statistic analysis - once corrected, there were no differences. Nousaine in the Journal of AES has a good abbreviated review of the "data" so far.

I guess that's why many of us have doubts about the "trials." I suspect similar results exist for DAC's, even though many are convinced they sound different. I think there was also (at least as reported by JaZZ) a blind test in which people failed to distinguish between live music and speakers. Interesting.

 

[AtomikPi]

Quote:

Originally Posted by SmellyGas /img/forum/go_quote.gif Power amplifiers (for loudspeakers) have already been tested blind. At least 20 independently conducted and separately published trials have been performed, mostly in peer-reviewed journals, encompassing cheapo amps to the high-end exotic stuff, on various loudspeakers and ancillary equipment, included listeners who insisted there were amplifier differences, and NOT ONE has been able to demonstrate an audible difference as long as the equipment was used within specification (i.e. not driven to distortion). The handful of studies that showed a weak ability for individuals to identify differences among amps were actually the result of improperly applied statistic analysis - once corrected, there were no differences. Nousaine in the Journal of AES has a good abbreviated review of the "data" so far.

There have been a few positive results for both dacs and amps with tubes versus SS, but there have also been negative results.

I think what royalcrown was trying to suggest was testing the differences between headphone amps. That HAS NOT been done to my knowledge. I think one could at least superficially claim that there would be a difference because with headphones we have the added issue of output impedance that can have measurable effects. I'm more interested in seeing a comparison of a lower power (but sufficient to not create distortion) and low (10 or less ohms maybe) impedance amplifier versus a higher end SS amplifier.

There has been testing of dacs which has shown no audible differences. There's one between the Behringer DEQ and the DAC1, there have also been tests of $50 dvd players vs $5000 cdp that revealed no audible differences.

Also, regarding the OP's question, maybe I'm missing something, but isn't the point of switching cables that you're isolating the differences to just the differences between the cables - if there's a flaw or something in the internal wiring it would create a systematic error so a positive result would still mean an audible difference exists between the cables. The only way we could say this would have an impact would be (1) some sort of synergy or "dissynergy" between internal and external passive components or (2) the internal wiring is a sort of handicap or maximum on SQ meaning that the cables can't overcome the inherent limitations in the internal wiring thus no audible difference. This is a bit troublesome because it means we can always say "well there are differences but you can only tell with better equipment" as the haters of lossy audio compression often claim.

 

[rsaavedra]

Some time ago I wrote about some of the difficulties a DBT of headphone cables would have to overcome.

Testing different headphone cables would also face the issue of slightly different positioning of the headphone drivers over the ears with every swapping of the cables. Those minor differences alone might cause important audible differences between auditions, even without changing cables.

So a rigurous enough DBT of cables definitely would be much easier to carry out for interconnects than for headphone cables.

 

[SmellyGas]

Quote:

Originally Posted by AtomikPi /img/forum/go_quote.gif There have been a few positive results for both dacs and amps with tubes versus SS, but there have also been negative results.

Would you kindly provide the link or reference for the amplifier study where there were positive results in a blinded comparison? I'd like to examine the methodology and analysis. Thanks! As far as I know, none exist that are reputable or authoritative.

 

[SmellyGas]

Quote:

Originally Posted by PhilS /img/forum/go_quote.gif I guess that's why many of us have doubts about the "trials." I suspect similar results exist for DAC's, even though many are convinced they sound different. I think there was also (at least as reported by JaZZ) a blind test in which people failed to distinguish between live music and speakers. Interesting.

Generally speaking, when you have TWENTY independent groups performing a controlled experiments to test the same hypothesis, using different but comparably valid methodologies, and the results are published in peer-reviewed journals, most reasonable people who are familiar with science and the scientific process will ACCEPT the results.

On the other hand, you could choose not to accept such evidence. For instance, you could argue that you believe smoking does not lead to heart disease, lung cancer, and emphysema. Your argument could be, "well my neighbor smoked since he was 15 and now he's 75 and perfectly healthy...this is why my smoking friends and I all have doubts about these medical 'trials.'"

 

[PhilS]

Quote:

Originally Posted by SmellyGas /img/forum/go_quote.gif Generally speaking, when you have TWENTY independent groups performing a controlled experiments to test the same hypothesis, using different but comparably valid methodologies, and the results are published in peer-reviewed journals, most reasonable people who are familiar with science and the scientific process will ACCEPT the results. On the other hand, you could choose not to accept such evidence. For instance, you could argue that you believe smoking does not lead to heart disease, lung cancer, and emphysema. Your argument could be, "well my neighbor smoked since he was 15 and now he's 75 and perfectly healthy...this is why my smoking friends and I all have doubts about these medical 'trials.'"

I don't think that's a good analogy at all (in fact, I think it's pretty absurd for several reasons), but I'm not going to argue with you. You've pretty much made up your mind on the issue of the validity of the tests (BTW, assuming they are valid assumes away the entire argument), notwithstanding some reasonable issues and questions raised by people in other threads. So I guess I should not have said anything at all.

 

[AtomikPi]

Quote:

Originally Posted by SmellyGas /img/forum/go_quote.gif Would you kindly provide the link or reference for the amplifier study where there were positive results in a blinded comparison? I'd like to examine the methodology and analysis. Thanks! As far as I know, none exist that are reputable or authoritative.

Sure. These are what I was referring to. Not particularly professional, I know, but Matrix Hi-Fi has had plenty of very interesting negative results in addition to a few positives.


Here is a positive abx of 300B tube vs SS - 300B is known as a "tubey" tube so this is not contradictory to their general view that properly designed non distorting tube cannot be distinguished from SS
Google Translate

Here is a positive of a discman vs a tube cdp - subjects preferred the discman
Google Translate

I just found this one, which really doesn't make sense to me:

Google Translate

 

[SmellyGas]

Quote:

Originally Posted by PhilS /img/forum/go_quote.gif I don't think that's a good analogy at all (in fact, I think it's pretty absurd for several reasons), but I'm not going to argue with you. You've pretty much made up your mind on the issue of the validity of the tests (BTW, assuming they are valid assumes away the entire argument), notwithstanding some reasonable issues and questions raised by people in other threads. So I guess I should not have said anything at all.

Well, consider this. Anyone can do an experiment. But in order for it to be accepted for publication by a peer-reviewed journal, it has to go through peer review. Peer review is a process by which experts in the field receive a copy of the manuscript and they scrutinize and criticize the paper. Part of this process involves examining whether or not the test methodology is valid. Peer review analyzes whether the experiment actually tests what it's supposed to be testing and whether the results are interpreted correctly. If the peer reviewers believe the experient was conducted poorly or the methods were inappropriate, etc., they will NOT accept the paper to be published in the journal. If the reviewers have certain questions or objections they feel need to be addressed, they send the paper back and require the authors to redo part of the experiment or add additional work before it can be published.

Blinded controlled studies published in peer-reviewed journals are among the most authoritative levels of evidence we have in this type of science. The only thing better is perhaps the meta-analysis, which combines the results of all the blinded controlled studies.

20+ papers that have all been peer-reviewed and basically point to the same conclusion is more than enough evidence for me. The exact methodology and analysis is published in the article itself so that readers can also critique for themselves. I think it's a little silly for people (who may be experts in their own fields, but with zero experience interpreting scientific data/studies) to be dismissing a large body of published work that has already been scrutinized by the scientific and engineering community and overwhelmingly points to the same conclusion (specifically that differences among power amplifiers are not likely audible).

 

[SmellyGas]

Quote:

Originally Posted by AtomikPi /img/forum/go_quote.gif Sure. These are what I was referring to. Not particularly professional, I know, but Matrix Hi-Fi has had plenty of very interesting negative results in addition to a few positives. Here is a positive abx of 300B tube vs SS - 300B is known as a "tubey" tube so this is not contradictory to their general view that properly designed non distorting tube cannot be distinguished from SS Google Translate Here is a positive of a discman vs a tube cdp - subjects preferred the discman Google Translate I just found this one, which really doesn't make sense to me: Google Translate

AtomikPi, thank you for taking the time to provide links! What is worrisome to me for all 3 of them is that they are not peer-reviewed, they do not explain their methodology in a way that it can be reconstructed, and they don't do any statistical analysis so we don't actually know if the results they saw could have been achieved by guessing alone. Are you aware of any positive results in a published journal, or at the very least, one that clearly explains the methodology, provides raw results (rather than "difference heard"), and the analysis?

 

[PhilS]

Quote:

Originally Posted by SmellyGas /img/forum/go_quote.gif I think it's a little silly for people (who may be experts in their own fields, but with zero experience interpreting scientific data/studies) to be dismissing a large body of published work that has already been scrutinized by the scientific and engineering community and overwhelmingly points to the same conclusion (specifically that differences among power amplifiers are not likely audible).

Fair enough. You're entitled to your opinion. And my opinion is that your tone is rude and condescending, and this post and a few others convey an arrogance that I find distasteful. I won't discuss this with you further.

 

[odigg]

royalcrown - You nailed it with what you said. I'm actually a little embarrassed that I rambled on for so long to elucidate my point and you said it so well in 3 paragraphs.

AtomkPi - Actually, people on this board have done controlled tests between headphone amps. royalcrown has done them. I have done them. I have also been in contact (through PM) with others who have done them.

You don't want to know the results

Many people have been silent or subtle about reporting their experiences because when experiences like this are reported, attacks get very personal. royalcrown can tell you a lot about that.

I won't state the equipment I used (for a number personal and ethical reasons), but I used my computer's motherboard onboard sound card as a baseline. The motherboard sound had some issues (electronic noise, RMAA verified bass rolloff). I did controlled comparisons with a bunch of other equipment, including amps that are well reviewed and desired on Head-Fi.

Let me say this. If I had no headphone equipment and $600 to spend, $550 of that would go towards the headphone. If I needed I headphone amp or dac, I'd just buy something inexpensive from the pro audio world or build a Mini3/Pimeta. Based on my tests, I have little interest in the gamut of headphone equipment typically discussed on Head-Fi.

Look at my sig

Quote:

Originally Posted by SmellyGas /img/forum/go_quote.gif 20+ papers that have all been peer-reviewed and basically point to the same conclusion is more than enough evidence for me. The exact methodology and analysis is published in the article itself so that readers can also critique for themselves. I think it's a little silly for people (who may be experts in their own fields, but with zero experience interpreting scientific data/studies) to be dismissing a large body of published work that has already been scrutinized by the scientific and engineering community and overwhelmingly points to the same conclusion (specifically that differences among power amplifiers are not likely audible).

Before you think I'm just being defensive, read what I've written above about my results with controlled testing and my conclusions.

You're badly mistaken if you think everybody here who sees flaws in current ABX protocols is a used car salesman. To point to the best example I know, wavoman has a PhD in statistics and is an established researcher in his field. He is trying to modify the typical ABX protocol.

Some people dismiss blind tests because they don't like the results and then turn around and say non-blind tests are right. With all that's known about psychological effects, I think non-blind tests can never be valid.

However, if people (scientists or not) have sound objections to current ABX protocols while maintaining blinded testing, they should try to modify the protocols then come back here and report on their methodology and test results. Then we can judge for ourselves the validity of the experiment.

And yes, I'm a scientist. I read buckloads academic articles and get cranky about methodologies and research theory, read peer reviewed articles, and all that. I think it's bad science to say "There are 20 papers and so all further research is stupid."

I think there's enough historical evidence to demonstrate that science should always welcome new ideas if they follow the scientific method, which some people on Head-Fi (although many are not) are trying to. Otherwise scientists end up looking very stupid when, after chanting "This claim that has no evidence to support it and is therefore stupid" for years, are shown to be wrong by some rogue who "Doesn't know their place or understand science."

So, for people trying to come up with new ABX protocols and who are willing to submit it to to the scientific process, I applaud you.

 

[moonboy403]

I just read something interesting regarding audio blind testing and it might be of interest to some here:

"The right brain/left brain theory applies here. You use one side
of your brain for objective logical kinds of processing, and the
other for subjective and emotional. When you are enjoying music,
the subjective half is in play, and its process does not relate to
the specs very well.

The same phenomenon explains why A/B comparisons, no matter
how well staged, do not resolve what audiophiles claim to hear.
When you are sitting there trying to hear the differences
between products, the objective side of your brain begins
working and your subjective responses get locked out by the
pressure to make an objective decision."

Perhaps that might bridge the gap between the two camps?

Any thought and comment?