[sfoclt]

http://www.newyorker.com/online/blogs/frontal-cortex/2012/06/wine-taste.html

 

[purrin]

Quote:

http://www.newyorker.com/online/blogs/frontal-cortex/2012/06/wine-taste.html

 
There are limitations to blind on-the-spot AB testing, but that's an entirely another subject!
 
What I would like to do is blind long term testing. Put the DACs in a box and have my wife mess with the switch every other day. See if I can tell the difference. LOL, that would be funny, and probably a more meaningful test.

 

[billybob_jcv]

Quote:

 
There are limitations to blind on-the-spot AB testing, but that's an entirely another subject!
 
What I would like to do is blind long term testing. Put the DACs in a box and have my wife mess with the switch every other day. See if I can tell the difference. LOL, that would be funny, and probably a more meaningful test.

 
You must have a MUCH more tolerant wife than I do...  
 
If I said "Honey would you help me with some audio testing?"  I would get a middle-finger salute and 20 minutes of haranguing about my stupid obsessions, and the general idiocy of all things geek...

 

[CashNotCredit]

I agree with you completely.

A large amount of non-headphone recommended products on this site go do nothing toward a true audiophile's quest to find the best sound possible. I think that anytime you come into a group like this, some members will have a feeling of "Me too!", and buy things just to fit in. It's like high school all over again. Nothing says "I HAVE MONEY AND HAVE SUCCEEDED IN LIFE!" than blowing $500 on a cable, right? The issue comes when people recommend these products that have been scientifically proven to do next to nothing, and cost more than the headphones themselves. A large portion of what you'll find here is completely snake oil. 

Most reviews of products contain large amounts of expectation bias. If you get your $1000 DAC or your $300 cable or your $800 tube amp in the mail and expect everything to sound awesome, then your brain will look for ways to interpret what you hear from these devices as awesome. I know that ASIO/WASAPI drivers do literally nothing over DirectSound, as long as the volume's turned up all the way in Windows and I mute all other applications. But when I listen on ASIO/WASAPI, I think that things "sound" better. It's all placebo. BS. Nonsense. The audio is bit-identical across all-3 drivers. But, somewhere along the line, I told myself that "ASIO SOUNDS BETTER THAN DIRECTSOUND", so now, I'm biased towards it.

Apply this same principal to things that cost hundreds, if not thousands, of dollars, and you have an issue. Get these people in a community where they can recommend things to other people, using their subjective bias as proof, and it suddenly becomes much more severe.

 

[sfoclt]

I agree that on-the-spot testing results sometimes can obscure real differences that are (or might be) there.  Still, I did find parts of that wine article interesting.  I'm definitely no wine expert so it's easier to read the results without having a personal rooting interest.  Not having the original studies I don't know how much weight to give them but they're still interesting to compare to a lot of hi-fi talk:
 
 
 
The perceptual ambiguity of wine helps explain why contextual influences—say, the look of a label, or the price tag on the bottle—can profoundly influence expert judgment. This was nicely demonstrated in a mischievous 2001 experiment led by Frédéric Brochet at the University of Bordeaux. In the first test, Brochet invited fifty-seven wine experts and asked them to give their impressions of what looked like two glasses of red and white wine. The wines were actually the same white wine, one of which had been tinted red with food coloring. But that didn’t stop the experts from describing the “red” wine in language typically used to describe red wines. One expert praised its “jamminess,” while another enjoyed its “crushed red fruit.”

The second test Brochet conducted was even more damning. He took a middling Bordeaux and served it in two different bottles. One bottle bore the label of a fancy grand cru, the other of an ordinary vin de table. Although they were being served the exact same wine, the experts gave the bottles nearly opposite descriptions. The grand cru was summarized as being “agreeable,” “woody,” “complex,” “balanced,” and “rounded,” while the most popular adjectives for the vin de table included “weak,” “short,” “light,” “flat,” and “faulty.”

 

[nick_charles]

Quote:

 
There are limitations to blind on-the-spot AB testing, but that's an entirely another subject!
 
What I would like to do is blind long term testing. Put the DACs in a box and have my wife mess with the switch every other day. See if I can tell the difference. LOL, that would be funny, and probably a more meaningful test.

 
Tom Nousaine compared rapid switching blind testing versus long term blind testing. The rapid switching blind testing was much better for detecting the presence of 4% distortion or not in a CDR. In the long term tests listeners did no better than chance (10/16) , in the rapid switching tests a listener who got it wrong in long term tests got it right !
 
http://www.nousaine.com/pdfs/Flying%20Blind.pdf

 

[Shike]

Quote:

I got the impression he was saying the two DACs sounded different and one sounded better than the other.
I didn't get night and day out of his statement.
 
What were your impressions/results of the 4 DACs?

 
I couldn't tell them apart honestly, which explains my position quite easily.
 
@Purrin
 
Quote:

EDIT: Blind test! Blind test! Why is it that the fringe objectivists never shout "Blind test! Prove it" when the ODAC defeats a $3000 DAC but not the other way around. Hmmm, strange how standards are applied.

 
As someone that relies on the scientific method (according to your hobby) you should already know.  We accept the null hypothesis till there is proof otherwise.  As for the proof itself, there has to be a certain reproducibility too.  Last but not least, many will also consider occam's razor.  Many would argue the possibility there was a flaw in your test is substantially higher than the possibility of there being an audible difference.

 

[Chris J]

And a Social Scientist friend of mine (who's opinion I trust) tells me that DBT does not work, and there is a large body of scholarly opinion backing this up.
 
Oh!  What a ball of confusion!

 

[estreeter]

Gotta love the way we keep flogging the same dead horse on HF year in and year out. And the way Voldemort's newfound faithful seem to think this debate is somehow new and that they alone are the first to swing by with these 'revelations'.  Just accept that the ODAC/O2 are both rare creations - DiY designs that are being assembled and sold for peanuts, largely on the back of the designer's fervent desire to prove that he can walk the walk after a few years of talking the talk. I actually think the JDS Labs casework for the standalone ODAC isnt half-bad - at least from the photos - but normally this would have a sticker price between 400 and 500 dolllars - particularly if it was sold through third-parties - lets start comparing like with like.
 

 
 
The most annoying thing about these debates is that trying to 'disprove' any of the assertions would take a lottery win - who is going to buy a ~30K dCS stack simply to be able to tell people whether they think it sounds 'way better' than their $149 ODAC ? As Purrin said earlier, anyone with an expensive DAC doesn't bother with these debates - they have spent their money and the 30-day return period is up. Its great that V has given us another choice - for minimal bucks - but does his rabid fanbase have to beat everyone else over the head with it ? 

 

[Shike]

I don't have an ODAC myself, but have advocated low priced DACs for quite a while . . . at least relatively with certain measured criteria*

 

[JadeEast]

Quote:

Because he's not a politican who always has to be politically correct lest his followers desert him.  He's a hobbyist who spends too much time and effort in the hobby to see it ruined and occasionally voices his thoughts about how and why they ruin it.

I care too and didn't think that characterizing people who questioned things as poor souls added much to what was a good response to the issue at hand. Framing a person's character in an unfavourable light ( as was my interpretation of the comment) adds no new information to the question at hand. 
Quote:

Because one aspect of this is science. The science behind the motivation behind these types of posts. As we all know, motivation can skew outcomes. I myself have been criticized for this and I think it's a valid criticism. I think it would be interesting to graph income mean/averages and distribution on a bell curve for the following two groups:
 

I'm all for discussing motivation, values, belief, authority, and limits of knowledge, but I'm not sure that it's the domain of science. 

 

[nick_charles]

Quote:

And a Social Scientist friend of mine (who's opinion I trust) tells me that DBT does not work, and there is a large body of scholarly opinion backing this up.
 
Oh!  What a ball of confusion!

 
Which particular "Social Science" is his/her domain ?
 
I would personally ask for a handful of citations from this body of scholarly opinion as DBT is extremely widely used in a large number of disciplines not least of which are medicine and pharmacology. If DBT is seriously flawed I'd like to know before I take any medication

 

[JadeEast]

Quote:

What I would like to do is blind long term testing. Put the DACs in a box

 

 

[TWIFOSP]

Quote:

 
Which particular "Social Science" is his/her domain ?
 
I would personally ask for a handful of citations from this body of scholarly opinion as DBT is extremely widely used in a large number of disciplines not least of which are medicine and pharmacology. If DBT is seriously flawed I'd like to know before I take any medication

 
Clinical trials do NOT use ABX or double blind testing methodologies in a way that can be compared to a binary yes/no audio test.  
 
Double blind tests in the audio realm are extremely flawed.  I have yet to see one that qualifies what the test subjects are capable of hearing prior to testing for it.  Testing for something without knowing what your measurement system (the humans) is capable of, is useless.  How do you know that the test subjects weren't capable of detecting slight changes that other people are capable of?  Given the variability of human performance in every other background, that isn't a wild theory at all.  
 
A measurement systems analysis should be conducted prior to any double blind test.  A simple one in this case would be to introduce known changes to the frequencies and measure at what accuracy the user can detect them past random chance.  You've then established a baseline for how capable your listener is.  If this sounds [size=10pt]familiar [/size], it's because it is essentially what audiologists use to measure people's ability to process speech sounds.
 
I'm not trying to get into a dick measuring contest, but my background is in statistics and testing methodologies.  I currently design experiments for in house clinical trials for a large medical device company, so I know a thing or two about good and bad science.

 

[estreeter]

Quote:

 
Which particular "Social Science" is his/her domain ?
 
I would personally ask for a handful of citations from this body of scholarly opinion as DBT is extremely widely used in a large number of disciplines not least of which are medicine and pharmacology. If DBT is seriously flawed I'd like to know before I take any medication

 
This may come as a bitter pill for you - 
 
http://www.avguide.com/forums/the-difference-between-medical-dbts-and-audio-dbts
 
This sums it up for me:
 
medical DBT's are massive. They involve thousands of patients with strict entry criteria (the disease being studied is strictly defined, you can not have other medical conditions which may interfere with data interpretation, you must be of a certain age, etc etc). These studies are carefully designed, take months or years to complete, months to analyze, and then months for the peer review process and finally publication.
 
Anyone who has had anything to do with academia will know how closely any and all data is scrutinised and any perceived weakness in the methodology behind a given study is ruthlessly examined. Having a sample size of yourself, your partner and the car just isnt going to cut the mustard.
 
The other excellent point is that we somehow approach the DBT as an end in itself - that isnt the case with medical DBTs. 
 
in a medical DBT, we know what we are looking for. The trials are designed to demonstrate the primary or secondary endpoint. For example - if a drug is purported to reduce the incidence of stroke, the primary endpoint would be number of new strokes per year in the treatment and control group. The study design would identify other stroke reducing drugs in both groups, and specify what is permissible and what isn't. We don't just take a new drug, give it to 5,000 patients and give placebo to 5,000 controls, and look at both populations to see what happens.
 
In an audio DBT, what is the primary endpoint? Does the testing panel even know what they are supposed to be looking for? Is there a score sheet that says "image width was xxx meters" or "frequency response: skewed to bass or treble"? Well, there isn't. You are expected to notice a difference, whatever it is, and then use that as a basis for comparison.
 
Audio DBT is less science and more self-fulfilling prophecy - grow a beard and carry a cross long enough and someone is bound to subscribe to your religion.