Wining about the placebo effect
Feb 8, 2013 at 8:38 PM Thread Starter Post #1 of 105

scuttle

500+ Head-Fier
Joined
May 23, 2009
Posts
668
Likes
43
http://blindtaste.com/2012/06/13/the-judgment-of-princeton/
 
 
Last Friday, nine judges blind-tasted twenty wines, some from France (mostly expensive) and some from New Jersey (mostly cheap), at the author George Taber’s homage to the 1976 California-vs.-France Judgment of Paris. Of those 20, there was just one bottle for which we... can infer to have done better than the others for any reason other than chance alone...

Even that single winning wine—the Drouhin Clos des Mouches 2009, a white Burgundy—was ranked dead last (#10 out of 10) by one of the nine judges, and tied for second-to-last by two others. There was one red wine that seemed to have done worse than the others in a meaningful way—the Four JG’s Cabernet Franc 2008, from New Jersey—but as for the rest, it was an 18-way tie (or, to be more accurate, two nine-way ties). If the entire Judgment of Princeton were replayed, from start to finish, on another day, we wouldn’t even have enough evidence to conclude that any one of those 18 wines would be likely to do better or worse than any other.

...Regardless, the fact that a $500 bottle can’t set itself apart definitively, statistically, from one that costs under $20, even with ten wine experts comparing the two, casts us right back out to Neverland.

 

 
All hail the New Jersey wine industry!
 
Feb 10, 2013 at 3:38 AM Post #2 of 105
This is the millionth example of why (fast switch) Blind Testing  is an unreliable method of judging certain types of subjective differences.
How many more millions of examples do sound scientists need before they take the fallibility of (fast switch) blind tests more seriously?.
 
Blind tests are brilliant at removing placebo/expectation bias, but rubbish in some other respects.
I realise that it's theoretically possible to arrange a blind test to appease the blind test skeptic, but in practice they're mostly done like the wine tasting event - at which our poor human minds just fall apart.
 
I've said this before, but I know it just gets ignored or shot down, so I may not necessarily say it again...
smile_phones.gif

 
Feb 10, 2013 at 4:31 AM Post #3 of 105
But the research out there seems to indicate that fast switching is better than long-term comparisons (higher power, yields significantly better positive detection rate of differences), at least for audio.  I don't know if there's a difference when looking at other senses like taste, but I don't think that's the most likely scenario.
 
It should be noted that for this wine test, people were asked to rank rather than see if they could tell the wines apart.  It's possible that different people just had different valid preferences that would hold from test to test, though that doesn't seem like the most plausible explanation at all.
 
As always, a null result or something similar is not that exciting and leaves a lot of skeptics to believe they have wiggle room to question the results in terms of the "what about X?" factor.  Sometimes there are some pretty valid concerns, as a lot of experiments are not run very well, or they have limited resources.  That said, barring some egregious testing conditions, it really does cast into doubt any claimed ability to strongly distinguish the best from the worst on content alone rather than reputation.  Differences that are claimed to be easily apparent should still be at least discernible under pretty much any conditions, right?  
 
Maybe a blind testing procedure doesn't have enough power to yield significant results when testing really really small differences?  Maybe some hypothetical difference is on the fringe of audibility.  But if some golden ears / tongues / whatevers can't figure it out when trying really hard to tell the difference, it's just not going to be a big enough deal for me to worry about.  To the hardcore obsessives that care anyway, I salute you.
 
So non-blind testing?  Problem is, when effects of expectation bias seem to be a lot greater (but possibly inconsistent, hard to predict exactly) than any effect you're testing for, can you really justify any non-blind test and analyze that with a straight face?
 
Feb 10, 2013 at 6:40 AM Post #4 of 105
Quote:
This is the millionth example of why (fast switch) Blind Testing  is an unreliable method of judging certain types of subjective differences.
How many more millions of examples do sound scientists need before they take the fallibility of (fast switch) blind tests more seriously?.
 
Blind tests are brilliant at removing placebo/expectation bias, but rubbish in some other respects.
I realise that it's theoretically possible to arrange a blind test to appease the blind test skeptic, but in practice they're mostly done like the wine tasting event - at which our poor human minds just fall apart.
 
I've said this before, but I know it just gets ignored or shot down, so I may not necessarily say it again...

 
I'm not surprised that it get ignores. Because you haven't given anybody a reason why you might possibly be right...
 
Feb 10, 2013 at 10:16 AM Post #5 of 105
I promised myself I wouldn't get into these kind of debates, as they never end well. I just wanted to make one simple point. I'll try just once to explain it more fully:
 
Whenever you get one of these wine/whiskey/food blind test results, the commentator proudly declares one of two conclusions:
 
       1. There is no discernible difference between the wines, so you may as well get the cheapest one.
       2. The wine "experts" are fools, or at least no better than than the average Joe, so you may as well get the cheapest wine.
 
The commentators rarely offer the 3rd possibility: The test was flawed, so that no serious conclusion could be drawn from it.
The reason they ignore the 3rd possibility is that they consider a blind test to beyond reproach.
 
Now, for any wine enthusiast, option 1 is nonsense. Of course there are differences between the wines - differences that are worth spending money on. I shouldn't need to spell this out to anyone who likes their wine.
 
Option 2 is a possibility in isolated cases, but after a few million (slight exaggeration) similar results, this option becomes highly unlikely.
 
This leaves Option 3 as very plausible, at least worthy of further investigation. In isolated cases, it could be that the particular test wasn't conducted well. But afer a few million similar results,you start to question if blind testing itself has some limitations for certain kinds of subjective tests.
This is my reasoning that I "might possibly be right".
 
FWIW, I think that blind testing is valuable in many circumstances. And that sighted or non-rapid switch tests aren't that great - for all the reasons previously mentioned in this forum.
 
Feb 10, 2013 at 10:47 AM Post #6 of 105
Quote:
I promised myself I wouldn't get into these kind of debates, as they never end well. I just wanted to make one simple point. I'll try just once to explain it more fully:
 
Whenever you get one of these wine/whiskey/food blind test results, the commentator proudly declares one of two conclusions:
 
       1. There is no discernible difference between the wines, so you may as well get the cheapest one.
       2. The wine "experts" are fools, or at least no better than than the average Joe, so you may as well get the cheapest wine.
 
The commentators rarely offer the 3rd possibility: The test was flawed, so that no serious conclusion could be drawn from it.
 

 
Why are you wasting time re-stating this? If there you have a reason you believe the test is flawed then we are ready to listen.
 
 
 
The reason they ignore the 3rd possibility is that they consider a blind test to beyond reproach.
 
Now, for any wine enthusiast, option 1 is nonsense. Of course there are differences between the wines - differences that are worth spending money on. I shouldn't need to spell this out to anyone who likes their wine.
 

 
It is beyond doubt that large groups of people can persist in making false distinctions for years. That you personally would feel like an idiot if the test is correct does not invalidate the test.
 
 
Option 2 is a possibility in isolated cases, but after a few million (slight exaggeration) similar results, this option becomes highly unlikely.
 
This leaves Option 3 as very plausible, at least worthy of further investigation. In isolated cases, it could be that the particular test wasn't conducted well. But afer a few million similar results,you start to question if blind testing itself has some limitations for certain kinds of subjective tests.
This is my reasoning that I "might possibly be right".
 
FWIW, I think that blind testing is valuable in many circumstances. And that sighted or non-rapid switch tests aren't that great - for all the reasons previously mentioned in this forum.
 

 
Ok: so you have no argument other than "For reasons previously mentioned in the forum." Why did you waste time posting?
 
Feb 10, 2013 at 11:03 AM Post #7 of 105
This is the millionth example of why (fast switch) Blind Testing  is a reliable method of judging certain types of subjective differences.
 
Science says there are no differences. A huge group of people say there are differences. So either science is wrong and all the advances we have made over the centuries are. . . idk magic?
 
Or people generally have ******** opinions based on no real evidence except their flawed senses. 
 
The later is obviously the case for anyone with any sense. 
 
So stop whining about people whining about the placebo effect because you only serve to make yourself look silly with nonsense arguments. "Whining" about general lack of acceptance of a proven phenomenon isn't a bad thing. 
 
Feb 10, 2013 at 11:06 AM Post #8 of 105
As "j.j." would say, as blind test procedure & set-up should be itself tested.  Using a null test where both A & B are the same sample and a positive test where A & B are know to be different enough to be identified.
Once again ABX tests are about JND (Just Noticeable Differences) not about preferences.
The typical complaint about doing well run ABX tests, is that the test is to sensitive, the smallest uncontrolled difference and X can be identified.
 
Feb 10, 2013 at 11:09 AM Post #9 of 105
This whole "fast swtiching" thing is a huge misunderstanding.
 
"Fast switching" doesn't mean you rapidly switch back and forth from A to B to A to B to A to B.
 
You may listen to A for as long as you wish and you may listen to B for as long as you wish. What "fast switching" refers to is that when you DO decide to switch from A to B or from B to A, that the transition be as quick as possible. For example the instantaneous switch that a switch allows versus say 10 to 15 seconds for someone to manually unplug one set of cables and plug in another set of cables. That's all that "fast switching" refers to.
 
And this has been borne out in research using KNOWN AUDIBLE DIFFERENCES. The longer it took to transition from A to B or B to A, the less reliably the listeners could actually discern subtle differences that were known to be audible.
 
Finally, the blind wine tasting linked to above had ABSOLUTELY NOTHING TO DO with there being no discernible differences among wines. It was about RANKING AND PREFERENCE. Professional wine tasters routinely undergo blind testing to demonstrate that the can taste differences between wines. If they can't then they're out of a job. But that's not what the article was about. It was about what wines the judges felt tasted best. So this notion that Two Buck Chuck TASTES THE SAME as some other, more expensive wine is a load of nonsense. The only thing that the article points out is that price and where the wine was made doesn't determine what tastes best to a given individual.
 
se
 
Feb 10, 2013 at 11:51 AM Post #12 of 105
Quote:
Also, there's no requirement for fast switching. You can conduct a DBT over a span of several days, weeks, or months, if you like.

 
Well again, the "fast switching" only refers to the length of time it takes to make the switch from A to B or B to A WHEN YOU DECIDE to make the switch. You can take as long as you like to decide when to switch.
 
se
 
Feb 10, 2013 at 11:56 AM Post #13 of 105
Quote:
Originally Posted by Satellite_6 /img/forum/go_quote.gif
 
Science says there are no differences.

 
Since science tends to stay clear of absolutes, I would amend that to say "Science says no one as yet has demonstrated there are actual audible differences."
 
se
 
Feb 10, 2013 at 11:58 AM Post #14 of 105
Quote:
The typical complaint about doing well run ABX tests, is that the test is to sensitive, the smallest uncontrolled difference and X can be identified.

 
Or as jj would also say, getting positive results is EASY. It's getting null results that are difficult. And of course by that he meant there are so many things that can lead to false positives.
 
se
 
Feb 10, 2013 at 12:00 PM Post #15 of 105
I don't know what got into me to post in this forum this morning - seemed so simple at the time.
Lesson learned. I obviously can't quite articulate the point in the way I meant it, so I'll bow out now (whilst still ahead
wink_face.gif
).
 

Users who are viewing this thread

Back
Top