Head-Fi.org › Forums › Equipment Forums › Sound Science › The case for caution in interpreting null results
New Posts  All Forums:Forum Nav:

The case for caution in interpreting null results - Page 2

post #16 of 20
7/12/09 at 12:25am
Quote:
Originally Posted by Koyaan I. Sqatsi View Post
I don't see that A/B/X inherently asks such a question.

The only thing it asks is to identify X as either A or B. It places no specific demands on how the listener comes to that identification and does not force a "same/different" paradigm.

A listener may switch between A and B and choose one or the other based purely on preference. Once they've done that, then they can do the same between their preference and X, again going purely by preference. And once that's done, then the identification of X takes care of itself by way of simple logic.

For example, let's say the listener prefers A to B. If, when comparing between A and X they prefer A to X, then logic says that X is B. If they find they have no preference between A and X, then logic says that X is A.

So again, I don't see that A/B/X inherently demands any sort of same/different paradigm.

k
We agree at first, then disagree. I am saying that A/B/X does NOT force a "same/different" question. We agree on this (go back and read my post).

But it does force some very unnatural question, and you have yourself laid out the tortured logic it takes to reach a very simple conclusion, namely, what do you like better?

Just say "No" to A/B/X. Instead, play two samples for the listeners, and ask the natural question ("which do you like better"), which has 4 answers:
  1. I prefer the first sample.
  2. I prefer the second sample.
  3. I hear no difference between the the two samples.
  4. I hear a difference, but have no preference.
Now suppose you have two pieces of equipment (or cables, or digital files), A and B.

You constitiue the first sample, and then the second sample, randomly as follows:
  1. A then B
  2. B then A
  3. A then A (swindle)
  4. B then B (swindle)
I am trying at present to work out the correct statistical tests given the four possible answers above, not pooling over subjects, using "play the winner sampling", and the optimum way to assign the random weights to the four test conditions above, and working on an academic paper ... but this is not my day job, so it is a slow go. Sorry.

The idea is to avoid a number of subtle response bias problems that sensory tests suffer from, especially those in groups with leaders of the test at the front of the room (some folks try to please them, some try to foil them, both are bad).
Reply

Head-Fi's Best Sellers

post #17 of 20
7/12/09 at 12:35am
Quote:
Originally Posted by royalcrown View Post
I have personally seen several AES articles, all done in the US, that only pool subjects into broad categories, such as "expert" and "novice" distinctions, etc. No test to this date has shown a difference between so-called "golden ears" and normal regular folk.
I agree, I have seen those. I do think I recall correctly that they still compute statistics aggregated over their pools. Better than one big pool, for sure, but still not real sample size one analysis.

I do not think the issue is "golden ears". There is no judgment here. Simply this: can one given individual hear a difference between two different cables or DACS etc. when he/she is blinded during the experiment.

It is remarkable that most experiments say "no", but people non-blind say "yes" ... cannot be explained by the placebo effect alone.

Take me, I want all DACs to sound the same, so I can pay less. But I hear a difference between the cheap DAC in my Oppo CDP and the uber expensive one in my Wadia. And I only bought the Wadia (at deep discount) after much auditioning. I listened to an equally high-end Nagra and heard nothing better than my Oppo. But the Wadia blew me away, on specific passages in specific musical selections. Because I bought it, my wife will not let me buy a new car, and I pick up customers in a junker 2002 Toyota that drops plastic pieces all over the roadway. This is not fun. But I enjoy my music at night. Placebo? I doubt it.
Reply
post #18 of 20
7/12/09 at 3:42am
Quote:
Originally Posted by wavoman View Post
I agree, I have seen those. I do think I recall correctly that they still compute statistics aggregated over their pools. Better than one big pool, for sure, but still not real sample size one analysis.

I do not think the issue is "golden ears". There is no judgment here. Simply this: can one given individual hear a difference between two different cables or DACS etc. when he/she is blinded during the experiment.
If a group of trained individuals cannot notice a difference, and if a group of regular individuals also cannot notice a difference, I think it's pretty presumptuous for an individual to assume that he or she lies outside of those bell curves.

Quote:
Originally Posted by wavoman View Post
It is remarkable that most experiments say "no", but people non-blind say "yes" ... cannot be explained by the placebo effect alone.
That's like saying that people can get a perfect score on a test when they know what the answers are, and then complaining when they get a poor score on a test when they answers are hidden. Of course people can tell a difference when they know which one is which. Everybody wants to have golden ears.

Quote:
Originally Posted by wavoman View Post
Take me, I want all DACs to sound the same, so I can pay less. But I hear a difference between the cheap DAC in my Oppo CDP and the uber expensive one in my Wadia. And I only bought the Wadia (at deep discount) after much auditioning. I listened to an equally high-end Nagra and heard nothing better than my Oppo. But the Wadia blew me away, on specific passages in specific musical selections. Because I bought it, my wife will not let me buy a new car, and I pick up customers in a junker 2002 Toyota that drops plastic pieces all over the roadway. This is not fun. But I enjoy my music at night. Placebo? I doubt it.
The whole point of expectation biases, placebo, etc. is that they're subconscious. If they were so easily determined by conscious motivation, they'd be easy to control, but that's just not the case. It's not as if these biases are subject only to some people and not others. They apply to just about everybody.
Reply
post #19 of 20
7/12/09 at 11:50am
Quote:
Originally Posted by wavoman View Post
And I only bought the Wadia (at deep discount) after much auditioning. I listened to an equally high-end Nagra and heard nothing better than my Oppo. But the Wadia blew me away
The Wadia is one of the few cases where you might find a real and significant measurable difference. Wadia are renowned for their use of a particular type of filter, If you look at the FRs for their digital products they frequently have a ~3db roll off at the high end.

http://www.stereophile.com/cdplayers/156/index6.html

http://www.stereophile.com/cdplayers/189/index5.html

http://www.stereophile.com/cdplayers/540/index6.html

http://www.stereophile.com/cdplayers/592/index6.html
Reply
post #20 of 20
7/14/09 at 1:26am
nick -- I guess I like that filter. Are you suggesting I could have saved $$$ by using my equalizer (dbx, a few hundred) on the output of the Oppo? I might try that.

royal -- we're not quite communicating. I agree with you, I put no stock in "golden ears" either. But I put no stock either in the artificiality of blind testing as currently practiced.

Here's my point: serious people who understand the placebo effect, who have strong economic reasons to avoid falling prey to it, can hear differences in high-end equipment vs cheap equipment when listening quietly over long periods at home, but fail blind tests.

You would say: therefore, there are no real differences, and these people suffered from the placebo effect even though they were trying to avoid it (it is subconscious, as you said).

My rejoinder is this: you may be right, but it also plausible that today's blind tests are flawed.

We need more work -- much more -- to figure out which position is right. I have no dog in this fight ... I don't care how it comes out, I only want the truth. But I am not ready to believe these very artificial ABX tests, that in no way control for response bias nor allow for certain types of sensory discrimination (which may or may not exist) to come in to play.
Reply
New Posts  All Forums:Forum Nav:
  Return Home
  Back to Forum: Sound Science

Head-Fi's Best Sellers

Head-Fi.org › Forums › Equipment Forums › Sound Science › The case for caution in interpreting null results