Difficulty of blind testing
Jun 14, 2009 at 5:35 PM Post #78 of 117
Quote:

Originally Posted by Uncle Erik /img/forum/go_quote.gif
SeeAnyBlackDots.jpg


The black dots really are there because you can see them. Right?

Do not underestimate your brain. It is extremely difficult to control for suggestion, expectations, the influence of beliefs, not to mention the quirks that make you see - and hear - things that aren't there.

The only way to get rid of those problems is if people can't see what cable they're listening to.

No matter the methodology, no one has ever passed a blind test.

The problem isn't the methodology.



Well then I guess I better not drive a car anytime soon, since I cant trust my eyes.
rolleyes.gif
Oh, Brother.
 
Jun 20, 2009 at 2:07 AM Post #79 of 117
Quote:

Originally Posted by wavoman /img/forum/go_quote.gif

Think this way: a lot of us (trained, experienced listeners) hear differences in cables in our systems, but fail blind tests. So there are two explanations:

(a) We are all fooled by the placebo effect. We spent good money on the cables, we want there to be a difference, so we hear one.

(b) There is something about blind testing that we don't fully understand yet, and it makes passing them difficult even though we should pass.



Hi Larry

I haven't had a chance to talk to you since the last meet but what you said above about cables seems to apply to components as well. We can hear differences while listening separately but fail under direct comparison.

At the meet, I listened to SiBurning's BenchMark through my system and thought it sounded differently than my Stello, but when we A-B'd it with the Stello, we couldn't reliably tell the difference between the two.

Now the same thing has happened with another DAC I bought.

If you have the time, please check out my "Conundrum" thread and comment on what you think happened.

Regards

Eric
 
Jun 20, 2009 at 4:06 AM Post #81 of 117
My recent experience makes me even more sure that an "imagination contamination" effect exists. My current HP system is the best I've ever had (thanks to the DNA Sonett amp and other improvements I've made since then). I am hearing nuances in music that I haven't heard before in an audio system. The weird thing is, when I drive around and listen to the crappy system in my car, I'm hearing similar nuances.

This new HP system has gotten me all excited about microdynamics: the expressive potential in small changes of dynamic. I didn't realize how much I cared about microdynamics until I heard this HP system. Well, now I'm hearing subtle changes of dynamics in my car stereo. I'm hearing them in my freaking clock radio!

Let me give a visual analogy. Suppose that you love cats, and you found this particular picture, which you have been enjoying on your wall for a while:
cat_blurry.jpg



Later you discover a better, in-focus version:


cat_sharp.jpg



To me, the cat in the first picture appears to be looking slightly downward and seems a little dull-eyed. In the second picture, he is obviously alert and looking straight at the camera. For me, having seen the second picture, it changes my impression of the first picture. I can't forget any more how alert (and cute) this cat looks. I get that impression from the first picture now.

This is an obvious example, but more subtle examples from audio abound. I recently did a cable comparison, using some orchestral music as a test track. When I first put in the better cable, I heard something in a particular cymbal crash I had never heard before. It had a really neat way of swelling and obviously the percussionist was doing something fancy with aiming the cymbals. Now, it's hard NOT to hear this detail, no matter what system I'm using. I still believe this effect is much clearer with a better system, but the original obvious "new" quality is forever gone.
 
Jun 20, 2009 at 4:35 AM Post #82 of 117
Quote:

Originally Posted by mike1127 /img/forum/go_quote.gif

...This is an obvious example, but more subtle examples from audio abound. I recently did a cable comparison, using some orchestral music as a test track. When I first put in the better cable, I heard something in a particular cymbal crash I had never heard before. It had a really neat way of swelling and obviously the percussionist was doing something fancy with aiming the cymbals. Now, it's hard NOT to hear this detail, no matter what system I'm using. I still believe this effect is much clearer with a better system, but the original obvious "new" quality is forever gone.



I think you were listening for details with the new cable and you noticed one that you had not paid attention to before.... nothing more.

Check my Conundrum thread and see what happened to me. More strangeness.
normal_smile .gif


USG
 
Jun 20, 2009 at 4:55 AM Post #83 of 117
Quote:

Originally Posted by mike1127 /img/forum/go_quote.gif
My recent experience makes me even more sure that an "imagination contamination" effect exists. My current HP system is the best I've ever had (thanks to the DNA Sonett amp and other improvements I've made since then). I am hearing nuances in music that I haven't heard before in an audio system. The weird thing is, when I drive around and listen to the crappy system in my car, I'm hearing similar nuances.

This new HP system has gotten me all excited about microdynamics: the expressive potential in small changes of dynamic. I didn't realize how much I cared about microdynamics until I heard this HP system. Well, now I'm hearing subtle changes of dynamics in my car stereo. I'm hearing them in my freaking clock radio!

Let me give a visual analogy. Suppose that you love cats, and you found this particular picture, which you have been enjoying on your wall for a while:
cat_blurry.jpg



Later you discover a better, in-focus version:


cat_sharp.jpg



To me, the cat in the first picture appears to be looking slightly downward and seems a little dull-eyed. In the second picture, he is obviously alert and looking straight at the camera. For me, having seen the second picture, it changes my impression of the first picture. I can't forget any more how alert (and cute) this cat looks. I get that impression from the first picture now.

This is an obvious example, but more subtle examples from audio abound. I recently did a cable comparison, using some orchestral music as a test track. When I first put in the better cable, I heard something in a particular cymbal crash I had never heard before. It had a really neat way of swelling and obviously the percussionist was doing something fancy with aiming the cymbals. Now, it's hard NOT to hear this detail, no matter what system I'm using. I still believe this effect is much clearer with a better system, but the original obvious "new" quality is forever gone.



I've had similar experiences before. Once you hear new nuances in a recording made apparent by a high resolution system, you now know they're there. I'll hear the same song on the radio in my car which has less resolution than my home hi-fi and know they're there even though I don't hear them directly.
 
Jun 20, 2009 at 5:08 AM Post #84 of 117
Quote:

Originally Posted by upstateguy /img/forum/go_quote.gif
I think you were listening for details with the new cable and you noticed one that you had not paid attention to before.... nothing more.


That's one interpretation, but why do you favor it?

EDIT: Actually I wasn't "listening for details" with the new cable any more than I usually do. This particular detail just came to my attention.
 
Jun 20, 2009 at 5:41 AM Post #85 of 117
Quote:

Originally Posted by mike1127 /img/forum/go_quote.gif
That's one interpretation, but why do you favor it?

EDIT: Actually I wasn't "listening for details" with the new cable any more than I usually do. This particular detail just came to my attention.



Because some people just don't understand how that can happen and they use psychoanalysis as a crutch to explain it, like the Greeks used Gods to explain things they did not yet understand.

Flame away, thats all I have to say for now.
 
Jun 20, 2009 at 7:01 AM Post #86 of 117
Quote:

Originally Posted by mike1127 /img/forum/go_quote.gif
That's one interpretation, but why do you favor it?

EDIT: Actually I wasn't "listening for details" with the new cable any more than I usually do. This particular detail just came to my attention.



Hi Mike

(This is a tough medium to deal with because you can't hear my voice, but I want to say the following with the utmost respect for a fellow caser.)

Why do I favor that interpretation?

Well, it was a new piece of equipment to audition..... and you were auditioning it the way we all audition new equipment, .... there were expectations because you referred to it as the "better cable", ...and so it was, for you, the "better cable".

Did you get to read Wavoman's explanation of what happened to me in my Conundrum thread? I couldn't have been more convinced that the two DACs had different sound signatures.... until I A-B'd them through my GS-1. It was a humbling experience.

Btw, I read your comments about the DNA amp and was wondering if your DT880s were '03 or '05?

Regards

USG
 
Jun 20, 2009 at 7:18 AM Post #87 of 117
Quote:

Originally Posted by upstateguy /img/forum/go_quote.gif
Hi Mike

(This is a tough medium to deal with because you can't hear my voice, but I want to say the following with the utmost respect for a fellow caser.)

Why do I favor that interpretation?

Well, it was a new piece of equipment to audition..... and you were auditioning it the way we all audition new equipment, .... there were expectations because you referred to it as the "better cable", ...and so it was, for you, the "better cable".

Did you get to read Wavoman's explanation of what happened to me in my Conundrum thread? I couldn't have been more convinced that the two DACs had different sound signatures.... until I A-B'd them through my GS-1. It was a humbling experience.

Btw, I read your comments about the DNA amp and was wondering if your DT880s were '03 or '05?

Regards

USG



Hi USG,

I understand your respect and I will try to be respectful as well (I confess this topic gets me "hot under the collar" sometimes because I find my careful thinking dismissed by one side or the other).

The DT880s are the 250 ohm 2005 version.

Well, I certainly acknowledge that expectation bias is there. To me there are several possibilities, and I find them all worth thinking about. I think there are many good reasons to believe that the test protocols themselves the the problem.

Here is a quote from you:

Quote:

First, I'd like to say that until you volume balance you don't have a clue. Second, unless you can A-B during a track, you're relying on your audio memory, and I don't care who you are, your audio memory is lousy, and by the time you disconnect and reconnect cables and reset the volume you have lost your reference point and any impression you think you have is extremely unreliable. This has happened to me more than once and I'm here to tell you that Crow does not taste good. All my initial impressions were nothing more than imagination.


It appears that you used a quick-switch comparison technique, and perhaps even switched while the music was in progress. You also talk about "audio memory" in a certain way. It may help to clarify my position on "audio memory"--see this thread:

http://www.head-fi.org/forums/f133/r...-sound-430023/

The particular qualities that set DACs apart in sighted listening are qualities I cannot perceive during a quick switch test, especially while the music is in progress. (If you switch while the music is in progress, you are not controlling the signal you are using---it is changing even as you switch.) If you don't switch while the music is in progress, but your concept of audio memory is "short", then you have to use small snippets, which also completely obscure the musical qualities I care about.

So out of the many interesting explanations we have for your experience, and for my experience, it's possible that your test protocol was creating an insensitivity to the very things you wanted to test.

Let me clarify: I am not opposed to blind testing. I will be conducting some this weekend, now that I found a friend who is also interested and can help me switch cables while I'm not looking. I just think that we need to find sensitive protocols and I'm not convinced anyone has done a good job of that.
 
Jun 20, 2009 at 7:20 AM Post #88 of 117
Quote:

Originally Posted by upstateguy /img/forum/go_quote.gif
Hi Mike

(This is a tough medium to deal with because you can't hear my voice, but I want to say the following with the utmost respect for a fellow caser.)

Why do I favor that interpretation?

Well, it was a new piece of equipment to audition..... and you were auditioning it the way we all audition new equipment, .... there were expectations because you referred to it as the "better cable", ...and so it was, for you, the "better cable".

Did you get to read Wavoman's explanation of what happened to me in my Conundrum thread? I couldn't have been more convinced that the two DACs had different sound signatures.... until I A-B'd them through my GS-1. It was a humbling experience.

Btw, I read your comments about the DNA amp and was wondering if your DT880s were '03 or '05?

Regards

USG



After rolling lots of tubes and cables I came to realize nothing about my hearing or hearing method changed when a new component was added. The only best way for me was to spend more time with each component to truly appreciate what it had to offer, good or bad, as compared to what it replaced.

As with most things, a rushed judgement is a bad one. Factor that into blind testing, along with all the other variables and what you come up with is people need to decide these things for themselves. Blind testing will never = some kind of blanket statement. A confused statement maybe, but never one that that has any significant results worth mentioning. Unless you intend to stay blindfolded for a few weeks at a time.
 
Jun 20, 2009 at 8:37 AM Post #89 of 117
Quote:

Originally Posted by mike1127 /img/forum/go_quote.gif
Hi USG,

I understand your respect and I will try to be respectful as well (I confess this topic gets me "hot under the collar" sometimes because I find my careful thinking dismissed by one side or the other).

The DT880s are the 250 ohm 2005 version.

Well, I certainly acknowledge that expectation bias is there. To me there are several possibilities, and I find them all worth thinking about. I think there are many good reasons to believe that the test protocols themselves the the problem.

Here is a quote from you:



It appears that you used a quick-switch comparison technique, and perhaps even switched while the music was in progress. You also talk about "audio memory" in a certain way. It may help to clarify my position on "audio memory"--see this thread:

http://www.head-fi.org/forums/f133/r...-sound-430023/

The particular qualities that set DACs apart in sighted listening are qualities I cannot perceive during a quick switch test, especially while the music is in progress. (If you switch while the music is in progress, you are not controlling the signal you are using---it is changing even as you switch.) If you don't switch while the music is in progress, but your concept of audio memory is "short", then you have to use small snippets, which also completely obscure the musical qualities I care about.

So out of the many interesting explanations we have for your experience, and for my experience, it's possible that your test protocol was creating an insensitivity to the very things you wanted to test.

Let me clarify: I am not opposed to blind testing. I will be conducting some this weekend, now that I found a friend who is also interested and can help me switch cables while I'm not looking. I just think that we need to find sensitive protocols and I'm not convinced anyone has done a good job of that.



Hi Mike

Let me recap a little:

I listened to the new DAC over a 2 week period. I identified various aspects of it's sound and was able to pick them out every time I turned on the rig. I was also very familar with the sound of my Stello, having listened to it for a long time. I swapped the DACs back and forth a number of times and each time I was able to easily hear the differences between the two, which were not insignificant. There were certain things I had identified each one could do.

It was during the last 2 hours of that 2 week period that I hooked them up in tandem to better be able to hear the differences I had identified. Just as you were able to hear identifying traits from your better cable I was able to hear identifying traits from the new DAC. When I first hooked them up, they had the sound signatures I had previously described but as I started to volume balance I began to realize that I couldn't tell the difference between the DACs any longer. As I switched the optical leads and the inputs on my GS-1back and forth it became confusing which DAC I was listening to because all the identifiers had melted away. As I went from A to B there was absolutely no change in the sound what so ever. I was so surprised by this that at one point I thought that I might have had only one DAC attached to the inputs and stopped to recheck the leads in the back of the amp. The part of my "rant" you quoted shows the utter frustration I had after making statements about the DACs and then finding out that I couldn't tell them apart.

There was only astonishment in what happened.

Imagine flipping the A-B switch and absolutely nothing happens. The music went on as though the switch was connected to nothing. It was quite unsettling.

The really unnerving part is that the preconceptions I developed during the 2 weeks of listening did not go away when I went back to listening separately.

Wavoman, who has a PhD in testing says, "I am not surprised by your conclusion. The first time around you were polluted by poor audio memory, placebo, level differences, the works."

Eric
 
Jun 20, 2009 at 1:04 PM Post #90 of 117
I'm not surprised by your conclusion because you said it yourself, you started to panic when you couldn't hear a difference. Panicking like anti-cablers is the last thing you want to do when doing something as subtle as comparing dac's.
 

Users who are viewing this thread

Back
Top