[Joe Bloggs]

Wellll... yes you do... i f you want your conclusions to be generally useful.

They conducted a very nice test of three different amplifiers using a single sample of a single model of headphone.
However, we all know that different headphones make it easier or more difficult for us to hear certain types of information, and emphasize different things.
Therefore, from that limited test, we really cannot reach conclusions about "all headphones" or "all headphone amplifiers".
All we can conclude is that their results are probably correct with the single pair of headphones they used and the particular amplifiers they used.
(And, with luck, they won't change much with production units of those models from next month.)

One serious limitation is that they used ONLY high impedance headphones.
This means that they would have totally missed issues related to the output impedance of those headphone amplifiers.
(If the amplifiers' output impedance is high, then they will interact with, and sound different with, different headphones - especially low impedance models.)
I would have tried it with at least a few different models of different impedance ranges.

As it sits, you can assume that their results are somewhat valid if you plan to use high impedance headphones, but may not even be close if you use low impedance headphones.
This is a perfect example of how, when you know the details of the test, you can consider both what it shows, and what it fails to take into account.

We don't need a generalized conclusion that no two DAC/amps sound different, we need ONE specific example from you that the pair you found to be different, can actually be perceived to be different.

Find those two DACs and the particular speakers you heard the difference on, conduct your test on that. Don't tell me that a man of your resources can't put together ONE system that can reveal these purported differences between DACs in a blind test and tell me with a straight face that somebody needs to start a Kickstarter campaign based on your theories.

To imply that studies need to be conducted to prove the null hypothesis for generally all types of combos on the market is obviously shifting the burden of proof where it doesn't belong.

 

[bigshot]

I'm not looking for "proof" nor do I think that's attainable, but I think we have a ways to go with doing more and better research and testing before we can reach conclusions with confidence.

You said a couple of months ago that you were going to do a careful listening test with level matching, switching and blind evaluation. How did that go?

You can criticize testing controls better if you actually employ them yourself.

I’ve given up on him Joe Bloggs. He only pretends to be interested in figuring things out. I think there is a reason why he doesn’t want to put it to the test.

 

[Phronesis]

You said a couple of months ago that you were going to do a careful listening test with level matching, switching and blind evaluation. How did that go?

You can criticize testing controls better if you actually employ them yourself.

I’ve given up on him Joe Bloggs. He only pretends to be interested in figuring things out. I think there is a reason why he doesn’t want to put it to the test.

Didn't get around the blind phase of the testing, other life stuff came up. With the sighted testing, I found that results varied with the controls used, and generally differences vanished with tighter controls. Didn't need to run any stats to conclude that. But I remain unsure of how much results of that type of testing tell us about normal listening, since the two settings involve very different types of listening experiences and perception could work differently in the two settings. I'm ok with saying that I don't yet know enough to reach any solid conclusions. I try to tell myself there can be fun in the mystery, but I'd rather just have real answers.

 

[Joe Bloggs]

Didn't get around the blind phase of the testing, other life stuff came up. With the sighted testing, I found that results varied with the controls used, and generally differences vanished with tighter controls. Didn't need to run any stats to conclude that. But I remain unsure of how much results of that type of testing tell us about normal listening, since the two settings involve very different types of listening experiences and perception could work differently in the two settings. I'm ok with saying that I don't yet know enough to reach any solid conclusions. I try to tell myself there can be fun in the mystery, but I'd rather just have real answers.

You can make your randomized trials days long between choices if you want. Nothing about blind testing stipulates picking a choice within a few minutes. The test just takes longer otherwise... And you do still need to run the same minimum number of trials for statistics

 

[KeithEmo]

I don't disagree with your first assertion.....
If you want to test a specific claim then you should in fact test that claim (same equipment, same conditions).

However, I do disagree with your assertion that there is such a thing as "an obvious null hypothesis".
Is it obvious that a human being will be able to detect if you add one part per billion of a chemical to a glass of water?
It may seem so, but, in fact, some chemicals can be detected at that absurdly low concentration, even though most cannot.
Likewise, I do not find it to be obvious that different DACs do or do not sound different.
Therefore, to me, the answer to that question is simply "unknown".... which means that either hypothesis is equally valid as a null hypothesis.

I also didn't say that a Kickstarter campaign would be necessary to find two DACs that sound audibly different.
However, even if I were to recommend two DACs and, after a thorough study, you were to find that they were indistinguishable, all that would prove was that my specific choice of examples was bad.
I would have failed to prove my assertion (that some DACs sound different); BUT YOU WOULD STILL NOT HAVE PROVEN YOUR ASSERTION (that NO DACs sound different).
What I said was that, if you want results that can reasonably be generalized to "MOST IF NOT ALL DACs SOUND AUDIBLY THE SAME", then you need to use a sizeable sample, and test them all under a variety of conditions.
You cannot assume that your assertion is correct just because a single attempt to disprove it fails.
(It's very difficult, and sometimes downright impossible, to prove a negative.)

There is no "burden of proof" here; we simply have two contradictory theories about what we might find out if we preformed an actual test.

We don't need a generalized conclusion that no two DAC/amps sound different, we need ONE specific example from you that the pair you found to be different, can actually be perceived to be different.

Find those two DACs and the particular speakers you heard the difference on, conduct your test on that. Don't tell me that a man of your resources can't put together ONE system that can reveal these purported differences between DACs in a blind test and tell me with a straight face that somebody needs to start a Kickstarter campaign based on your theories.

To imply that studies need to be conducted to prove the null hypothesis for generally all types of combos on the market is obviously shifting the burden of proof where it doesn't belong.

 

[Phronesis]

You can make your randomized trials days long between choices if you want. Nothing about blind testing stipulates picking a choice within a few minutes. The test just takes longer otherwise... And you do still need to run the same minimum number of trials for statistics

But that introduces its own problems. If you listen to A and B for say an hour each, how are you going to compare them? In the span of an hour, there's stuff going on with multiple time frames of memory, fatigue becomes a factor, emotions become a factor, etc. Randomization doesn't fix that, because there's a fundamental problem with how the 'data' is being gathered in the first place. The basic issue is that the ear/brain is a very complex and variable 'measuring instrument' that we have only a limited understanding of. It's very different from situations where measurements can be made relatively objectively, then you just do you statistics on the results to see if there's statistical significance.

 

[bigshot]

Didn't get around the blind phase of the testing, other life stuff came up. With the sighted testing....

I'm really not all that interested in sighted testing. I know that won't give accurate results, particularly when bias is so strong.

1) Line level matched
2) Direct A/B switchable synchronized to the same spot in the track
3) Blind testing

That is how you compare to similar sounds correctly. It isn't difficult. I think you're just coming up with excuses. It seems to be contagious.

I'm not even looking for a big scientific study. All I want is an example of two DACs that sound clearly different in normal use. That doesn't require rocket science or lab coats or kickstarters. Everyone seems convinced that such a thing exists. Why is it so hard to just find an example and have a few people listen to it carefully and verify that there is a clear difference? That should be really easy. But every listening test like this that I've ever seen ends up deciding that there is no clear difference between DACs, even ones with radically different price points. All of the careful listening tests I've done bears that out too.

I think "common knowledge" in audiophile forums is wrong in this case. No big deal. Audiophile common knowledge has certainly been wrong before!

 

[Phronesis]

I'm really not all that interested in sighted testing. I know that won't give accurate results, particularly when bias is so strong.

1) Line level matched
2) Direct A/B switchable synchronized to the same spot in the track
3) Blind testing

That is how you compare to similar sounds correctly. It isn't difficult. I think you're just coming up with excuses. It seems to be contagious.

No need to make it personal ...

You might be surprised by what you can learn by doing variations on sighted testing protocols. Gives some real insight on how biases, and how they can be counteracted or not. For example, the effect of varying switching time was very obvious. This kind of experimentation is part of doing 'sound science'. I'm interested in both differences in gear and how our brains deal with gear and differences in gear.

 

[bigshot]

I'm not intending to be insulting. I am just pointing out that you appear to have a very specific point of view and can't seem to see things in any other way. Everyone is biased. That isn't an insult. That is a fact. I'm biased too. That's why I use careful controls when I compare equipment so I don't get biased results.

And I totally see the value of sighted tests. If you don't care if differences are slight, and you only care if a difference is big enough to matter, a sighted test is fine. It's still a good idea to level match and do direct A/B switching though. Those things can make a big difference.

I really don't care about how my brain processes sound subjectively though. I can't change that by buying new equipment. And I am guessing that a brain transplant would be beyond my budget. I have to just live with it.

 

[Phronesis]

I'm not intending to be insulting. I am just pointing out that you appear to have a very specific point of view and can't seem to see things in any other way. Everyone is biased. That isn't an insult. That is a fact. I'm biased too. That's why I use careful controls when I compare equipment so I don't get biased results.

And I totally see the value of sighted tests. If you don't care if differences are slight, and you only care if a difference is big enough to matter, a sighted test is fine. It's still a good idea to level match and do direct A/B switching though. Those things can make a big difference.

I really don't care about how my brain processes sound subjectively though. I can't change that by buying new equipment. And I am guessing that a brain transplant would be beyond my budget. I have to just live with it.

Actually, I think sighted tests can be unreliable or misleading even for fairly big differences, depending on the controls used.

To deal with bias, my skepticism is fairly sweeping. I'm not taking a point of view other than saying "I don't know," since I don't have convincing enough to draw conclusions. If you want to call that a bias, I'm ok with that.

 

[bigshot]

Actually, I think sighted tests can be unreliable or misleading even for fairly big differences, depending on the controls used.

Are you reading what I'm writing? I'm trying to be very clear and your reply is as if I never said it. Specifically...

"I totally see the value of sighted tests. If you don't care if differences are slight, and you only care if a difference is big enough to matter, a sighted test is fine. It's still a good idea to level match and do direct A/B switching though. Those things can make a big difference."

I totally don't believe that your skepticism is capable of overcoming your bias. Unconscious bias is unconscious. You can't control that. I would think that you would be the first to admit that. Convincing you of something is entirely different the dealing with bias. I don't think that is very easy to do, but I'm trying to help you figure out how to convince yourself. You just aren't helping! I haven't given up on you like several others in this thread. I still see hope for you if you'd just get off your theoretical duff and start doing some controlled testing for yourself.

 

[Phronesis]

Are you reading what I'm writing? I'm trying to be very clear and your reply is as if I never said it. Specifically...

"I totally see the value of sighted tests. If you don't care if differences are slight, and you only care if a difference is big enough to matter, a sighted test is fine. It's still a good idea to level match and do direct A/B switching though. Those things can make a big difference."

I totally don't believe that your skepticism is capable of overcoming your bias. Unconscious bias is unconscious. You can't control that. I would think that you would be the first to admit that. Convincing you of something is entirely different the dealing with bias. I don't think that is very easy to do, but I'm trying to help you figure out how to convince yourself. You just aren't helping! I haven't given up on you like several others in this thread. I still see hope for you if you'd just get off your theoretical duff and start doing some controlled testing for yourself.

Do you have any idea how condescending you sound? Or are you too biased to see it?

 

[sonitus mirus]

I don't disagree with your first assertion.....
If you want to test a specific claim then you should in fact test that claim (same equipment, same conditions).

However, I do disagree with your assertion that there is such a thing as "an obvious null hypothesis".
Is it obvious that a human being will be able to detect if you add one part per billion of a chemical to a glass of water?
It may seem so, but, in fact, some chemicals can be detected at that absurdly low concentration, even though most cannot.
Likewise, I do not find it to be obvious that different DACs do or do not sound different.
Therefore, to me, the answer to that question is simply "unknown".... which means that either hypothesis is equally valid as a null hypothesis.

I also didn't say that a Kickstarter campaign would be necessary to find two DACs that sound audibly different.
However, even if I were to recommend two DACs and, after a thorough study, you were to find that they were indistinguishable, all that would prove was that my specific choice of examples was bad.
I would have failed to prove my assertion (that some DACs sound different); BUT YOU WOULD STILL NOT HAVE PROVEN YOUR ASSERTION (that NO DACs sound different).
What I said was that, if you want results that can reasonably be generalized to "MOST IF NOT ALL DACs SOUND AUDIBLY THE SAME", then you need to use a sizeable sample, and test them all under a variety of conditions.
You cannot assume that your assertion is correct just because a single attempt to disprove it fails.
(It's very difficult, and sometimes downright impossible, to prove a negative.)

There is no "burden of proof" here; we simply have two contradictory theories about what we might find out if we preformed an actual test.

If one claims sample A is different than sample B, the null hypothesis is that A and B are the same. That would be obvious enough for most people that were not in the business of selling high priced DACs to audiophiles. Nothing in modern observations is truly identical if we include atoms and particles, but audible differences could be determined to be identical if a specific qualifying threshold is established. Nobody can prove a null hypothesis, such as that audio sample A is identical to B; though, a difference should be well within the realm of even basic scientific steps that just about anyone can conduct.

 

[TheSonicTruth]

Do you have any idea how condescending you sound? Or are you too biased to see it?

While bigshot and I disagree bitterly on the amount of dynamic compression used in recent pop music releases and the necessity of legacy album remasters, I do agree with him on the methodologies he suggested to ensure accurate testing of alleged audible differences between DACs and between examples of pieces of other categories of audio equipment.

 

[bigshot]

I'm attempting to be friendly and to keep the conversation light and focused. You must be misinterpreting my tone.