[KeithEmo]

Someone suggested the idea of a Kickstarter campaign to finance some tests.
I think that would be an excellent idea.

I just want to make it perfectly clear that I am 100% behind doing proper testing.
But I am very much against reaching premature conclusions based on the results of inadequate tests.
I've designed proper scientific tests - which is why I'm unwilling to ignore serious flaws when I see them.
It's not really all that difficult to do it right.
However, sadly, it rarely seems to work out that way.

And, when it comes to testing, the biggest part of the cost is usually time and labor.
This is what makes it expensive for a company to do it - unless they are certain of a return on their cost.
On the other hand, an audio club or group of enthusiasts can save a lot of that cost by using volunteer labor.

Another way to encourage the most rigorous testing is to encourage "self selection".
You DO NOT arbitrarily choose "a good pair of headphones" to use for the test.
You hold a meet - and invite everyone to "bring their most revealing pair of headphones".

Then you create a set of test files, with varying amounts of THD, and ringing, and perhaps noise, in them.
Then you test the test equipment...
With the goal of finding out which of those headphones make the known differences easiest to notice.
This gives you the best chance at detecting audible unknown differences if they really exist...
And makes it plausible when you say you did your best to detect differences if there are any...

As far as I'm concerned, the basic methodologies of various blind testing methods aren't the big problem.
The biggest problem is simply sloppy test procedure.

For example, if you were trying to test whether a certain additive would make a noticeable change in the taste of a product, you would start by adding known amounts of the substance, and you would analyze your samples to make sure they really contained the proper amounts before letting anybody taste them. Yet, in contrast, in most of the various tests that have been run about "audible differences", nobody bothered to get out an analysis microphone and confirm whether that ringing, or ultrasonic content, or whatever, was actually ARRIVING AT THE EARS OF THE TEST SUBJECTS. (It seems pretty obvious that you cannot conclude that something is "inaudible" if you haven't first confirmed that it is arriving at the ears of your test subjects. Yet almost all tests omit this basic step, preferring to rely on the idea that "good speakers" are delivering the test signal properly, without actually confirming it.)

You guys keep on piling up the imagined reasons for why existing blind test methodologies may be flawed / not sensitive enough. But the correct action from there would be to devise and conduct improved blind tests using your purported state of the art equipment and improved methodologies, not sit here blabbing about why we should accept your SIGHTED test results. SIGHTED tests are the lowest of low in terms of methodology and can never be accepted as any sort of evidence no matter how much detail you go into the perceived differences or how baldly obvious you swear the differences are!

Really, for the amount of fruitless bickering recorded here, this thread should be locked until somebody posts results from a new blinded test he conducted!

 

[c1ferrari]

New to this forum and wanted to thank the original poster for initiating the thread.
Many pages to review, perhaps I'll catch up at some point.

 

[bigshot]

The problem - as I've personally experienced - is that we can easily and clearly perceive a difference between A and B, seemingly as clear as the difference between red and blue, which isn't objectively there.

Not if you do a blind test and randomize the order of A and B.

Many pages to review, perhaps I'll catch up at some point.

If you have better things to do, don't bother. Too many of the comments are just people arguing a point without backing it up like the original poster.

 

[castleofargh]

my point of view on all this is fairly simple(perhaps too simple?):
we have tools to confirm things, or we don't.
when our most reliable and available testing methods for audibility, fail to disprove the null hypothesis, the conclusion should be that the subjects didn't perceive a difference. even if we know that the test isn't perfect, if we don't have results from a more reliable one, what right do we have to declare that we know better? even deciding that a test isn't very reliable, requires proper evidence, else it's wishful thinking.
our scientific knowledge base, relies fully on how much confidence we can have in experiences and how much confidence we can have in what is inferred from the data. if we had to stop reaching conclusions anytime we notice that we don't have the full picture, we would still be wondering what to do with a wooden stick. so anybody arguing that we can't draw conclusions because we most likely are missing out on some stuff, to you I say: you're wrong and using double standards. the very fact that you can argue on those subjects implies that you have trusted a all lot of things and drawn a all lot of conclusions on partial data, so why the sudden need for absolute certainty just for some audio topic?

you all know how I feel about jumping to conclusions and making empty claims. but conclusion from some of the most reliable test methods we know of, are not empty claims. they're the most reliable data we have and as such represent our contemporary knowledge on that specific matter. the one most likely to be closer to the truth which is all we aspire and can aspire to. so IMO, the only legit reason to dismiss such tests and their results, is to have better method and more reliable results. anything else is dishonest because it implies rejecting the more reliable idea for the sake of a less reliable one, and that isn't an act driven by reason.

 

[Phronesis]

To really do this audio stuff like science, you need to develop theoretical frameworks which encompass all of the factors in the tests - physics, electronics, ear anatomy/physiology, neuropsychology of perception, etc. - and do a wide variety of tests with many variations of parameters. That will build up a body of data and interpretations of the data, and enable us know some things with confidence. By comparison to that ideal, the tests described in the first post of this thread seem rather limited and amateurish. A problem is that there isn't much motivation for qualified scientists (the kind with PhDs who publish in peer-reviewed journals) to undertake this kind of research, since audio gear is more of a consumer thing than a science thing.

 

[Joe Bloggs]

Someone suggested the idea of a Kickstarter campaign to finance some tests.
I think that would be an excellent idea.

I just want to make it perfectly clear that I am 100% behind doing proper testing.
But I am very much against reaching premature conclusions based on the results of inadequate tests.
I've designed proper scientific tests - which is why I'm unwilling to ignore serious flaws when I see them.
It's not really all that difficult to do it right.
However, sadly, it rarely seems to work out that way.

And, when it comes to testing, the biggest part of the cost is usually time and labor.
This is what makes it expensive for a company to do it - unless they are certain of a return on their cost.
On the other hand, an audio club or group of enthusiasts can save a lot of that cost by using volunteer labor.

Another way to encourage the most rigorous testing is to encourage "self selection".
You DO NOT arbitrarily choose "a good pair of headphones" to use for the test.
You hold a meet - and invite everyone to "bring their most revealing pair of headphones".

Then you create a set of test files, with varying amounts of THD, and ringing, and perhaps noise, in them.
Then you test the test equipment...
With the goal of finding out which of those headphones make the known differences easiest to notice.
This gives you the best chance at detecting audible unknown differences if they really exist...
And makes it plausible when you say you did your best to detect differences if there are any...

As far as I'm concerned, the basic methodologies of various blind testing methods aren't the big problem.
The biggest problem is simply sloppy test procedure.

For example, if you were trying to test whether a certain additive would make a noticeable change in the taste of a product, you would start by adding known amounts of the substance, and you would analyze your samples to make sure they really contained the proper amounts before letting anybody taste them. Yet, in contrast, in most of the various tests that have been run about "audible differences", nobody bothered to get out an analysis microphone and confirm whether that ringing, or ultrasonic content, or whatever, was actually ARRIVING AT THE EARS OF THE TEST SUBJECTS. (It seems pretty obvious that you cannot conclude that something is "inaudible" if you haven't first confirmed that it is arriving at the ears of your test subjects. Yet almost all tests omit this basic step, preferring to rely on the idea that "good speakers" are delivering the test signal properly, without actually confirming it.)

Do you really need a kickstarter campaign to go back to the equipment you "heard" the differences on and conduct some volume matched randomized trials? When it is within reach of homebrew operations like this? https://www.head-fi.org/threads/bli...nd-yamaha-stereo-headphone-amplifiers.891001/

If you do, why are you on the other hand suggesting bigshot to *buy all the gear listed in your comparison to test them out* ???

 

[bigshot]

Do you really need a kickstarter campaign to go back to the equipment you "heard" the differences on and conduct some volume matched randomized trials?

No. It's just more goalpost shifting to justify an opinion that serves a particular commercial interest.

I'm betting that if a difference in DACs exists, it is going to exist at the very bottom of the market. That is exactly why I bought a $40 Walmart DVD player. I really didn't need a DVD player. I bought it so I could compare it to my Oppo equipment head to head. I was actually expecting a difference there. But I didn't find one.

I haven't actually spent the time pricing out the cost of the two DACs he's suggesting I buy. And I don't know if they are available at Amazon where I buy all my equipment with a no questions asked return policy. Should I spend the time to google that or is it a waste of time?

 

[Jaywalk3r]

To really do this audio stuff like science, you need to develop theoretical frameworks which encompass all of the factors in the tests - physics, electronics, ear anatomy/physiology, neuropsychology of perception, etc. - and do a wide variety of tests with many variations of parameters.

No, that's not necessary in order to "really do this audio stuff like science." [C]astleofargh very eloquently described how to to this audio stuff like science in post #9949. I'll copy it here, because it bears repeating:

"we have tools to confirm things, or we don't.
when our most reliable and available testing methods for audibility, fail to disprove the null hypothesis, the conclusion should be that the subjects didn't perceive a difference. even if we know that the test isn't perfect, if we don't have results from a more reliable one, what right do we have to declare that we know better? even deciding that a test isn't very reliable, requires proper evidence, else it's wishful thinking.
"our scientific knowledge base, relies fully on how much confidence we can have in experiences and how much confidence we can have in what is inferred from the data. if we had to stop reaching conclusions anytime we notice that we don't have the full picture, we would still be wondering what to do with a wooden stick. so anybody arguing that we can't draw conclusions because we most likely are missing out on some stuff, to you I say: you're wrong and using double standards. the very fact that you can argue on those subjects implies that you have trusted a all lot of things and drawn a all lot of conclusions on partial data, so why the sudden need for absolute certainty just for some audio topic?

"you all know how I feel about jumping to conclusions and making empty claims. but conclusion from some of the most reliable test methods we know of, are not empty claims. they're the most reliable data we have and as such represent our contemporary knowledge on that specific matter. the one most likely to be closer to the truth which is all we aspire and can aspire for. so IMO, the only legit reason to dismiss such tests and their results, is to have better method and more reliable results. anything else is dishonest because it implies rejecting the more reliable idea for the sake of a less reliable one, and that isn't an act driven by reason."​

 

[Phronesis]

No, that's not necessary in order to "really do this audio stuff like science." [C]astleofargh very eloquently described how to to this audio stuff like science in post #9949. I'll copy it here, because it bears repeating:

"we have tools to confirm things, or we don't.
when our most reliable and available testing methods for audibility, fail to disprove the null hypothesis, the conclusion should be that the subjects didn't perceive a difference. even if we know that the test isn't perfect, if we don't have results from a more reliable one, what right do we have to declare that we know better? even deciding that a test isn't very reliable, requires proper evidence, else it's wishful thinking.
"our scientific knowledge base, relies fully on how much confidence we can have in experiences and how much confidence we can have in what is inferred from the data. if we had to stop reaching conclusions anytime we notice that we don't have the full picture, we would still be wondering what to do with a wooden stick. so anybody arguing that we can't draw conclusions because we most likely are missing out on some stuff, to you I say: you're wrong and using double standards. the very fact that you can argue on those subjects implies that you have trusted a all lot of things and drawn a all lot of conclusions on partial data, so why the sudden need for absolute certainty just for some audio topic?

"you all know how I feel about jumping to conclusions and making empty claims. but conclusion from some of the most reliable test methods we know of, are not empty claims. they're the most reliable data we have and as such represent our contemporary knowledge on that specific matter. the one most likely to be closer to the truth which is all we aspire and can aspire for. so IMO, the only legit reason to dismiss such tests and their results, is to have better method and more reliable results. anything else is dishonest because it implies rejecting the more reliable idea for the sake of a less reliable one, and that isn't an act driven by reason."​

I read castle’s post before I wrote mine. I stand by what I wrote. I don’t think much testing meeting scientific standards has been done so far. Let’s compile a list of papers published in peer-reviewed journals and see what we make of it. I know that peer review doesn’t guarantee anything (I’ve published peer-reviewed stuff, and have also been a reviewer), but at least the standards and expertise are generally higher than audio magazines for consumer audiences.

Here's an example of what I consider to be legit scientific research in this area:

https://www.nature.com/articles/s41598-018-24528-3.pdf

 

[KeithEmo]

I don't see this as a problem per-se... it's just something that you acknowledge and take into account.

There are plenty of optical illusions that are so persistent that, even if you KNOW it's a trick, "the lines that you know are straight look bent" or "one square looks bigger even though you know they're the same size".
This is also why we supposedly know that eyewitness identification of suspects is quite often incorrect.
However, it doesn't mean we assume that ALL eyewitness testimony is necessarily wrong, or that all lines that appear curved are actually straight.... just that we cannot trust our eyes as the final judge 100% of the time.

And, of course, I would be less inclined to believe a salesman trying to sell me something than someone working for a consumer advocacy group.
But I'm still going to bear in mind that they BOTH have their biases, either of them could be wrong, or even both of them.
(The salesman is obviously trying to sell me something; but the scientist working for the consumer advocacy group could also be trying to make a name for himself by "discovering something important", or just testing out a "pet theory".)

However, and this is important, I'm not going to consider ANYTHING without looking into the details of who is making the claim and what they've based it on.
Historically, a lot of the things we "knew" were true have in fact turned out to be wrong.
Therefore, HOW they were proven really is important, and the tests and methodology used are one of the things that is going to inform us about much weight we should place on the results.
And, yes, sloppy science, or out-of-date science is often more useful than no science at all..... but it is also more likely to turn out to be wrong.

Not if you do a blind test and randomize the order of A and B.



If you have better things to do, don't bother. Too many of the comments are just people arguing a point without backing it up like the original poster.

 

[KeithEmo]

Wellll... yes you do... i f you want your conclusions to be generally useful.

They conducted a very nice test of three different amplifiers using a single sample of a single model of headphone.
However, we all know that different headphones make it easier or more difficult for us to hear certain types of information, and emphasize different things.
Therefore, from that limited test, we really cannot reach conclusions about "all headphones" or "all headphone amplifiers".
All we can conclude is that their results are probably correct with the single pair of headphones they used and the particular amplifiers they used.
(And, with luck, they won't change much with production units of those models from next month.)

One serious limitation is that they used ONLY high impedance headphones.
This means that they would have totally missed issues related to the output impedance of those headphone amplifiers.
(If the amplifiers' output impedance is high, then they will interact with, and sound different with, different headphones - especially low impedance models.)
I would have tried it with at least a few different models of different impedance ranges.

As it sits, you can assume that their results are somewhat valid if you plan to use high impedance headphones, but may not even be close if you use low impedance headphones.
This is a perfect example of how, when you know the details of the test, you can consider both what it shows, and what it fails to take into account.

Do you really need a kickstarter campaign to go back to the equipment you "heard" the differences on and conduct some volume matched randomized trials? When it is within reach of homebrew operations like this? https://www.head-fi.org/threads/bli...nd-yamaha-stereo-headphone-amplifiers.891001/

If you do, why are you on the other hand suggesting bigshot to *buy all the gear listed in your comparison to test them out* ???

 

[KeithEmo]

Something else worth remembering is that, especially with science, there are varying degrees of "accuracy and correctness".

For example, Newtonian physics works just fine for building a building, or even targeting an artillery piece...
But, if you're calculating the route of a space probe, it's not exactly right; it isn't wrong either; it's just not right enough.

We often work with approximations, and generalizations that are usually, but not always, right...
For example, peanuts are generally nutritious, and certainly edible, except for the few people who are allergic to them.
For most of us, we don't worry about such things, and can safely ignore them.
However, we cannot simply assume that the one person in ten thousand who says that "eating a handful of peanuts would kill him" is lying either.
We keep it in perspective.

In the case of audio equipment, all that really matters when you or I buy equipment is what WE can hear, or what we notice, or find important.
Both science, and other people's reviews and opinions, are just tools we use to optimize our own experience.
And, yes, science is often an excellent safety check against claims based on pure speculation or opinion (or just plain false or mistaken claims).

I don't see this as a problem per-se... it's just something that you acknowledge and take into account.

There are plenty of optical illusions that are so persistent that, even if you KNOW it's a trick, "the lines that you know are straight look bent" or "one square looks bigger even though you know they're the same size".
This is also why we supposedly know that eyewitness identification of suspects is quite often incorrect.
However, it doesn't mean we assume that ALL eyewitness testimony is necessarily wrong, or that all lines that appear curved are actually straight.... just that we cannot trust our eyes as the final judge 100% of the time.

And, of course, I would be less inclined to believe a salesman trying to sell me something than someone working for a consumer advocacy group.
But I'm still going to bear in mind that they BOTH have their biases, either of them could be wrong, or even both of them.
(The salesman is obviously trying to sell me something; but the scientist working for the consumer advocacy group could also be trying to make a name for himself by "discovering something important", or just testing out a "pet theory".)

However, and this is important, I'm not going to consider ANYTHING without looking into the details of who is making the claim and what they've based it on.
Historically, a lot of the things we "knew" were true have in fact turned out to be wrong.
Therefore, HOW they were proven really is important, and the tests and methodology used are one of the things that is going to inform us about much weight we should place on the results.
And, yes, sloppy science, or out-of-date science is often more useful than no science at all..... but it is also more likely to turn out to be wrong.

 

[bigshot]

I stand by what I wrote. I don’t think much testing meeting scientific standards has been done so far.f

I think your particular form of bias expresses itself in not being willing to accept any evidence that conflicts with your theories of perception. You will put forward thinly verified psychological theories, but when it comes to hard and fast facts, you take the position that good enough is never good enough and we can't know anything because we can't know everything. That is pretty common among people who come to Sound Science with a preconceived conclusion. If the facts don't go their way, they start questioning the testing methodology. They don't accept any evidence until it's absolutely proven to them, and it's never quite good enough to be absolutely proven to their personal standards.

I look at things differently. I follow the prevailing winds and let the evidence set the direction. And once I find something out, I try to put that into perspective to see if it really matters. Ultimately, that way of thinking is much more practical for the purposes of improving the sound of an audio system, because you don't have to wait for all your questions to be answered before seeing improvement. And the evolution of your questioning can lead to improvements you weren't even looking for.

It's like the difference between Columbus refusing to sail to India until he knows all the logistics of how to get to the destination, and Columbus setting sail due East and discovering America as he goes.

 

[Phronesis]

I think your particular form of bias expresses itself in not being willing to accept any evidence that conflicts with your theories of perception. You will put forward thinly verified psychological theories, but when it comes to hard and fast facts, you take the position that good enough is never good enough and we can't know anything because we can't know everything. That is pretty common among people who come to Sound Science with a preconceived conclusion. If the facts don't go their way, they start questioning the testing methodology. They don't accept any evidence until it's absolutely proven to them, and it's never quite good enough to be absolutely proven to their personal standards.

I look at things differently. I follow the prevailing winds and let the evidence set the direction. And once I find something out, I try to put that into perspective to see if it really matters. Ultimately, that way of thinking is much more practical for the purposes of improving the sound of an audio system, because you don't have to wait for all your questions to be answered before seeing improvement. And the evolution of your questioning can lead to improvements you weren't even looking for.

It's like the difference between Columbus refusing to sail to India until he knows all the logistics of how to get to the destination, and Columbus setting sail due East and discovering America as he goes.

The evidence always needs to be interpreted in the context of how it was gathered, which then raises the question of how far the evidence can be generalized beyond that context. From that standpoint, I see significant problems with the currently available evidence. I'm not looking for "proof" nor do I think that's attainable, but I think we have a ways to go with doing more and better research and testing before we can reach conclusions with confidence.

 

[Joe Bloggs]

Wellll... yes you do... i f you want your conclusions to be generally useful.

They conducted a very nice test of three different amplifiers using a single sample of a single model of headphone.
However, we all know that different headphones make it easier or more difficult for us to hear certain types of information, and emphasize different things.
Therefore, from that limited test, we really cannot reach conclusions about "all headphones" or "all headphone amplifiers".
All we can conclude is that their results are probably correct with the single pair of headphones they used and the particular amplifiers they used.
(And, with luck, they won't change much with production units of those models from next month.)

One serious limitation is that they used ONLY high impedance headphones.
This means that they would have totally missed issues related to the output impedance of those headphone amplifiers.
(If the amplifiers' output impedance is high, then they will interact with, and sound different with, different headphones - especially low impedance models.)
I would have tried it with at least a few different models of different impedance ranges.

As it sits, you can assume that their results are somewhat valid if you plan to use high impedance headphones, but may not even be close if you use low impedance headphones.
This is a perfect example of how, when you know the details of the test, you can consider both what it shows, and what it fails to take into account.

We don't need a generalized conclusion that no two DAC/amps sound different, we need ONE specific example from you that the pair you found to be different, can actually be perceived to be different.

Find those two DACs and the particular speakers you heard the difference on, conduct your test on that. Don't tell me that a man of your resources can't put together ONE system that can reveal these purported differences between DACs in a blind test and tell me with a straight face that somebody needs to start a Kickstarter campaign based on your theories.

To imply that studies need to be conducted to prove the null hypothesis for generally all types of combos on the market is obviously shifting the burden of proof where it doesn't belong.