Rob watts DAC design talk
Aug 21, 2018 at 11:31 AM Post #166 of 468
I’ve seen that post. The testing was generally not designed, conducted, and reported to scientific standards, so it has limited value for me.

There are dozens of controlled tests there. How can you generalize about them like that? I think your bias is showing. You know, psychological issues can affect one's thinking and perception!
 
Aug 21, 2018 at 11:58 AM Post #167 of 468
There are dozens of controlled tests there. How can you generalize about them like that? I think your bias is showing. You know, psychological issues can affect one's thinking and perception!

I'm looking for designs and reports of testing of the caliber I see in scientific journals. The tests cited in that post don't meet that standard, and instead reflect the typically amateur/hobbyist credentials of the people doing the testing. There may be some exceptions, but that's my general impression of the tests.
 
Aug 21, 2018 at 3:08 PM Post #168 of 468
I'm looking for designs and reports of testing of the caliber I see in scientific journals. The tests cited in that post don't meet that standard, and instead reflect the typically amateur/hobbyist credentials of the people doing the testing. There may be some exceptions, but that's my general impression of the tests.

I would guess that part of the problem is that nothing new has been introduced to contradict basic principles that have been established long ago. Is there anything specific that we can discuss? Maybe we can go back to the roots and dig up something in the past that might be used as a reference to a standard or benchmark? I'm curious too, but nobody is going to spend the resources to research something that has no benefit to anyone and nothing tangible has been shown that would indicate further testing is even required.

When it comes to digital audio, we didn't have to wait for a solar eclipse to further strengthen a theory such as Einstein's, we have evidence all around us in the very real products and software that exists today that we got the math right.
 
Aug 21, 2018 at 4:44 PM Post #169 of 468
I would guess that part of the problem is that nothing new has been introduced to contradict basic principles that have been established long ago. Is there anything specific that we can discuss? Maybe we can go back to the roots and dig up something in the past that might be used as a reference to a standard or benchmark? I'm curious too, but nobody is going to spend the resources to research something that has no benefit to anyone and nothing tangible has been shown that would indicate further testing is even required.

When it comes to digital audio, we didn't have to wait for a solar eclipse to further strengthen a theory such as Einstein's, we have evidence all around us in the very real products and software that exists today that we got the math right.

I would be particularly interested in studies which delve into details of how fallibility and variability of perception and memory can affect results of controlled blind tests. If you know of any, please point us to them.
 
Aug 21, 2018 at 4:48 PM Post #170 of 468
I'm looking for designs and reports of testing of the caliber I see in scientific journals. The tests cited in that post don't meet that standard, and instead reflect the typically amateur/hobbyist credentials of the people doing the testing. There may be some exceptions, but that's my general impression of the tests.

Well if you're looking for peer reviewed tests on audio reproduction, you should join the AES. They have a ton of reports available online. But they're focused on audio fidelity, not fallibility of human perception. For that, you'll need to find a psychological testing group. I'm not up on that hoodoo.
 
Aug 21, 2018 at 6:22 PM Post #171 of 468
Well if you're looking for peer reviewed tests on audio reproduction, you should join the AES. They have a ton of reports available online. But they're focused on audio fidelity, not fallibility of human perception. For that, you'll need to find a psychological testing group. I'm not up on that hoodoo.

You're asking someone to listen to A, B, and X, and based on their perception and memory of what they heard, make a judgement of whether X is A or B. The psychological aspects are inherent to the testing protocol and interpretation of the test, they can't simply be ignored based on the evidently erroneous assumption that people's reports of what they're hearing are an objective and stable measure. It's ironic that so many 'objectivists' are so quick to point out the problems with perception and memory related to sighted listening comparisons (which go far beyond just problems with expectation bias), yet can't see these problems in controlled blind testing. Don't be mislead by phrases like 'controlled blind testing' - it's not the same situation as randomized double-blind trials in medicine where what's being measured is objective (blood pressure, cholesterol level, tumor sizes, survival durations, etc.).
 
Aug 21, 2018 at 7:43 PM Post #172 of 468
I don't see the logic of doing controlled tests to determine the lack of effectiveness of controlled tests, but I know you've got a bug in your bonnet to prove that. I think you'd be more likely to see discussion of that in woo woo circles. Audiophiles are out to prove that their impressions aren't subject to bias, the only error is in the testing methods. It's a dead end, but go crazy and have fun with it.

I don't care about my blood pressure and brain waves. They can take care of themselves. I just want high fidelity sound reproduction. My physiognomy can fend for itself.
 
Last edited:
Aug 21, 2018 at 9:53 PM Post #173 of 468
You're asking someone to listen to A, B, and X, and based on their perception and memory of what they heard, make a judgement of whether X is A or B. The psychological aspects are inherent to the testing protocol and interpretation of the test, they can't simply be ignored based on the evidently erroneous assumption that people's reports of what they're hearing are an objective and stable measure. It's ironic that so many 'objectivists' are so quick to point out the problems with perception and memory related to sighted listening comparisons (which go far beyond just problems with expectation bias), yet can't see these problems in controlled blind testing. Don't be mislead by phrases like 'controlled blind testing' - it's not the same situation as randomized double-blind trials in medicine where what's being measured is objective (blood pressure, cholesterol level, tumor sizes, survival durations, etc.).
Why does it have to be so complicated? What about a blind test used to determine what is preferred? Consider something as simple as deciding whether test sample A sounds better than test sample B? This test could take as much time as required and conform to whatever settings make the subjects feel most comfortable.

What exactly is being attributed to a criterion for blind testing that somehow manifests itself into a situation that wasn't already present in a similar, yet sighted evaluation? One first has to overcome the extremely obvious and solidly entrenched understanding of biases that are heavily prevalent with sighted testing when determining preferences before something else can be introduced and attributed to the decision-making process. That is a rather tall order.

If some alternative technique can be established that could reliably eliminate the myriad of biases potentially introduced in any sighted evaluation, perhaps a reasonable attempt could be made to compare a blinded experience under similar circumstances. Currently, blinded experimentation is the preferred procedure to remove biases. It is largely because of the logic that supports this notion that blind and double-blind testing are crucial to advancements through the modern scientific method.

Until proven otherwise, blind testing is still the clearest solution to remove biases, and the most accurate at determining if any difference of significance can be perceived.
 
Aug 21, 2018 at 10:35 PM Post #174 of 468
Why does it have to be so complicated? What about a blind test used to determine what is preferred? Consider something as simple as deciding whether test sample A sounds better than test sample B? This test could take as much time as required and conform to whatever settings make the subjects feel most comfortable.

What exactly is being attributed to a criterion for blind testing that somehow manifests itself into a situation that wasn't already present in a similar, yet sighted evaluation? One first has to overcome the extremely obvious and solidly entrenched understanding of biases that are heavily prevalent with sighted testing when determining preferences before something else can be introduced and attributed to the decision-making process. That is a rather tall order.

If some alternative technique can be established that could reliably eliminate the myriad of biases potentially introduced in any sighted evaluation, perhaps a reasonable attempt could be made to compare a blinded experience under similar circumstances. Currently, blinded experimentation is the preferred procedure to remove biases. It is largely because of the logic that supports this notion that blind and double-blind testing are crucial to advancements through the modern scientific method.

Until proven otherwise, blind testing is still the clearest solution to remove biases, and the most accurate at determining if any difference of significance can be perceived.

I agree that blinded is better than sighted, and I also agree that a blinded preference test has value - I plan to do some of those myself. What I'm questioning is how reliably inferences can be made from results of controlled blinded tests to normal listening. I'm not saying that such inferences can't be made, just that I'm not aware of good research to explore that possible connection. I think it's plausible that null results in blind controlled tests (e.g., no consistent preference between A vs B) do indicate that no significant difference or preference would be experienced in normal listening, but it's also plausible that test conditions result in the perception and memory operating in ways that are different from normal listening so that such inferences can't reliably be made. If these questions have been explored, I'd love to read about that research.
 
Aug 21, 2018 at 10:56 PM Post #175 of 468
I don't need to test for my delusions. I just need to do measurements to determine if the degree of fidelity to the original signal is high, and test to see if I can't hear the difference in real world music listening conditions. My blind tests involve real world listening, not lab coats and clipboards. Too many people in Sound Science think it's their job to split atoms and do "talk therapy" to make us realize our Freudian tendencies. All we need to do is make music sound good. I keep my eye on that prize. Once I achieved that, I turned my focus to experiencing and understanding music. I am able to listen to music with my brain turned on and simultaneously appreciate music and analyze it. That understanding is what leads to true connection with music. That requires equipment capable of high fidelity reproduction, good music and an active mind. Maybe not that many people have all three at the same time.
 
Last edited:
Aug 21, 2018 at 11:10 PM Post #176 of 468
I agree that blinded is better than sighted, and I also agree that a blinded preference test has value - I plan to do some of those myself. What I'm questioning is how reliably inferences can be made from results of controlled blinded tests to normal listening. I'm not saying that such inferences can't be made, just that I'm not aware of good research to explore that possible connection. I think it's plausible that null results in blind controlled tests (e.g., no consistent preference between A vs B) do indicate that no significant difference or preference would be experienced in normal listening, but it's also plausible that test conditions result in the perception and memory operating in ways that are different from normal listening so that such inferences can't reliably be made. If these questions have been explored, I'd love to read about that research.
I have a background in experimental psychology and as you are aware all attempts at measuring things have inherent error that can never be completely eliminated. You are essentially setting a bar that can't be attained if you are asking for infallible testing. And randomized assignment is no guarantee that groups will be equivalent, there still exists the chance that during random assignment the groups will become biased simply by chance. And yes, while there are differences for certain in the fact that an experimental situation can differ from a natural situation, you can't simply assume that the possible differences are significant enough to invalidate any findings. That is why we have statistical analysis that control for measurement error. Ultimately the more sets of data that converge around a question, the greater the sample of the population becomes and the greater confidence we can have in the results. Is it perfect? No, nothing ever is, but to suggest that because some measurement error exists that the results are unreliable is a gross over-simplification.

When it comes to memory issues and subjective bias, the body of scientific evidence is enormous and if people like to hear it or not, audio memory is not very robust at all. If you think that our brain can listen to a song on a particular system, memorize the tiny sonic nuances that differentiate it from another similar, but not identical system, hold those impressions in a robust, easy to access memory location, retrieve it hours, days, months later and allow us to compare very minute differences with any degree of accuracy I'm not sure what to say. I believe in the subjective joy of this wonderful hobby, it is crucial and important to me, but for people to disregard science simply because it doesn't confirm what they want confirmed is difficult to reconcile. How about taste, that is another subjectively experienced sense in some respects. Do you think that our brain can remember with incredible precision taste experiences from the past? Could you reliably pick out the differences between two garlic samples of the same species, but grown in different locations? Not likely, so why is our hearing brain supposed to be capable of these extraordinary feats of memory magic? Why on earth would evolution give us such a capacity, how would it help the species survive? Our hearing brain is remarkable, but it is also extremely limited in many ways and people just need to accept that.

Many of the claims made here at head-fi are simply assumptions people make because they have heard others tell them that they can do it so transitively people just take it for granted that they have these abilities and there is nothing you can do to convince them otherwise. Rather than ask for perfect scientific evidence it might be fruitful to ask why you NEVER EVER read of these professional reviewers doing blind listening tests, not ever. And they have been challenged to do so. Ultimately they operate from a position of scientific fallacy in so much as they propose theories that can't be falsified (tested) and therefore fail one of the most important steps used to evaluate if a theory is legitimate. I'm not talking about you here, but when somebody says it is true that I can hear these minute differences because I know I can, well, that is simply foolishness in at least the sense of scientific evidence. I can't test that claim therefore it can't be the basis of a valid theory as it can't be disproven. Not because it is true and thus can't be disproven, it can't be refuted because the interior experience of a person is currently inscrutable and we need to find proxy measures that can be exposed to observation such as well controlled, multiple subject, multiple trial blind listening tests. They aren't perfect and there is error there, but at least it is testable and possible to refine. I can't test the claims of somebody who says trust me, I'm right, I hear that difference.
 
Aug 21, 2018 at 11:58 PM Post #177 of 468
I have a background in experimental psychology and as you are aware all attempts at measuring things have inherent error that can never be completely eliminated. You are essentially setting a bar that can't be attained if you are asking for infallible testing. And randomized assignment is no guarantee that groups will be equivalent, there still exists the chance that during random assignment the groups will become biased simply by chance. And yes, while there are differences for certain in the fact that an experimental situation can differ from a natural situation, you can't simply assume that the possible differences are significant enough to invalidate any findings. That is why we have statistical analysis that control for measurement error. Ultimately the more sets of data that converge around a question, the greater the sample of the population becomes and the greater confidence we can have in the results. Is it perfect? No, nothing ever is, but to suggest that because some measurement error exists that the results are unreliable is a gross over-simplification.

When it comes to memory issues and subjective bias, the body of scientific evidence is enormous and if people like to hear it or not, audio memory is not very robust at all. If you think that our brain can listen to a song on a particular system, memorize the tiny sonic nuances that differentiate it from another similar, but not identical system, hold those impressions in a robust, easy to access memory location, retrieve it hours, days, months later and allow us to compare very minute differences with any degree of accuracy I'm not sure what to say. I believe in the subjective joy of this wonderful hobby, it is crucial and important to me, but for people to disregard science simply because it doesn't confirm what they want confirmed is difficult to reconcile. How about taste, that is another subjectively experienced sense in some respects. Do you think that our brain can remember with incredible precision taste experiences from the past? Could you reliably pick out the differences between two garlic samples of the same species, but grown in different locations? Not likely, so why is our hearing brain supposed to be capable of these extraordinary feats of memory magic? Why on earth would evolution give us such a capacity, how would it help the species survive? Our hearing brain is remarkable, but it is also extremely limited in many ways and people just need to accept that.

Many of the claims made here at head-fi are simply assumptions people make because they have heard others tell them that they can do it so transitively people just take it for granted that they have these abilities and there is nothing you can do to convince them otherwise. Rather than ask for perfect scientific evidence it might be fruitful to ask why you NEVER EVER read of these professional reviewers doing blind listening tests, not ever. And they have been challenged to do so. Ultimately they operate from a position of scientific fallacy in so much as they propose theories that can't be falsified (tested) and therefore fail one of the most important steps used to evaluate if a theory is legitimate. I'm not talking about you here, but when somebody says it is true that I can hear these minute differences because I know I can, well, that is simply foolishness in at least the sense of scientific evidence. I can't test that claim therefore it can't be the basis of a valid theory as it can't be disproven. Not because it is true and thus can't be disproven, it can't be refuted because the interior experience of a person is currently inscrutable and we need to find proxy measures that can be exposed to observation such as well controlled, multiple subject, multiple trial blind listening tests. They aren't perfect and there is error there, but at least it is testable and possible to refine. I can't test the claims of somebody who says trust me, I'm right, I hear that difference.
I wish I could multi-like this post.
 
Aug 22, 2018 at 3:32 AM Post #178 of 468
I would be particularly interested in studies which delve into details of how fallibility and variability of perception and memory can affect results of controlled blind tests.

Why, what has blind, sighted, ABX or any other testing methodology got to do with this thread? Well before we can even start thinking about the potential flaws/errors in the results of a listening test, we have to be able to construct a test in the first place and in this instance, we can't, not by a long chalk. Such a test is a physical impossibility, even the very finest of speakers (or HPs) are orders of magnitude away from being able to reproduce anything at -200dB, let alone -350dB. So, what has the "fallibility of human perception and memory" (when trying to identify a specific difference) got to do with anything, if we can't even reproduce that difference to start with? We covered this pages ago and yet you're still pushing an entirely irrelevant argument.

Do you think that a pedestrian being hit head on by a family car travelling at 200,000mph would be killed? If so, why? There are no scientific studies that demonstrate or prove this, because such a test is impossible to start with, we cannot make a family car that travels at 200,000mph. The fastest family car available is about 10,000 times slower than 200,000mph, just as the very best speakers or HPs are about 10,000 times away from being able to reproduce a difference at -200dB. Would you make the argument that crash tests are fallible and do not provide incontrovertible proof that in every case a pedestrian definitely will or will not survive? Wouldn't you consider such an argument to not only be utterly ridiculous but irrelevant anyway, because cars cannot travel at 200,000mph?

G
 
Aug 22, 2018 at 6:59 AM Post #179 of 468
I have a background in experimental psychology and as you are aware all attempts at measuring things have inherent error that can never be completely eliminated. You are essentially setting a bar that can't be attained if you are asking for infallible testing. And randomized assignment is no guarantee that groups will be equivalent, there still exists the chance that during random assignment the groups will become biased simply by chance. And yes, while there are differences for certain in the fact that an experimental situation can differ from a natural situation, you can't simply assume that the possible differences are significant enough to invalidate any findings. That is why we have statistical analysis that control for measurement error. Ultimately the more sets of data that converge around a question, the greater the sample of the population becomes and the greater confidence we can have in the results. Is it perfect? No, nothing ever is, but to suggest that because some measurement error exists that the results are unreliable is a gross over-simplification.

When it comes to memory issues and subjective bias, the body of scientific evidence is enormous and if people like to hear it or not, audio memory is not very robust at all. If you think that our brain can listen to a song on a particular system, memorize the tiny sonic nuances that differentiate it from another similar, but not identical system, hold those impressions in a robust, easy to access memory location, retrieve it hours, days, months later and allow us to compare very minute differences with any degree of accuracy I'm not sure what to say. I believe in the subjective joy of this wonderful hobby, it is crucial and important to me, but for people to disregard science simply because it doesn't confirm what they want confirmed is difficult to reconcile. How about taste, that is another subjectively experienced sense in some respects. Do you think that our brain can remember with incredible precision taste experiences from the past? Could you reliably pick out the differences between two garlic samples of the same species, but grown in different locations? Not likely, so why is our hearing brain supposed to be capable of these extraordinary feats of memory magic? Why on earth would evolution give us such a capacity, how would it help the species survive? Our hearing brain is remarkable, but it is also extremely limited in many ways and people just need to accept that.

Many of the claims made here at head-fi are simply assumptions people make because they have heard others tell them that they can do it so transitively people just take it for granted that they have these abilities and there is nothing you can do to convince them otherwise. Rather than ask for perfect scientific evidence it might be fruitful to ask why you NEVER EVER read of these professional reviewers doing blind listening tests, not ever. And they have been challenged to do so. Ultimately they operate from a position of scientific fallacy in so much as they propose theories that can't be falsified (tested) and therefore fail one of the most important steps used to evaluate if a theory is legitimate. I'm not talking about you here, but when somebody says it is true that I can hear these minute differences because I know I can, well, that is simply foolishness in at least the sense of scientific evidence. I can't test that claim therefore it can't be the basis of a valid theory as it can't be disproven. Not because it is true and thus can't be disproven, it can't be refuted because the interior experience of a person is currently inscrutable and we need to find proxy measures that can be exposed to observation such as well controlled, multiple subject, multiple trial blind listening tests. They aren't perfect and there is error there, but at least it is testable and possible to refine. I can't test the claims of somebody who says trust me, I'm right, I hear that difference.

Good post, thank you. I agree with the general points you make. My main issue is that it’s not evident to me that a null result in a controlled blind test necessarily indicates that no difference would be perceived in normal listening, since the former involves a kind of conscious attention and processing which is different from normal listening, and could result in differences in subconscious processing. As a simple example of how perception can be affected by conscious attention, in the gorilla experiment, people fail to see the gorilla when they’re focused on counting the passes, whereas if they casually observe the scene I expect that a high percentage would notice the gorilla.
 
Aug 22, 2018 at 8:02 AM Post #180 of 468
When blindtesting dacs last weekend we came across the same ‘problem’ ie how long time do we spend with each unit before we go to the next, what do we listen for, what are we listening for and where exactly do we locate it? etc etc
Fairly quickly all of this became redundant simply because we couldn’t hear ANY differences. We spent hours with both speakers and headphones alternating between factors and nothing came up.
We were expecting to hear minor differences too.
Now in a hobby that is about music and listening to the very same how come we keep coming back to stuff no wellfunctioning human being can hear?

Do some blindtest for yourselves out there. I was pretty damn skeptical when I first began mostly because I fought back the idea that my ears and brain were deceiving me....but I could not overlook the overwhelming scientific proof once I made my first peep into music without my eyes fixed on some shine box.
 

Users who are viewing this thread

Back
Top