is this really a problem with blind tests?

[spruce music]

  I made my question very clear. I didn't ask: 
 
"Is there some circumstance in which you can tell from A-B if the difference is inaudible?"
 
I asked:
 
"Given A, B, and A-B, can you tell me if the difference is audible?" and as a bonus question "Can you do it without listening tests?"
 
You probably think I'm being pedantic. But this question is at the heart of our knowledge about audio. And a few special cases won't help.
 
I didn't ask "Are there special cases in which we can make a conclusion?"
 
I'm talking about the general case.
 
So, I'm not asking for a scientific paper, but give me a sense of how you would go about answering that question. How about in the case that we aren't running listening tests.

How can you miss it?  If A-B is 80 db below either A or B then it is inaudible without a listening test.
 
So for instance, concerning interconnects being audible or not, I ran signals thru a DAC, recorded them with an ADC.  I used various expensive and cheap cables.  I subtracted the results from each other and heard nothing.  Difference signal levels were basically in the noise floor a bit less than -100 db.  Without listening I could say they sound the same. Listening to and hearing nothing in the A-B signal confirmed it.  But listening was not a necessary step. All that was left after subtracting $800 cable A from $2 cable B was a low level noise signal so far down as to be inaudible.  In this case the remainder was -100db. 

So there are no differences loud enough to be heard by themselves.  There certainly are not differences heard while the music itself is audible as it will be fully masked. 
 
If I saw residuals in the subtracted signals at the -40 db level I would know these might well be audibly different.  If you get higher than that I know the two will be audibly recognizable as different.

 

[spruce music]

Now for the rest of the story. Just speaking null tests.
 
I have seen two pieces of gear that nulled to just over 40 db.  The difference: one was flat to 2 hz and the other featured a gentle (yes rather gentle) roll off below 50 hz.  These two are very difficult to distinguish.  And of course if your speakers stop at 55 or 50 hz they will both sound the same.  
 
Similarly I have a bit of kit that is flat to 15 khz and down -10db at 20 khz.  The residuals are not all that low vs a wideband piece of gear.  However, with music even people with extended hearing don't find it easy to hear. With test tones it is obvious.  For older people whose hearing dies above 15 khz the two are fully identical.
 
So the further you go, the less I see what it is you are trying to find out.  Hopefully this fleshes out something about null tests.
 
If they are -80db, then no frequency response, nor distortion level, nor anything else is enough different for someone to hear.

 

[Joe Bloggs]

 


That is all I have claimed too.  If you give me A or B, and then give me A-B I can listen to the former at a normal volume, and then listen to A-B.  If I hear silence, I know these two will sound identical.  In fact, they will sound identical at some point where you still can barely hear A-B at normal volume.  If you hear nothing though you have a 10-20 db margin of error and need not worry about it.

Now in most systems at normal volume levels the point where this happens is maybe -80 db.  So if you tell me A-B relative to either of the others is -80 db, then I don't even have to do the listening to A-B or anything else. Somewhere around -60 or -70db is where prior work shows we no longer hear a difference between two sources.  But you will hear those levels in an A-B signal.  The extra margin makes it a surety.

There are some simple ways you can listen to this yourself if you just need to experience it.

I made my question very clear. I didn't ask: 

"Is there some circumstance in which you can tell from A-B if the difference is inaudible?"

I asked:

"Given A, B, and A-B, can you tell me if the difference is audible?" and as a bonus question "Can you do it without listening tests?"

You probably think I'm being pedantic. But this question is at the heart of our knowledge about audio. And a few special cases won't help.

I didn't ask "Are there special cases in which we can make a conclusion?"

I'm talking about the general case.

So, I'm not asking for a scientific paper, but give me a sense of how you would go about answering that question. How about in the case that we aren't running listening tests.

If such an all knowing predictor of audibility existed, nobody would need any listening tests anymore. Never mind that hearing thresholds are different per person.

I don't know why you would consider such a ridiculous requirement to be central to our understanding of audio.

I mean sure, we understand that some distortions sound more offending than others, but if the distortion itself has been beaten down to inaudible levels (which as I point out occurs in more cases than you think) don't you consider that a significant achievement? And if I point out to you that the difference between an original performance and any played back recording is still grossly audible with any current technology (never mind what a combination of commercial studio mastering decisions and typical audiophile listening setups produce), won't you be interested in knowing what are causing those differences and how best to combat them?

 

[johncarm]

If such an all knowing predictor of audibility existed, nobody would need any listening tests anymore. Never mind that hearing thresholds are different per person.

I don't know why you would consider such a ridiculous requirement to be central to our understanding of audio.

I mean sure, we understand that some distortions sound more offending than others, but if the distortion itself has been beaten down to inaudible levels (which as I point out occurs in more cases than you think) don't you consider that a significant achievement? And if I point out to you that the difference between an original performance and any played back recording is still grossly audible with any current technology (never mind what a combination of commercial studio mastering decisions and typical audiophile listening setups produce), won't you be interested in knowing what are causing those differences and how best to combat them?

 
 Investigating that question is at the heart of our knowledge about audio. Of course there is no complete result.
 
I directed that question at Spruce because he has given me nothing but special cases (until his most recent post). He doesn't seem to realize what I'm asking.
 
You have given me more specifics, such as the use of masking theory and the results of the search for inverse masking. I'll see if I can get the textbooks.
 
You claim that distortion has been beaten down to audible levels. I am investigating the evidence for that. I am not impressed by the argument that an inaudible null signal means the difference is inaudible. Mostly it's the listening test protocols I'm skeptical about. How do you know the null is inaudible? How have you verified the difference is inaudible?
 
Also it would be nice to have some theory about the operation of the ear that predicts that. Can you describe any feature of the ear/brain that would predict that result? Is it only your "intuition" and the "absence of evidence"? That's not very satisfying if you don't have any theory that goes along with that result.
 
Regarding listening tests....
 
Let's say we are investigating whether device A and B sound different. We know that a wide variety of signals may be played through A and B. We know that listening can take place in a wide variety of contexts... and probably the greatest variable of all is the question of what the listener is focusing on (or trained to listen for).
 
So if we play just a few signals through A and B, and we do that in only one context, and we don't train the listeners, we wouldn't be very satisfied with the resulting knowledge. Right?
 
In other words, a meaningful answer has to apply to a broad enough set of contexts to be useful.
 
I will look in the textbooks to see what listening protocols were used in developing masking theory. I will be checking to see if the theory really has any meaningful generalization outside the context of the original listening tests.
 
If you would like to comment now, I'm interested. If you want to comment on the question: "How do we know that masking theory can be generalized outside the original listening tests?" I would be interested. 

 

[johncarm]

 
How can you miss it?  If A-B is 80 db below either A or B then it is inaudible without a listening test.
 
 

 
How can you miss that I was asking about the general case?
 

 
 
So for instance, concerning interconnects being audible or not, I ran signals thru a DAC, recorded them with an ADC.  I used various expensive and cheap cables.  I subtracted the results from each other and heard nothing.  Difference signal levels were basically in the noise floor a bit less than -100 db.  Without listening I could say they sound the same. Listening to and hearing nothing in the A-B signal confirmed it. 
 
 

 
What was your test protocol for the result that you heard nothing in the A-B?
 

 
If I saw residuals in the subtracted signals at the -40 db level I would know these might well be audibly different.  If you get higher than that I know the two will be audibly recognizable as different.
 

 
This is a little better but still an extremely non-specific answer. I asked how you would go about answering "yes" or "no". You expressed extreme  confidence that you could answer that from the signals without conducting listening tests. Now you are saying the answer could be "maybe". How about the -41 dB level? How about the -39 dB level? Give me an answer that would satisfy a scientist... or simply state what areas of knowledge are incomplete.

 

[johncarm]

  Now for the rest of the story. Just speaking null tests.
 
I have seen two pieces of gear that nulled to just over 40 db.  The difference: one was flat to 2 hz and the other featured a gentle (yes rather gentle) roll off below 50 hz.  These two are very difficult to distinguish.  And of course if your speakers stop at 55 or 50 hz they will both sound the same.  
 
Similarly I have a bit of kit that is flat to 15 khz and down -10db at 20 khz.  The residuals are not all that low vs a wideband piece of gear.  However, with music even people with extended hearing don't find it easy to hear. With test tones it is obvious.  For older people whose hearing dies above 15 khz the two are fully identical.
 
So the further you go, the less I see what it is you are trying to find out.  Hopefully this fleshes out something about null tests.
 
If they are -80db, then no frequency response, nor distortion level, nor anything else is enough different for someone to hear.

 
But these are all special cases. Give me some theory that would predict these results. That's what a scientist would want. Another thing a scientist would do is have a general sense of what areas of knowledge are uncertain.

 

[johncarm]

Here is a general comment about listening tests.
 
If someone said, "I have searched, and I claim there is a low likelihood of any life existing in Arizona," to be confident in this result we would want to know two things. (1) Did you search pretty much everywhere? (2) Are you sure your method of detecting life wouldn't miss any?
 
Now, let's say that we don't need to be 100% sure of this result. But we certainly need to have confidence in the truth of assertions (1) and (2).
 
So if someone says, "I did a listening test, and devices (or signals) A and B are indistinguishable," we would want to be confident that  (1) their test result could generalize to a large set of listening contexts, and (2) the listener was skilled at hearing the difference under investigation.
 
Are we on the same page so far?

 

[gregorio]

  Aren't you describing the difference signal again as if it were something that actually existed? If you say "the difference signal is audible" that implies there is some physical signal that equals A-B, when no such signal exists.

 
If there is no signal after a null test, then there is no difference between A and B, in fact there is no B, only A. If there is a difference then well have a "difference signal" which completely defines the difference. If the difference is audible it is only because that difference signal (or some part of it) is audible. If you've read and understood what a null test is, I can't see how you fail to understand this!
 

  Reading some of the links provided in this thread, I see only experiments asking the question "Can A be distinguished from A+B?" where B is usually a low-level signal. But that's a different question!

 
No, it's not! In a null test, the question we are asking is: What is the result of A + (-B). If the result is zero, then B must precisely equal A and there is no difference. If the result is not zero then the result precisely defines the difference. Then the question is, is this non-zero result audible/potentially audible? In the question the way you've phrased it, B is the difference signal. If you do not add B to A (or if B = 0) then there is no difference and you are trying to distinguish A from A! If you are talking about audible rather than perceivable, how is this question different from what you're asking?
 

 
Let's use the word "audible" for now to mean you can become consciously aware of a signal or difference between two signals in a reliable way (i.e. mirrors the reality of the signal).
 
We can ask these questions about a sound X: "Can the microphone/ear pick it up?" (Does it have sufficient resolution, bandwidth, noise floor, etc.) The answer must be "yes" for X to be audible.

 
No, using your definition of "audible" the answer can be "no" because you can become aware of a difference between two signals in a reliable way even if there is no difference!
 

  "What is happening in the lower brain?" Three things can go wrong in the lower brain: insufficient resolution, filtering, and distortion.
 
(1) Insufficient resolution would be the signal getting swamped by the noise, or fading due to memory inadequacies, that kind of thing.
 
(2) Filtering is the idea that not all the raw information gets through to consciousness. We are never aware of every bit of information, but rather experience a condensed version of reality.
 
(3) "Optical illusions" in the visual domain are an example of distortion, such as the way we see a curved line when it actually straight. I don't know much about "audio illusions" but I'm sure there are some.

 
1. Agreed, although I would say in the case of insufficient resolution we don't necessarily have to go as far as the brain, if the resolution is insufficient to trigger a response from the microphone/ear. And obviously, in the case of zero resolution we don't even have to go as far as the microphone/ear.
 
2. Filtering implies only the remove of information. While the brain does present a condensed version of reality, it not only removes information but also adds information. We could phrase this as: the brain initially filters so much information that gaps are left which the brain then fills with it's own invention.
 
3. Audio perception is rife with audio illusions. In fact, the vast majority of commercial audio content almost continuous relies on audio illusion. The fact that audio illusions exist is evidence of the statement in point #2.
 

  Any of these things could potentially affect audibility. Distortion may not prevent a person from being aware that something is there, but might be a form of misdirection instead, causing a person to fail a listening test because they have wrong expectations.

 
Absolutely!! There is a large list of cognitive biases which affect the perception/audibility of sound and expectation bias is certainly a very significant one. That's the whole point of a double blind test, it's the most effective way of eliminating biases!
 

  In any area of science, a test can give a null result due to Reason 2.

 
That obviously depends on the test. An audio null test can give a null result which never has anything to do with reason 2.
 

  What is maddening is there's no reason to believe Reasons 1 and 2 can't be investigated and understood. It's not like these are imaginary phenomena.

 
Huh? In many cases it's EXACTLY like these are imaginary phenomena!! Did you watch the video I linked to? What, apart from an "imaginary phenomena", accounts for the difference?
 
G

 

[gregorio]

  How about the -41 dB level? How about the -39 dB level? Give me an answer that would satisfy a scientist...

 
The answer is, it depends! We've talked about -100dB and now you're talking about -40 and moaning about the fact the answer is maybe. OK, let's go to an extreme, what about -6dB would that be audible? The answer is still only maybe, it depends. My evidence? Anyone who has ever listened to a SACD and not complained that they couldn't hear any music above the noise!
 
G

 

[Joe Bloggs]

 
How about the -41 dB level? How about the -39 dB level? Give me an answer that would satisfy a scientist...

The answer is, it depends! We've talked about -100dB and now you're talking about -40 and moaning about the fact the answer is maybe. OK, let's go to an extreme, what about -6dB would that be audible? The answer is still only maybe, it depends. My evidence? Anyone who has ever listened to a SACD and not complained that they couldn't hear any music above the noise!

G

To clarify gregorio's last remark: raw decoded SACD data is composed of noise almost as loud as the music itself, but all of that noise is ultrasonic.

 

[johncarm]

   
If there is no signal after a null test, then there is no difference between A and B, in fact there is no B, only A. If there is a difference then well have a "difference signal" which completely defines the difference. If the difference is audible it is only because that difference signal (or some part of it) is audible. If you've read and understood what a null test is, I can't see how you fail to understand this!
 
 
No, it's not! In a null test, the question we are asking is: What is the result of A + (-B). If the result is zero, then B must precisely equal A and there is no difference. If the result is not zero then the result precisely defines the difference. Then the question is, is this non-zero result audible/potentially audible? In the question the way you've phrased it, B is the difference signal. If you do not add B to A (or if B = 0) then there is no difference and you are trying to distinguish A from A! If you are talking about audible rather than perceivable, how is this question different from what you're asking?
 
 
No, using your definition of "audible" the answer can be "no" because you can become aware of a difference between two signals in a reliable way even if there is no difference!
 
 
1. Agreed, although I would say in the case of insufficient resolution we don't necessarily have to go as far as the brain, if the resolution is insufficient to trigger a response from the microphone/ear. And obviously, in the case of zero resolution we don't even have to go as far as the microphone/ear.
 
2. Filtering implies only the remove of information. While the brain does present a condensed version of reality, it not only removes information but also adds information. We could phrase this as: the brain initially filters so much information that gaps are left which the brain then fills with it's own invention.
 
3. Audio perception is rife with audio illusions. In fact, the vast majority of commercial audio content almost continuous relies on audio illusion. The fact that audio illusions exist is evidence of the statement in point #2.
 
 
Absolutely!! There is a large list of cognitive biases which affect the perception/audibility of sound and expectation bias is certainly a very significant one. That's the whole point of a double blind test, it's the most effective way of eliminating biases!
 
 
That obviously depends on the test. An audio null test can give a null result which never has anything to do with reason 2.
 
 
Huh? In many cases it's EXACTLY like these are imaginary phenomena!! Did you watch the video I linked to? What, apart from an "imaginary phenomena", accounts for the difference?
 
G

 
The central point here is to investigate the behavior of the brain and ear during listening tests. If people sometimes imagine a difference when there is none, then we would like to investigate how that happens and have some kind of model of the brain that would allow some degree of prediction. You are getting hung up on the meaning of "imaginary" and "imagination." The fact of imagination is not imaginary. Therefore it is something that can be investigated.
 
What is your answer to this question: can cognitive biases cause a person to hear no difference when there really is an audible difference?

 

[johncarm]

   
The answer is, it depends! We've talked about -100dB and now you're talking about -40 and moaning about the fact the answer is maybe. OK, let's go to an extreme, what about -6dB would that be audible? The answer is still only maybe, it depends. My evidence? Anyone who has ever listened to a SACD and not complained that they couldn't hear any music above the noise!
 
G
 

 
I haven't "talked about -100 dB". I asked a question: given two signals A and B, how would you go about predicting whether the difference is audible, and in particular could you do so without running listening tests. both you and Spruce expressed supreme confidence that you could answer the question "are A and B audibly different" without listening tests. In fact at various times the idea was floated that it could be answered from a single number: the relative level of (A-B) and B. 
 
I think it's time to admit that claim is false.

 

[Joe Bloggs]

 

The answer is, it depends! We've talked about -100dB and now you're talking about -40 and moaning about the fact the answer is maybe. OK, let's go to an extreme, what about -6dB would that be audible? The answer is still only maybe, it depends. My evidence? Anyone who has ever listened to a SACD and not complained that they couldn't hear any music above the noise!

G

I haven't "talked about -100 dB". I asked a question: given two signals A and B, how would you go about predicting whether the difference is audible, and in particular could you do so without running listening tests. both you and Spruce expressed supreme confidence that you could answer the question "are A and B audibly different" without listening tests. In fact at various times the idea was floated that it could be answered from a single number: the relative level of (A-B) and B. 

I think it's time to admit that claim is false.

They would all like to think that if the level of A minus B is low enough relative to B, (such that switching from B at a normal listening level to A minus B at proportionally the same amplification volume, the listener can no more tell that anything is being played at all, even in a completely silent background. This can itself be determined by an ABX test, where A is A minus B and B is complete silence. The listener would be asked to tell whether X is A minus B or complete silence, and succeed in choosing the right answer for a statistically significant number and proportion of trials.) one can assert without even looking at the content of A and B that they would be indistinguishable. I would like to think so too, actually.

 

[castleofargh]

wow. you guys were active.
@johncarm.  you're demanding a all lot of things at a degree of precision that you should very well know can't be demonstrated while testing audibility on subjects. even MRI and other advanced scanning of the brain are questioned when it comes to making correlation between brain activity and what we can tell from them. not long ago there were some concerns about how such evaluations could lead to exaggerated conclusions. so it's very much a work in progress.
 
also you had your idea from the start and instead of looking into how to test the relevance of that idea yourself, you've been going on rampage to question and argue about pretty much anything but your own hypothesis. so of course people are starting to feel like instead of being curious, you're just here to play games. you can feel in in the air, read allusions in a few posts and I also got PMed on the matter. so I'm slowly starting on the path of statistical evidence against your good will.
 
 
about the null test, the 2 things you need to look up are obviously masking and hearing thresholds(and you could probably spend the next year just on those 2 points and still not have certainty like the one your demand from us). we can expect something at -80db below music to go unnoticed because of hearing threshold, ambient noise, and of course a very high chance of masking by the loudest music. but also because we simply don't have unlimited dynamic range when listening. our ear can trigger a protection mechanism that will move our sensitivity down by (IDK about 20db I guess?) and help reach higher total dynamic range, but it cannot be in both states at once, so when we can perceive the loudest sounds right before hurting, we can't notice the quiet sounds we would be able to detect in a quiet room without the loud noises. so the listening level will affect what we can hear(and counter to intuition, we tend to notice things better a little before the protection mechanism kicks in, so that's usually in the 60/70db loud.
and we can't tell if something at -40db will or won't be heard under music because while of course we can hear something 40db below music at normal listening level, it may or may not be masked by the louder music content.
 
 
 I suggest you spend some time doing a few blind tests(foobar's abx plugin is one of the easiest way to start), and a few null tests(audio diffmaker). and think about ways to test your idea about long samples vs short samples. maybe the results would convince you of what we apparently cannot.

 

[johncarm]

They would all like to think that if the level of A minus B is low enough relative to B, (such that switching from B at a normal listening level to A minus B at proportionally the same amplification volume, the listener can no more tell that anything is being played at all, even in a completely silent background. This can itself be determined by an ABX test, where A is A minus B and B is complete silence. The listener would be asked to tell whether X is A minus B or complete silence, and succeed in choosing the right answer for a statistically significant number and proportion of trials.) one can assert without even looking at the content of A and B that they would be indistinguishable. I would like to think so too, actually.

But that's a special case, not a general answer.
 
I may have given the impression that a "general answer" means "a perfect answer in all circumstances." My frustration is that I was getting absolutely nothing but flip answers or special cases.
 
Look at it this way. If we are investigating a question, we might be unable to find a perfect answer, but we can do a lot of interesting stuff. We can map out some cases where we do have an answer. We can ask what additional tools we might need in other cases. We can create some theories that help us understand what's going on "under the hood" and test those theories. We can ask some questions for further development.
 
That's what I'm looking for. You have given the best answer so far, which is that masking theory applies. I will be reading about masking theory as soon as the textbook arrives.