Testing the claim: "I can hear differences between lossless formats."

[money4me247]

Do you think that if someone hears a difference sighted, they should be able to hear it blind?
Do you think that if someone doesn't hear a difference sighted, they shouldn't be able to hear it blind?
Do you think that a blind test can be run in each case?

I'm aware that these questions sound elementary, so please don't think I'm trying to insult you by asking them. Thanks.

1) Yes, if you hear a difference sighted, you should be able to hear it blinded if in fact the sonic difference is truly there and is not due to expectation bias. The problem is that the phenomenon of expectation bias influencing results has been well demonstrated, so that is why blind testing is valuable. It has been well-demonstrated that sonic perceptions can be influenced by unconscious biases such as price or preconceived notions. There has been studies showing that many perceived differences or preferences in headphones become much more subtle or indistinguishable under blind testing conditions. There have also been funny sham comparisons using the same components but purposely give incorrect information to blinded listeners and their opinions of the sonics become influenced by that misinformation. One such study had an ABx switch between a tube amp and solid state amp, but actually used the same amp in both cases. Audiophiles preferred the tube amp while engineers preferred the solid state amp.
 
2) Yes, if you don't hear a difference sighted, you shouldn't hear a difference blinded.
 
3) A blind test is not necessary in every case. It is only necessary if you are trying to remove your expectation bias or other confounding variables that may be influencing your results. If you don't mind your results being influenced by expectation bias, there is no need to do a blind test. In the cases of claims that appear illogical, a successful blind test can be helpful to remove the listener's bias to make the results more acceptable for others.

 

[Claritas]

3) A blind test is not necessary in every case. It is only necessary if you are trying to remove your expectation bias or other confounding variables that may be influencing your results. If you don't mind your results being influenced by expectation bias, there is no need to do a blind test. In the cases of claims that appear illogical, a successful blind test can be helpful to remove the listener's bias to make the results more acceptable for others.

Thanks for the reply. I was asking if you think that when someone doesn't hear a difference, a blind test can be run to determine that.

 

[money4me247]

There are a number of discrimination testing methodologies that would work.

More importantly, the null hypothesis is "no discernable relationship." Discernable or detectable is the key word--failure to reject the null is evidence that a relationship has not been observed. NOT evidence that a relationship does not exist.

Also, replacing the word relationship with difference is incorrect. Youre erroneously representing "no relationship" as "no difference = same." Sameness is a relationship, and is a separate alternative hypothesis you would need to test.

 

Not sure why people are trying to tell me that i dont know how confidence intervals work. I've said nothing thats specific enough to even be incorrect. 99% confidence intervals exist. But that is not interesting.

Yes, youre right that the burden of proof is on the OP in this specific case. And i agree that he probably wont be able to demonstrate a difference between lossless formats, though some kind of computer error could be causing a difference on his system.

My comments are more broadly directed at the "because science" folks who dont understand how to properly interpret test results and who distort the meaning of failure to reject the null hypothesis. In this specific case as well as generally.

 

For this specific example, the null hypothesis would not be "no discernible relationship" as there is no specific relationship that we are testing. The way to determine whether there is an audible difference would be for OP to accurately identify the format in a series blind test to a percentage that is above random chance. From the way that this test is set-up, if OP fails to successfully accomplish that, it does indeed conclude that OP cannot consistently identify an audible difference, which allows us to reasonable conclude that the two formats are audibly transparent for his hearing. We already know there are "differences" between the two items due to their file formats and amount of space they take up. The point of such tests is to identify if there is any reproducible audible differences. 
 
Your broad comments that failing to reject the null hypothesis of "no difference" does not equate to the proving that both items are the "same" is correct on a theoretical level. Our conclusion is not that those two items are the "same," but rather if there is no audible difference between two things, we can practically say that any sonic differences that may exist are beneath our threshold of hearing, so from a practical standpoint, they are audibly transparent and identical to the human ear.
 
For example, one source may contain frequency responses in the 25khz region while the other source does not. However, since 25khz is beyond the range of frequency responses that the human ear can detect, those two sources are in essence the identical for practical listening purposes. There can be variations in factors such as distortion and noise of gear, but if they are below the threshold which the human ear can detect, those differences do not have any practical meaning.

 

[money4me247]

Thanks for the reply. I was asking if you think that when someone doesn't hear a difference, a blind test can be run to determine that.

 
Yes. It is easy to imagine that someone may have an expectation bias of no difference.
 
The test for something along those lines would have to be a bit more complicated though as it is too easy to cheat and say I hear no difference for everything. You will need to have two components that "sound the same" as well as one component that sounds "different."
 
You will have to establish that the tester can accurately detect the difference between "same" component and "different" component. Then you throw in the third component that sounds the same as the first component. Tester will have the option to pick A-B-C.
 
The results after a large enough sample size should be that the same component A and component C that supposedly sound the same will be consistently either correctly identified or misidentified as the alternative same component. The probability of a correct identification should be within the bounds of random chance. If there was a statistically significant deviation in the correct identifications, you begin to suspect that there may be something else influencing the results worth investigating.
 
Note that these tests are not trying to determine exactly what the difference is. They are just testing if there is an audible difference that is reproducible. If there is a statistically significant difference, it can be due to anything from the claim being correct, differences in volume, or even supernatural powers... further testing would be required to isolate the cause of the difference.
 
However, without first proving that there is a difference or no difference, further testing is irrelevant.

 

[esldude]

  I think I've said this before, but different Dac Chip sets decode the different formats differently. Those minor changes in the DACS output based on format input result in their being a different, are the differances HUGE not really, but they are some what noticeable. So then, does WAV sound better than FLAC? Depends on your Dac, keep in mind software is what dictates hardware [or firmware] so changes in software will result in slight differances in hard ware performance. 
 
or at least that's what I tell my self :/ if that's 100% bull schiit logic your more than welcome to let me know :3 

Actually err.....NO!   DAC chips typically get a SPDIF or PCM or in some DSD or I2S formatted data streams.  You have other items that convert your FLAC, or WAV, or MP3 or whatever into the correct data stream.  The DAC chip itself never knows what the source format was and doesn't act differently from different formats.  So no the DAC outputs don't have changes from format input changes.  For instance a FLAC will have been converted to the same exact bit for bit data stream the DAC sees as the WAV file. The DAC will have no way of knowing the difference in the two and will not output differently.  So the differences were in your imagination while not being in fact real at the DAC.  So does WAV sound better than FLAC?  No they will be the same data stream and sound the same. 
 
As for software changing hardware that isn't the case in the sense you are speaking about it.  So now you know, your logic was faulty as it was based upon false premises. 

 

[drez]

Anyone who doesn't care to burden themselves with proof would be wise to avoid this subforum.

 

[Claritas]

Anyone who doesn't care to burden themselves with proof would be wise to avoid this subforum.

The place would be empty because objectivists are shirking just as much as subjectivists. "You can't prove a negative." "You're shifting the burden of proof." If you believe in testing, test your own beliefs too instead of just taking them for granted.

 

[drez]

The place would be empty because objectivists are shirking just as much as subjectivists. "You can't prove a negative." "You're shifting the burden of proof." If you believe in testing, test your own beliefs too instead of just taking them for granted.

 
I'm not saying anyone should test anything, I personally find the burden of proof too burdensome given that this is a hobby and not a job.  Folks around here though are obsessed with evidence which can be used to support mental constructs of certainty etc. - it's like a fantasy island of science.
 
You are right though it is typically a pretty dead place, mostly because most people tend to avoid objective testing because it doesn't tend to show anything new.  It's like, yep, still same old "all amps sound the same" "jitter is inaudible" "inaudible is inaudible" "high res is isaudible".  But some people seem to find that interesting and prefer to maintain this mode of inquiry.
 
Yes there are some blind tests and claimed measurements out there that claim to show different results, but sound science needs evidence, like really super detailed evidence for them to be happy.  So we still have status quo.
 
I also think there is a certain interest in maintaining this view of the world and mode of inquiry, and maybe even interest in mythbusting etc to cut the high end down to size.
 
This is not for me, but I don't want to ruin the party.

 

[RRod]

The place would be empty because objectivists are shirking just as much as subjectivists. "You can't prove a negative." "You're shifting the burden of proof." If you believe in testing, test your own beliefs too instead of just taking them for granted.

 
Give us a test designed for the task and I'm sure people would take it.  The problem, as others are pointing out, is that ABX is designed for people who desire to show they *can* discern between two tracks.  But it's easy to deliberately fail such a test: just answer A the whole time.  We need a test that avoids this issue for testing if someone cannot discern two tracks.

 

[Makiah S]

Actually err.....NO!   DAC chips typically get a SPDIF or PCM or in some DSD or I2S formatted data streams.  You have other items that convert your FLAC, or WAV, or MP3 or whatever into the correct data stream.  The DAC chip itself never knows what the source format was and doesn't act differently from different formats.  So no the DAC outputs don't have changes from format input changes.  For instance a FLAC will have been converted to the same exact bit for bit data stream the DAC sees as the WAV file. The DAC will have no way of knowing the difference in the two and will not output differently.  So the differences were in your imagination while not being in fact real at the DAC.  So does WAV sound better than FLAC?  No they will be the same data stream and sound the same. 

As for software changing hardware that isn't the case in the sense you are speaking about it.  So now you know, your logic was faulty as it was based upon false premises. 

ok cool I was wrong no big deal I like flac sounds good. nice and small. now you guys have fun

 

[money4me247]

The place would be empty because objectivists are shirking just as much as subjectivists. "You can't prove a negative." "You're shifting the burden of proof." If you believe in testing, test your own beliefs too instead of just taking them for granted.

 
I have found that both people claiming to be subjectivists and people claiming to be objectivitists have a certain set of core beliefs that they like profess. this usually just ends up resulting in unproductive flame wars. there is a time and place for both subjective experiences and objective data.
 
Talking about burden of proof is silly and unhelpful (it doesn't matter who has the burden of proof, the point is to create an experiment that adequately demonstrates the claim). Talking about you cannot prove a negative is also unhelpful (there are definitely ways to frame a test to demonstrate a reasonable conclusion of no difference). Talking about how no difference does not technically equate to sameness is unhelpful (for our practical application of perceivable audible differences, sameness is defined as to the point where the human ear cannot reliable distinguish the difference).
 
All those above debates have absolutely no practical application!!! It is arguing for the sake of arguing. If you are interesting in arguing the relative merits of objectivism as a philosophy vs subjectivism, there should be a new thread created for that purpose.
 
If you are truly interested in objective testing, preconceived notions of the expected results is only helpful in terms of framing a hypothesis. The test will reveal the results. So far I see a lot of bickering and absolutely no testing!!!! I have already laid out a statistical test per specifications that can be used to analyze the results of a blind test.
 
...If there is no blind test or any attempt to do a blind test, this thread is basically pointless.....

 

[esldude]

   
Give us a test designed for the task and I'm sure people would take it.  The problem, as others are pointing out, is that ABX is designed for people who desire to show they *can* discern between two tracks.  But it's easy to deliberately fail such a test: just answer A the whole time.  We need a test that avoids this issue for testing if someone cannot discern two tracks.

Oh well done tests will have features for that.  Throw in a clearly audibly different track for comparison among the others tested.  If it too gets non-significant results then something about the test was faulty.  So mix in an obviously different track and someone picking A the whole time will be found out.  In non-formal hobby level tests that is a feature that should be included and usually isn't. 

 

[limpidglitch]

 
Oh well done tests will have features for that.  Throw in a clearly audibly different track for comparison among the others tested.  If it too gets non-significant results then something about the test was faulty.  So mix in an obviously different track and someone picking A the whole time will be found out.  In non-formal hobby level tests that is a feature that should be included and usually isn't. 

If that one track is obviously different, it would be trivial to pick that out and guess on the rest.

 

[esldude]

 
If that one track is obviously different, it would be trivial to pick that out and guess on the rest.

Yes, nevertheless, such "controls" are included in serious research testing.  You can find JJ Johnston discussing the need for these if you look around some.  And there are degrees of obvious.  For instance a 1 db level difference is not jump out at you obvious on a conscious level, but will be chosen near 100% of the time in an ABX type of testing. 

 

[limpidglitch]

 
Yes, nevertheless, such "controls" are included in serious research testing.  You can find JJ Johnston discussing the need for these if you look around some.  And there are degrees of obvious.  For instance a 1 db level difference is not jump out at you obvious on a conscious level, but will be chosen near 100% of the time in an ABX type of testing. 

Does the method have a name?
It reminds me a bit about the randomized response method, where you set up the questionnaire so that you can eek out the result using Bayes' theorem.