Testing audiophile claims and myths

[Sgt. Ear Ache]

Actually, even though the test was rather limited, I thought their results were interesting.....

Out of a bunch of participants, they did in fact find one listener who was reliably able to recognize the difference....
So they DID prove that the assertion that "nobody can reliably hear the difference" is UNTRUE after all.
(And the iTunes sales and marketing department will be very interested in the statistical data that most users could not.)

If we consider the results of that test to be valid, then I guess now we know that at least some people can hear the difference after all, and we can move on.
It should also be noted that they managed to find that listener with only a limited number of tests and a limited number of participants.
(If it were to turn out that 1/500 listeners could hear a difference - that would be a quarter of a million who can in the USA alone.)

lol, oh yes I knew you were going to like that. Yeah, they "found" one listener...who happened to be an sound engineer and knew exactly what he needed to look for...and still found it difficult and says he mostly made random choices and didn't really hear any difference. Your conclusion that what the test shows us is that "some" (really, MAYBE 1) people can (might be able to) hear a difference and that now having proven your point (which in fact is the opposite of what this test really does) we can now move on is utterly hilarious!

 

[Phronesis]

I know from the outset that anything I provide will not meet your high standards of scientific scrutiny of course, but here's one link...

https://cdvsmp3.wordpress.com/cd-vs-itunes-plus-blind-test-results/


now, if you can provide one link to a similar test that has an opposite result?

Thanks for the link, I read through it. It's not a scientific study which was conducted and documented to scientific standards, but not a worthless study IMO. Some problems are:

- We don't know the specifics of the gear people used. What does "excellent" mean? Some people might think a $150 headphone is excellent.

- Assuming that the difference between lossy and lossless is mainly in higher frequencies, we don't know if the lossless files were recorded well to properly capture the high frequencies in the first place. I didn't see any analysis of the files themselves.

- A significant percentage of people taking the test may have treated the test as a casual fun exercise and just guessed most of the time, rather than really trying.

Also, the p-value cutoff of 0.05 is arbitrary. There were some p-values close to 0.05, and one was below 0.05. To me, this is suggestive that there were subtle differences which some people picked up. And it seems that there were one or two people who could notice differences with some consistency.

The comments section under the article is worth reading, and confirms some of the problems I noted.

I think the study argues against the sweeping conclusion that the differences are inaudible for all listeners under all conditions, but supports the conclusion that the vast majority of listeners will find the differences to be very subtle at most.

 

[Sgt. Ear Ache]

It would be VERY interesting to survey all of the customers of a major service that uses lossy compression - like iTunes.
I would be curious to know how many of their customers are actually convinced that the lossy compressed music they're listening to is "indistinguishable from CD quality"...
And how many would say that, even though they personally don't care much either way, they just assume, or are outright convinced, that the CD really is a little bit better...
And, for that matter, it would be interesting to find out how many assume that vinyl is still better than both...
.

why would this be interesting? It's not a revelation that basically un-informed people buy into the notion that "lossy" is worse than "lossless." Really that's exactly what many of us are contending. They are believing what the salespeople tell them...

 

[Sgt. Ear Ache]

Thanks for the link, I read through it. It's not a scientific study which was conducted and documented to scientific standards, but not a worthless study IMO. Some problems are:

- We don't know the specifics of the gear people used. What does "excellent" mean? Some people might think a $150 headphone is excellent.

- Assuming that the difference between lossy and lossless is mainly in higher frequencies, we don't know if the lossless files were recorded well to properly capture the high frequencies in the first place. I didn't see any analysis of the files themselves.

- A significant percentage of people taking the test may have treated the test as a casual fun exercise and just guessed most of the time, rather than really trying.

Also, the p-value cutoff of 0.05 is arbitrary. There were some p-values close to 0.05, and one was below 0.05. To me, this is suggestive that there were subtle differences which some people picked up. And it seems that there were one or two people who could notice differences with some consistency.

I think the study argues against the sweeping conclusion that the differences are inaudible for all listeners under all conditions, but supports the conclusion that the vast majority of listeners will find the differences to be very subtle at most.

yes yes we all know. We need to adhere to insanely strict (essentially un-achievable) scientific principles here. Now if you can provide a link to a similar study that shows any statistically significant evidence that people using an array of equipment can reliably tell the difference between high br lossless and lossy...

 

[bfreedma]

I don't disagree at all - I agree with you that: "The majority of listeners who listen to stereo music in pure stereo will probably not notice any difference".
I wouldn't disagree at all with that assertion.
(However, you are incorrect in one thing; some people on this forum are trying to argue that "high quality lossy files are never audibly different".)

I also disagree with your guess about the percentages...
A significant percentages of the people these days listen to their music on various surround sound and home theater systems.
Likewise, many people with computers, and many separate headphone amps, include various "headphone processing" options of one sort or another.
(And many of those processors operate by acting on minor channel differences, or minor phase differences, which are altered by many lossy decoders.)


As I mentioned, here at Emotiva, we sell both stereo and surround sound equipment.
And, while we haven't kept count, I can tell you that, based on their support questions, a large percentage of our home theater customers listen to all their music in surround sound.
And I suspect Dolby Labs might dispute your suggestion that using their Dolby headphone plugin is NOT "a common listening situation".
(If you believe their counter - the plugin for "Dolby Atmos and Dolby Atmos Headphone for Windows" has been downloaded over 35,000 times from CNET alone.)
Or you might consider the lively debates about "which upmixer works better with stereo content" to suggest that in fact a lot of people use one or the other in that situation.

It might be interesting to do a survey on Head-Fi to find out how many headphone listeners use some sort of headphone enhancement or cross-mix plugin or gadget.
(And it might be equally interesting to actually test what effect various lossy compression methods and settings have on the AUDIBLE outputs of each.)

For someone who claims to have a background in statistics, you certainly do abuse them.

Windows has an install base exceeding 1 billion devices. 35000 downloads of a DSP shows how few people actually potentially use it, not that it comprises a significant percentage of users.

You also conflate the percentage of people who purchase home theater gear and use DSP with the overall population. While I certainly don’t have statistics at hand, I would be beyond shocked if the percentage of music listening done via multichannel and DSP enhanced stereo is anywhere close to 1%. The use rate of phones dwarfs all other playback hardware - it’s easy to forget that people who spend time on forums such as this do not reflect the typical listener. We are a fraction of a fraction of a percent. The vast majority use whatever device they are presented with, whether phone, tablet, computer, plug in speakers or headphones, and listen to whatever the default output of the device is.

To suggest that surveys your customers or head-FI members is somehow representative of global usage shows either a complete lack of statistical model awareness or an intentional attempt to skew results.

 

[Phronesis]

yes yes we all know. We need to adhere to insanely strict (essentially un-achievable) scientific principles here. Now if you can provide a link to a similar study that shows any statistically significant evidence that people using an array of equipment can reliably tell the difference between high br lossless and lossy...

Scientific standards can be adhered to if proper scientific testing is done.

I haven't tried to search the literature on this because I don't care about the issue beyond some curiosity, but will be happy to read anything that's pointed out here in the forum.

I'm not arguing that some people can reliably tell the difference, I'm saying that I haven't seen good evidence that the opposite has been demonstrated. Without conclusive evidence either way, we can only draw tentative conclusions (essentially hypotheses).

 

[Sgt. Ear Ache]

Scientific standards can be adhered to if proper scientific testing is done.

I haven't tried to search the literature on this because I don't care about the issue beyond some curiosity, but will be happy to read anything that's pointed out here in the forum.

I'm not arguing that some people can reliably tell the difference, I'm saying that I haven't seen good evidence that the opposite has been demonstrated. Without conclusive evidence either way, we can only draw tentative conclusions (essentially hypotheses).

you have seen "good" evidence. You choose to ignore it.

 

[Phronesis]

you have seen "good" evidence. You choose to ignore it.

Did you read my post on the article you cited? As I said, I think that article provides evidence against your conclusion.

 

[Sgt. Ear Ache]

Did you read my post on the article you cited? As I said, I think that article provides evidence against your conclusion.

my conclusion is that any test in which people are tasked with choosing which file is lossy and which is lossless, or which file is the "better sounding" of a set of lossless and (high bitrate) lossy files will essentially come out random - because the differences are essentially in-audible. That was the conclusion reached by the test I linked.

 

[castleofargh]

There was a time when "nobody could tell the difference between a cylinder recording and a live performer".
Then people insisted that vinyl "was so close to perfect that there was no point in looking for improvement".
Then we were told that most people couldn't tell "is it live or is it Memorex" (referring to cassettes).
Then there was a time when "most people were sure that 128k MP3 files were audibly perfect".
(Note that the developers of the MP3 compression process never made claims beyond that "most listeners" wouldn't notice a difference with "most music".)

I doubt this was true. well some probably said it in reviews, the previous generation of flowery extravaganza must have been full of "OMG I started playing the song and the wife ran away thinking people were in the house!". but let's be honest, a wax cylinder never contended as a perfect reproduction medium. same stuff with vinyls, it's never been all that hard for vinyl playback to fail at transparency.
I expect that those days were for audio the same way it has been for video/pictures in the decades. or tv screens and computer monitors. when we get the very best stuff, we certainly are very impressed, but that doesn't mean we're all convinced that we have achieved visual transparency. it's becoming harder and harder, but we're still clear about the limitations on colors, contrast, sometimes refresh rate. and those aren't conceptual ideas like hires masturbation, we can capture a scene, put it on a screen and anybody will be able to tell that it's different in a test where you can see both. not everybody will know how to call the difference, but noticing some is still to this day, no trouble at all. I imagine wax cylinders, vinyls and k7 tapes to have given the same feeling. at least that's how they always felt to me even before CDs were sold.

on the other hand, 256ACC falls into that marvelous category where almost nobody can pass a proper blind test, but apparently almost everybody "knows" that he can tell it apart from lossless. so as time goes by and the evidence continues to diverge from the claims, it becomes really hard not to consider that those people claiming audible difference are full of crap, or don't know what they're doing.

 

[KeithEmo]

You misunderstand.... I was being a bit sarcastic.

However, my statement stands....
If you accept their results, then they did in fact find someone who could reliably tell the difference, which makes the claim that "nobody can tell the difference" false.

Likewise, if you were to claim that "humans can't run a mile in less than four minutes", you would be incorrect.
It doesn't matter how many millions of humans you can find who cannot do so.
As soon as that one guy did so, it then became proven fact that "humans can do so", and the question has been answered.
(When you make an absolutely general claim, then a SINGLE exception proves your claim false... which is why making generalized claims is such a dangerous proposition.)

However, I quite agree, their test was not well controlled.
(That one subject could have cheated, we don't know what test samples they used, and they only tested using a single encoder, when we all know that lossy encoders aren't standardized.)

Odd how you toss away every other test based on your absurd and unachivable criteria but seem more than willing to accept this one without applying the same.

Not that I’m surprised.

 

[castleofargh]

@Sgt. Ear Ache , before you go mad about miscommunication like I did oh so many times with @Phronesis before getting what his actual point was, let me explain this one as I understand it: he doesn't care at all to say that high bitrate lossy isn't transparent, what he argues against is claiming that it universally is. it might look the same but it's really not. because for obvious reasons, it's hard to test every humans, every tracks, every playback gears with every encoders. so from his position, the absolute claim of transparency cannot and shoudn't be made. that's all.
if 256 AAC is transparent for the all universe but one guy with one audio file, he'll argue that the universal claim has been effectively disproved. which is strictly true, at least for people who aren't big on statistical concepts. ^_^

@Phronesis did I get that right for once?

 

[Sgt. Ear Ache]

@Sgt. Ear Ache , before you go mad on about miscommunication like I did oh so many times with @Phronesis before getting what his actual point was, let me explain this one as I understand it: he doesn't care at all to say that high bitrate lossy isn't transparent, what he argues against is claiming that it universally is. it might look the same but it's really not. because for obvious reasons, it's hard to test every humans, every tracks, every playback gears with every encoders. so from his position, the absolute claim of transparency cannot and shoudn't be made. that's all.
if 256 AAC is transparent for the all universe but one guy with one audio file, he'll argue that the universal claim has been effectively disproved. which is strictly true, at least for people who aren't big on statistical concepts. ^_^

@Phronesis did I get that right for once?

yes I understand that. Same goes for Kemo apparently.

Sure, there's humans who can run a sub-4 minute mile. But don't nobody better try and sell me a pair of shoes claiming they will make ME able to run a sub 4 mile...

 

[Phronesis]

my conclusion is that any test in which people are tasked with choosing which file is lossy and which is lossless, or which file is the "better sounding" of a set of lossless and (high bitrate) lossy files will essentially come out random - because the differences are essentially in-audible. That was the conclusion reached by the test I linked.

Nope, the conclusion isn't supported by the results. Multiple p-values close to 0.05 suggests that there could be a difference. A p-value of 0.07 can roughly be interpreted as saying that there's only a 7% chance that you would get a difference from null, at least as big as the difference observed, due to random chance, so there's a 93% chance that the difference is real. Whether this number is 7% or 5% or 4% doesn't really matter, they're all similarly low numbers.

 

[KeithEmo]

I absolutely agree.....

And what percentage of people who listen on their phone listen in pure stereo?
And what percentage use one of the many so-called "enhancement plugins" - many of which also perform various sorts of DSP-type audio processing?
And, while we're at it, how many people listen to music on their TV set these days (my cable company offers all sorts of music channels)?
And how many listen to music videos on their TV - probably in some sort of surround sound.
That is an excellent question, and one that would take a lot of research to answer....

My guess is that the majority of iPhone users don't even KNOW if the audio they're listening to has been processed or not.

In order to make any sort of meaningful statement you need to specify a lot of things.

my conclusion is that any test in which people are tasked with choosing which file is lossy and which is lossless, or which file is the "better sounding" of a set of lossless and (high bitrate) lossy files will essentially come out random - because the differences are essentially in-audible. That was the conclusion reached by the test I linked.

For someone who claims to have a background in statistics, you certainly do abuse them.

Windows has an install base exceeding 1 billion devices. 35000 downloads of a DSP shows how few people actually potentially use it, not that it comprises a significant percentage of users.

You also conflate the percentage of people who purchase home theater gear and use DSP with the overall population. While I certainly don’t have statistics at hand, I would be beyond shocked if the percentage of music listening done via multichannel and DSP enhanced stereo is anywhere close to 1%. The use rate of phones dwarfs all other playback hardware - it’s easy to forget that people who spend time on forums such as this do not reflect the typical listener. We are a fraction of a fraction of a percent. The vast majority use whatever device they are presented with, whether phone, tablet, computer, plug in speakers or headphones, and listen to whatever the default output of the device is.

To suggest that surveys your customers or head-FI members is somehow representative of global usage shows either a complete lack of statistical model awareness or an intentional attempt to skew results.