A Meta-Analysis of High Resolution Audio Perceptual Evaluation (or How We Learned to Stop Worrying and Love Hi-Res)
Jun 27, 2016 at 9:26 PM Thread Starter Post #1 of 22

RRod

Headphoneus Supremus
Joined
Aug 25, 2014
Posts
3,371
Likes
972
This is free right now from AES. Haven't had a chance to go through it yet, but thought I'd start up a discussion.
 
Edit:
Had a chance to do a cursory read. The take-home seems to be: the studies that passed the author's muster combine to suggest an ability to discriminate high-resolution tracks, that training improves this ability, and that there is no way to say what is being heard.
 
I'm left with lots of questions:
.How do you "train" people to hear frequencies beyond what they can detect at normal listening levels?
.Are trained listeners more likely to listen at levels where we might expect higher levels of distortion?
.Throughout these studies, was CD material *always* a downsampling of the same master as the hi-res?
.Would we not have significance if we demanded a higher confidence level?
 
Jun 27, 2016 at 10:31 PM Post #2 of 22
I'd be curious to read the discussions at AES on this paper.  Individually, many of those tests that were significantly weighted have been questioned and their conclusions remain suspect on their own.  Putting them all together and calling it a consensus gives me pause.
 
I'm always skeptical, especially when there is potential benefit in the form of sales.  LANDR offers to master tracks at HD WAV at twice the cost of standard WAV.  At least one company that is indirectly behind this AES paper is looking to benefit from selling their product in HD format, so they better be able to justify that the sound quality "improvements" are legitimately worth it.
 
https://www.landr.com/en/pricing
 
Jun 27, 2016 at 10:33 PM Post #3 of 22
  I'd be curious to read the discussions at AES on this paper.  Individually, many of those tests that were significantly weighted have been questioned and their conclusions remain suspect on their own.  Putting them all together and calling it a consensus gives me pause.
 
I'm always skeptical, especially when there is potential benefit in the form of sales.  LANDR offers to master tracks at HD WAV at twice the cost of standard WAV.  At least one company that is indirectly behind this AES paper is looking to benefit from selling their product in HD format, so they better be able to justify that the sound quality "improvements" are legitimately worth it.
 
https://www.landr.com/en/pricing

 
Yes the comments almost always make these little details come out of the woodwork. And my first thought was "now I have to go back and recall what the objections were to each of these studies when they came out." Oh the joys of meta-analysis.
 
Jun 28, 2016 at 9:58 AM Post #4 of 22
Here is an article with some response from Dr. Reiss about the study.
 
https://www.sciencedaily.com/releases/2016/06/160627214255.htm
 
Dr Reiss explained: "One motivation for this research was that people in the audio community endlessly discuss whether the use of high resolution formats and equipment really make a difference. Conventional wisdom states that CD quality should be sufficient to capture everything we hear, yet anecdotes abound where individuals claim that hi-res content sounds crisper, or more intense. And people often cherry-pick their favourite study to support whichever side they're on.

"Our study is the first attempt to have a thorough and impartial look at whether high res audio can be heard. We gathered 80 publications, and analysed all available data, even asking authors of earlier studies for their original reports from old filing cabinets. We subjected the data to many forms of analysis. The effect was clear, and there were some indicators as to what conditions demonstrate it most effectively. Hopefully, we can now move forward towards identifying how and why we perceive these differences."

 
Again, the issue I have is that there were rational problems brought about surrounding the methodologies of some of these tests.  I think a proper study would concern itself with attempting to identify the underlying biases introduced with many of these tests that might account for the small but statistically significant ability to discriminate high resolution content.
 
Where is our resident Statement Analyst when we need him? 
 
Why did the test extend to studies as far back as 1980?  Why not 2000, or even 2005?  Was it important to include the Oohashi high frequency brain-shaking balderdash to get the statistical relevancy necessary to form the conclusion?  
 
In no way do I see the results of this study as being final.  It needs to be impartially reviewed by experts in the field, and not simply an endorsement from Bob Stuart and the Meridian team.  This study's conclusion will have overwhelming support from popular audio reviewers, the music industry, and most of the audio gear manufacturers, and it will be difficult to find genuine feedback.
 
Jun 28, 2016 at 11:21 AM Post #5 of 22
  In no way do I see the results of this study as being final.  It needs to be impartially reviewed by experts in the field, and not simply an endorsement from Bob Stuart and the Meridian team.  This study's conclusion will have overwhelming support from popular audio reviewers, the music industry, and most of the audio gear manufacturers, and it will be difficult to find genuine feedback.

 
But that's the thing: this is published in JAES. That kind of review should have already happened, and should have included reviewers with no corporate interest as well as an external statistical reviewer to vet the meta-analytics. What's the real story behind this paper, then, or is it bannination-worthy to talk about it?
 
Discussion going a bit on hydrogen as well.
 
Jun 30, 2016 at 5:54 AM Post #6 of 22

whoa whoa... wait a second, so people had easier time distinguishing CD from MP3 than HRA from MP3?
 
How can they spot the MP3 better vs. CD than HRA?
 
So 128 kbps MP3 is "closer" to HRA than a CD? Really? Am I missing something here or reading it wrong?
 
Jun 30, 2016 at 7:12 AM Post #7 of 22
All,
 
It's all about the promise of hi-res, not the truth of hi-res. People will buy it simply to not be left out of things. Thing is, if CD was actually perfect sound, perfect forever, which indeed seems to be fact, then can anything be more perfect? So, far I can't discern any benefit from hi-res. It does not prompt me to spend more for opportunity to listen to it. I believe I'll  pass on hi-res until I no longer have a choice. 
 
Jul 6, 2016 at 9:49 PM Post #8 of 22
I read the abstract but not the article. I've been wondering if all the positive tests were using the Pono Player and it's hardware built in bias for higher sampling rates? LOL

Seriously, though, I wondered if the playback equipment was properly tested to confirm that it was not biasing the results in any of the tests.

Regardless, doesn't affect me. I'm too old to train now to hear the difference. :dt880smile:
 
Jul 8, 2016 at 10:42 PM Post #10 of 22
  http://blogs.plos.org/absolutely-maybe/2016/07/05/psychologys-meta-analysis-problem/
 
This might be a little bit relevant. 

 
But that's just it. *Before* the paper gets published would be a good time to have figured out if it were a bad application of meta-analysis. The fact is that the numbers for the non-training studies teeter right on the edge of significance: a different model, a dropped/added study, a higher significance level can all change the result for this group, which is of course the group into which we'd place most people saying how great DSoTM sounds at DSD1024.
 
Jul 9, 2016 at 11:45 AM Post #11 of 22
   
But that's just it. *Before* the paper gets published would be a good time to have figured out if it were a bad application of meta-analysis. The fact is that the numbers for the non-training studies teeter right on the edge of significance: a different model, a dropped/added study, a higher significance level can all change the result for this group, which is of course the group into which we'd place most people saying how great DSoTM sounds at DSD1024.

 
The particular wording used by Dr. Reiss in the paper is specific and calculated in order to get by peer review.  Facing a much larger audience and without the same rigors necessary for validity, his press release statement vaults to conclusions that do not appear to be as clearly evident.
 
In no way does this analysis come across as an objective search for the truth.  There was a significant amount of effort and finagling to find a tenuous result that catered to the business side of this increasingly misguided industry and its plethora of unwitting supporters.
 
Jul 9, 2016 at 12:48 PM Post #12 of 22
   
The particular wording used by Dr. Reiss in the paper is specific and calculated in order to get by peer review.  Facing a much larger audience and without the same rigors necessary for validity, his press release statement vaults to conclusions that do not appear to be as clearly evident.
 
In no way does this analysis come across as an objective search for the truth.  There was a significant amount of effort and finagling to find a tenuous result that catered to the business side of this increasingly misguided industry and its plethora of unwitting supporters.


You hit the nail squarely on the head. The whole thing reeks of the usual stench that continually wafts from the high end audio industry. Unfortunately now that this piece of trash masquerading as a serious study has been released to the public one can and should expect the entire high end audio industry to embrace it as if it came directly from God.
 
Jul 9, 2016 at 5:43 PM Post #13 of 22
 
You hit the nail squarely on the head. The whole thing reeks of the usual stench that continually wafts from the high end audio industry. Unfortunately now that this piece of trash masquerading as a serious study has been released to the public one can and should expect the entire high end audio industry to embrace it as if it came directly from God.

 
If as much energy was put forth into a double-blind test, we might enjoy a result that is significantly more definitive.  However, this result would most assuredly not be congruent with what the music industry stubbornly believes that it needs.  There is a huge push to get listeners to think that they need to replace their music library with a new format that costs more, and high resolution is seen as this panacea.  
 
Me?  I'm going to be content streaming Google Play All Access for the majority of my music entertainment.  This is so darn close, to the point of being practically identical, to what I enjoy with CDs, and miles above the quality of cassette tapes and records that I used in the 70's and early 80's.   
 
Jul 9, 2016 at 6:13 PM Post #14 of 22
   
If as much energy was put forth into a double-blind test, we might enjoy a result that is significantly more definitive.  However, this result would most assuredly not be congruent with what the music industry stubbornly believes that it needs.  There is a huge push to get listeners to think that they need to replace their music library with a new format that costs more, and high resolution is seen as this panacea.  
 
Me?  I'm going to be content streaming Google Play All Access for the majority of my music entertainment.  This is so darn close, to the point of being practically identical, to what I enjoy with CDs, and miles above the quality of cassette tapes and records that I used in the 70's and early 80's.   


I'm not 100% sure but I believe that some of the tests included in the meta-analysis were double blind but poorly controlled, e.g. not using downsampled high resolution files for the standard (CD) resolution tests.
 
I do agree with you that paid content streaming is a good way to go for the vast majority of people. Services like Tidal, Apple Music, Spotify and Google Play offer vast music libraries and very good sound quality (at least with a paid subscription).
 
Plus what good is high resolution audio if one is just going to listen via bluetooth.
 
Jul 9, 2016 at 6:28 PM Post #15 of 22
 
I'm not 100% sure but I believe that some of the tests included in the meta-analysis were double blind but poorly controlled, e.g. not using downsampled high resolution files for the standard (CD) resolution tests.
 
I do agree with you that paid content streaming is a good way to go for the vast majority of people. Services like Tidal, Apple Music, Spotify and Google Play offer vast music libraries and very good sound quality (at least with a paid subscription).
 
Plus what good is high resolution audio if one is just going to listen via bluetooth.

Yes, the ABX in several of these tests were largely inconclusive as a key piece of the process may have introduced an anomaly outside of the intended parameters that potentially disrupted the results.  Put several of these types of tests together, and manipulate the results in a way to get the "answer" you are searching for, and this sums up the latest meta-analysis paper.
 

Users who are viewing this thread

Back
Top