Why Audio Science Review's measurements are so different compared to other sources?
post-15709196
Thread Starter
Post #1 of 13

KeithPhantom

100+ Head-Fier
Joined
Apr 3, 2020
Messages
111
Reaction score
64
Location
Orlando, Florida
Joined
Apr 3, 2020
Location
Orlando, Florida
Posts
111
Likes
64
When researching for measurements I have seen multiple websites and sources that provide these. One of those websites (forum) is Audio Science Review directed by its creator Amir. He is really knowledgeable guy which his posts have taught me a lot of audio processing and measurement, but something I have noticed is how divergent some of his measurements are compared to other websites and/or the manufacturer itself. I know there could be bias or set-up measurements that do not really represent the actual performance of the gear, but there have been some discussion about some equipment that has been measured by Amir and other sources such as Jude and the manufacturers that show significant differences in the measurements that cannot be explained as simple production variations.

Others have explicitly said that Amir hacks the measurements to be worse because he does not like a manufacturer, thing I cannot affirm since I have never seen something that indicates this to be true, but his work can be radical compared to the results others get. I want to express that this is an honest question and I do not have another interest other than knowing what is this about and what could be the reasons for this. I do not want to attack him or anyone involved in this, I just would like to have a better framework of the people doing the measurements. Thanks for your time.

PS: this will be also published in ASR to know their opinion about this issue.
 
     Share This Post       
post-15709780
Post #2 of 13

Speedskater

500+ Head-Fier
Joined
Dec 27, 2006
Messages
832
Reaction score
120
Joined
Dec 27, 2006
Posts
832
Likes
120
As the types of tests that Amir uses are the ones available in Audio Precision test equipment and similar to the tests done by others with skill, I would trust his measurements. The 'Audio Science Review' forum is open to discussions about suitable test protocols.
 
     Share This Post       
post-15710116
Post #3 of 13
Joined
Jun 20, 2001
Messages
10,878
Reaction score
4,506
When researching for measurements I have seen multiple websites and sources that provide these. One of those websites (forum) is Audio Science Review directed by its creator Amir. He is really knowledgeable guy which his posts have taught me a lot of audio processing and measurement, but something I have noticed is how divergent some of his measurements are compared to other websites and/or the manufacturer itself. I know there could be bias or set-up measurements that do not really represent the actual performance of the gear, but there have been some discussion about some equipment that has been measured by Amir and other sources such as Jude and the manufacturers that show significant differences in the measurements that cannot be explained as simple production variations....
You won't get any argument from me that Amir is very knowledgable. We also finally met in person at CanJam SoCal last year, and it was a friendly meeting.

I looked for posts that you may be referring to, and had to search back to just about exactly two years ago to the day.

The discussion centered around measurements of a particular DAC model, the Schiit Audio Yggdrasil 2. I posted measurements of that DAC, and some did look different from Amir's, and there was definitely discussion about that -- and the discussion no doubt got heated at times. I think the biggest debate was about the linearity measurements and methodology. I summarized that back-and-forth discussion about linearity measurements in this post:

https://www.head-fi.org/threads/sch...e-technical-measurements.881867/post-14327606

It's a long read, but on the discussion of linearity in the context of the discussing and arguing then, it's rather thorough.

If you get through that and still feel like reading, here are a couple of other links related to it (and I'm sure you can find more on his forum):

https://www.head-fi.org/threads/sch...e-technical-measurements.881867/post-14332522

https://www.head-fi.org/threads/sch...e-technical-measurements.881867/post-14328947

...Others have explicitly said that Amir hacks the measurements to be worse because he does not like a manufacturer, thing I cannot affirm since I have never seen something that indicates this to be true, but his work can be radical compared to the results others get. I want to express that this is an honest question and I do not have another interest other than knowing what is this about and what could be the reasons for this. I do not want to attack him or anyone involved in this, I just would like to have a better framework of the people doing the measurements. Thanks for your time.

PS: this will be also published in ASR to know their opinion about this issue.
Hacks his measurements? Radical? You trying to stir things up? There was heat under the collars then, no doubt, but that was also two years ago.

I think there's a greater likelihood we were measuring two units that varied. Another gentleman who goes by the username atomicbob installed the Yggdrasil V2 upgrades on his first-generation Yggdrasil himself (as I recall it). (This was two years ago, but I think I'm remembering that right.) I think he and I had measurements that were more similar. Again, perhaps there were unit differences with the unit Amir had.

I do remember questioning the strange appearance of a THD+N measurement that he posted, in which it seemed to me it was likely too bandwidth-limited to show THD+N out to 20 kHz. As it turns out, his bandwidth was indeed set too low for that measure, and I think he corrected it after it was pointed out.

If you're itching to see the heat rekindled on this old issue, you may have more success over in his forum.
 
Last edited:
     Share This Post       
post-15710153
Post #4 of 13

bigshot

Headphoneus Supremus
Joined
Nov 16, 2004
Messages
21,230
Reaction score
3,395
Location
Hollywood USA
Joined
Nov 16, 2004
Location
Hollywood USA
Posts
21,230
Likes
3,395
Website
www.facebook.com
The differences were also likely measurable, but far below the threshold of hearing. All equipment has manufacturing tolerances. Numbers on a page can vary, but that doesn't matter if human ears can't hear it.
 
     Share This Post       
post-15710154
Post #5 of 13

KeithPhantom

100+ Head-Fier
Joined
Apr 3, 2020
Messages
111
Reaction score
64
Location
Orlando, Florida
Joined
Apr 3, 2020
Location
Orlando, Florida
Posts
111
Likes
64
You won't get any argument from me that Amir is very knowledgable. We also finally met in person at CanJam SoCal last year, and it was a friendly meeting.

I looked for posts that you may be referring to, and had to search back to just about exactly two years ago to the day.

The discussion centered around measurements of a particular DAC model, the Schiit Audio Yggdrasil 2. I posted measurements of that DAC, and some did look different from Amir's, and there was definitely discussion about that -- and the discussion no doubt got heated at times. I think the biggest debate was about the linearity measurements and methodology. I summarized that back-and-forth discussion about linearity measurements in this post:

https://www.head-fi.org/threads/sch...e-technical-measurements.881867/post-14327606

It's a long read, but on the discussion of linearity in the context of the discussing and arguing then, it's rather thorough.

If you get through that and still feel like reading, here are a couple of other links related to it (and I'm sure you can find more on his forum):

https://www.head-fi.org/threads/sch...e-technical-measurements.881867/post-14332522

https://www.head-fi.org/threads/sch...e-technical-measurements.881867/post-14328947



Hacks his measurements? Radical? You trying to stir things up? There was heat under the collars then, no doubt, but that was also two years ago.

I think there's a greater likelihood we were measuring two units that varied. Another gentleman who goes by the username atomicbob installed the Yggdrasil V2 upgrades on his first-generation Yggdrasil himself (as I recall it). (This was two years ago, but I think I'm remembering that right.) I think he and I had measurements that were more similar. Again, perhaps there were unit differences with the unit Amir had.

I do remember questioning the strange appearance of a THD+N measurement that he posted, in which it seemed to me it was likely too bandwidth-limited to show THD+N out to 20 kHz. As it turns out, his bandwidth was indeed set too low for that measure, and I think he corrected it after it was pointed out.

If you're itching to see the heat rekindled on this old issue, you may have more success over in his forum.
Actually it was an honest question. About "hacking" I meant that maybe measurements are set-up to have a certain outcome, but I don't think so in both of your cases (differences can be present just for any change in the measurement methodology, which guys over there just explained me) . I would not want to assume anything without having any evidence to back it up. I respect both of you since you both have taught me stuff I didn't understand, and that's something I'm grateful for.

Jude, I'm a guy that actually wants to learn how his stuff works, from the simplest resistor to the most complex FPGA, and it was itching me the fact that I found differences in the same product not just with the Yggdrasil (I only used it as example since it is the most know case of this), also with other products from different manufacturers. Some differences (I would even say most) are attributed to manufacturing divergences between units, but some were striking. I wondered the reasons, but guys at ASR showed me all the variables that increase the marginal deltas between samples.

I love audio (dreamed to study digital signal processing, but chose finance) and I still have my scientific arm always ready for the flexing (always been good in science). I'm humble, I learn and admit my mistakes if I ever make one. For now, I see that many of my questions have been answered, I can lay my doubts down and move to enjoy the music. By the way, thanks for answering, it is flattering to have the owner of a large forum to actually answer questions and inquires of an unknown,

Keith
 
     Share This Post       
post-15710161
Post #6 of 13

KeithPhantom

100+ Head-Fier
Joined
Apr 3, 2020
Messages
111
Reaction score
64
Location
Orlando, Florida
Joined
Apr 3, 2020
Location
Orlando, Florida
Posts
111
Likes
64
The differences were also likely measurable, but far below the threshold of hearing. All equipment has manufacturing tolerances. Numbers on a page can vary, but that doesn't matter if human ears can't hear it.
Well, Ethan Winer establishes that -80 dBFS is a safe threshold of inaudibility (I have blind tested it, but don't have the log, I'll repeat it someday) is inaudible but guys over ASR say anything over -100 or -110 dBFS is way to high.
 
     Share This Post       
post-15710167
Post #7 of 13

bigshot

Headphoneus Supremus
Joined
Nov 16, 2004
Messages
21,230
Reaction score
3,395
Location
Hollywood USA
Joined
Nov 16, 2004
Location
Hollywood USA
Posts
21,230
Likes
3,395
Website
www.facebook.com
Check out the videos in my sig. Ethan does a demonstration in them where the most annoying noise he could possibly come up with becomes inaudible under music at about -40dB or so.

A lot of people in audiophilia throw around numbers with no context about what those numbers mean in real world sound. The decibel scale is logarithmic. -40 is massively different than -80dB. If you look at normal music listening levels, 80dB is quite loud. Do you think you could hear something down at 1 or 2dB when Led Zeppelin is playing in your ears at 80dB? When you get up over 100dB, you're close to the threshold of pain and incurring hearing damage.

Science is great, but you also have to keep an eye on how the equipment is actually used in the home to listen to music while sitting on a couch. Real world counts. All it takes to figure out how those numbers relate to reality is a cheap audio app and a few experiments playing around with the settings. You will have a better sense of it all if you hear it for yourself, rather than taking someone else's word for it.
 
Last edited:
     Share This Post       
post-15710183
Post #8 of 13
Joined
Jun 20, 2001
Messages
10,878
Reaction score
4,506
Actually it was an honest question. About "hacking" I meant that maybe measurements are set-up to have a certain outcome, but I don't think so in both of your cases (differences can be present just for any change in the measurement methodology, which guys over there just explained me) . I would not want to assume anything without having any evidence to back it up. I respect both of you since you both have taught me stuff I didn't understand, and that's something I'm grateful for.

Jude, I'm a guy that actually wants to learn how his stuff works, from the simplest resistor to the most complex FPGA, and it was itching me the fact that I found differences in the same product not just with the Yggdrasil (I only used it as example since it is the most know case of this), also with other products from different manufacturers. Some differences (I would even say most) are attributed to manufacturing divergences between units, but some were striking. I wondered the reasons, but guys at ASR showed me all the variables that increase the marginal deltas between samples.

I love audio (dreamed to study digital signal processing, but chose finance) and I still have my scientific arm always ready for the flexing (always been good in science). I'm humble, I learn and admit my mistakes if I ever make one. For now, I see that many of my questions have been answered, I can lay my doubts down and move to enjoy the music. By the way, thanks for answering, it is flattering to have the owner of a large forum to actually answer questions and inquires of an unknown,

Keith
Sorry for rushing to judge you a rabble-rouser, Keith.

Welcome to Head-Fi. :)
 
     Share This Post       
post-15710185
Post #9 of 13

KeithPhantom

100+ Head-Fier
Joined
Apr 3, 2020
Messages
111
Reaction score
64
Location
Orlando, Florida
Joined
Apr 3, 2020
Location
Orlando, Florida
Posts
111
Likes
64
Check out the videos in my sig. Ethan does a demonstration in them where the most annoying noise he could possibly come up with becomes inaudible under music at about -40dB or so.

A lot of people in audiophilia throw around numbers with no context about what those numbers mean in real world sound. The decibel scale is logarithmic. -40 is massively different than -80dB. If you look at normal music listening levels, 80dB is quite loud. Do you think you could hear something down at 1 or 2dB when Led Zeppelin is playing in your ears at 80dB? When you get up over 100dB, you're close to the threshold of pain and incurring hearing damage.

Science is great, but you also have to keep an eye on how the equipment is actually used in the home to listen to music while sitting on a couch. Real world counts. All it takes to figure out how those numbers relate to reality is a cheap audio app and a few experiments playing around with the settings. You will have a better sense of it all if you hear it for yourself, rather than taking someone else's word for it.
That is what I'm thinking about, even if they are not frequency related or even close, the masking effect by loudness differential should be more enough to mask the weaker signal for virtually every listener (assuming in my ignorance your example). Harmonic distortion is usually related to the fundamental, so it is to be expected to be masked at those levels. This is something I would like to have more research about.

PS: I have watched Ethan's videos already, very informing.
 
     Share This Post       
post-15710186
Post #10 of 13

KeithPhantom

100+ Head-Fier
Joined
Apr 3, 2020
Messages
111
Reaction score
64
Location
Orlando, Florida
Joined
Apr 3, 2020
Location
Orlando, Florida
Posts
111
Likes
64
Sorry for rushing to judge you a rabble-rouser, Keith.

Welcome to Head-Fi. :)
Don't worry, I have been a lurker for years (mind you, 2014?). Thanks for the welcoming.
 
     Share This Post       
  • Like
Reactions: jude
post-15710475
Post #11 of 13

castleofargh

Sound Science Forum Moderator
Joined
Jul 2, 2011
Messages
8,962
Reaction score
4,295
Joined
Jul 2, 2011
Posts
8,962
Likes
4,295
If you provide a few examples of such differences it could help make sense of what might have happened(or if it's even worth looking at the deviation depending on the magnitudes measured). and if you think there are trends, having the ability to see them would surely help us make our mind.
I assume that if Amir or some other guy made a mistake or had equipment issues, they'd like to find out about it. I don't imagine people bother to spend so much time following measurement protocols and getting measurement equipment(some really expensive) just to mislead others. Usually when they do, I'd think it's unwillingly and unknowingly.
We're all just humans, even at my so very amateurish level, I can't count the number of times I've measured something under the wrong conditions/settings. And those are the times I noticed!!!! It's even harder to pick up on a problem/mistake when testing a device we just got and know little about. If I go measure stuff I've measured 30 times, of course I'll tend to notice anything out of the ordinary, but with gears I just got... I'll be more likely to assume that it's how the device behaves and not question my results too much.
 
     Share This Post       
post-15710779
Post #12 of 13

KeithPhantom

100+ Head-Fier
Joined
Apr 3, 2020
Messages
111
Reaction score
64
Location
Orlando, Florida
Joined
Apr 3, 2020
Location
Orlando, Florida
Posts
111
Likes
64
If you provide a few examples of such differences it could help make sense of what might have happened(or if it's even worth looking at the deviation depending on the magnitudes measured). and if you think there are trends, having the ability to see them would surely help us make our mind.
I assume that if Amir or some other guy made a mistake or had equipment issues, they'd like to find out about it. I don't imagine people bother to spend so much time following measurement protocols and getting measurement equipment(some really expensive) just to mislead others. Usually when they do, I'd think it's unwillingly and unknowingly.
We're all just humans, even at my so very amateurish level, I can't count the number of times I've measured something under the wrong conditions/settings. And those are the times I noticed!!!! It's even harder to pick up on a problem/mistake when testing a device we just got and know little about. If I go measure stuff I've measured 30 times, of course I'll tend to notice anything out of the ordinary, but with gears I just got... I'll be more likely to assume that it's how the device behaves and not question my results too much.
Well, I'll post my link of one of my entries that show some differences. They already have explained then and Jude also explained me with some more information, but thank you for caring about this personal question of mine.

Link to the thread: https://www.audiosciencereview.com/...t-compared-to-other-sources.14435/post-443604
 
     Share This Post       
post-15711841
Post #13 of 13

bigshot

Headphoneus Supremus
Joined
Nov 16, 2004
Messages
21,230
Reaction score
3,395
Location
Hollywood USA
Joined
Nov 16, 2004
Location
Hollywood USA
Posts
21,230
Likes
3,395
Website
www.facebook.com
That is what I'm thinking about, even if they are not frequency related or even close, the masking effect by loudness differential should be more enough to mask the weaker signal for virtually every listener
The human ear also adapts to volume levels, so if you are listening to loud music, even after the music stops, you won't hear a soft whisper as well as if you were in a quiet library. Thinking about it, your -80dB number might be correct if the ears have time to acclimate to total silence, but that wouldn't be the figure if you are listening to music. It's important to take the way you listen into account when you refer to figures too. A number in one context might not apply at all in another.

For comparison, the best an LP can produce in terms of signal to noise is about -40 to -50dB, and that can sound very good. If you had a totally silent LP with a very good pressing, you could probably barely hear the surface noise at normal listening volume, but you wouldn't hear the surface under music.

Good enough is never good enough for some folks. CDs were designed to be completely transparent, and they are. But that didn't stop audiophiles from buying SACDs or arguing about whether 24/192 sounds better than 24/96. If you hold up a bigger number, someone will say they can hear the difference because their ears are "trained" and they know a whole lot of stuff about it. The truth though is that human ears are human ears. They hear what human ears can hear and that's all.

The guys throwing around all kinds of specs rarely have a clue how those specs compare to the specs on their ears.
 
Last edited:
     Share This Post       

Users Who Are Viewing This Thread (Users: 0, Guests: 1)

Top