Why you can't trust audio measurements

[castleofargh]

Came across the vid on a FB group and this is not the 1st time I saw this kind of posts. I'm neither a subjectivist nor a objectivist but one striving to be objective, rational, logical, pragmatic and realistic. But all this doesn't mean that I'm immune from infection of ignorance so I'm constantly reflecting, and learning. But then the phenomenon that in the name of science, half baked science itself has turned into a perceived antidote of ignorance and self-constructed myth of an ideal world, is in itself quite phenomenal! (Btw this also serves as a warning reminder to myself).

Disclaimer: I don't own nor have used the product in the vid, and its being pick merely serving as a convenient example to deliberate my points.

ASR and its founder AMR has been doing some great stuff but at the same time he's kinda inadvertently spreading "misinformation" that his measurement is a "be all and end all" judgement of a piece of gear, or at least being treated as such by his followers. But is it really so? Is he measuring all things I mean ALL things? Yes, no doubt he's measuring those common, salient aspects of the gears he's dealing with, but if a significant no of ppl in a community said they hear a diff, an audible diff, has he gone out to investigate like through a ABX? And see if the "audible diff" claimed really exists? Yes or No? If yes why?, and he then should go back to his drawing board and look further! If no, the conclusion is more complete! Many of the "measure well" audio equipment manufacturers place huge, equal emphasis on measuring AND listening, and ABX as well before concluding, including Benchmark. The ASR forum has evolved into a cult that AMR's measurement is a holy grail and to his followers it seems that no further discussion tolerated and points to those who've heard diff as fools!, and that's divisive! Is he then doing the audiophile community a blessing or a curse?

I would argue that the over reliance on one source of measurement is easily explained by the scarcity of alternatives. Forget reliable, all online sources of audio measurement put together about a device still don't amount to much.
So I understand how easy it would be for me or like minded people to desperately hang onto ASR's graphs, like at some point in time, I did with Innerfidelity's graphs or Stereophile's. Even if you have a clear idea that you're not really trusting a source and that it's not conclusive of anything, you will still tend to rely on the source you distrust the least and draw conclusions from it. It's the magic of heuristics.

And that's not limited to objective data. We know the power that the first review has on the following ones and on the product's success. Last paper I read on this was about Amazon reviews but we can easily see evidence of it here, on IMDB, and just about anywhere. It's to a point where those who are consistently the first to review something are almost certain to work for the brand or benefit from it, "influencer" style. Because it works, and brands would be stupid not to abuse the system and let honest reviews be the first. I know that, but when some new stuff I'm interested in has a review out, I'll read it and get so very biased by that review that it's not even fun.
Like with most biases, knowing doesn't save me from it. I'm still curious, I still wish to make a decision as a consumer, and so I will inevitably rely on what I can find, no matter how I totally shouldn't trust it. The information I get will in practice tend to become "a fact" about the gear for me, if only deep down in my spongy brain. measurements and graphs in particular have the same power. the first graph tends to become law, not because we're gullible, but because there is all too often nothing more reliable to confirm or disprove it.
TBH, many domains of science do the same. The difference is that in the presence of too little data, a scientist will write "this could suggest ....", while the average Joe will write "it's 100% like that and I'll fight anybody who disagrees because he's evil incarnate", or something along those lines. I don't have to explain, you've seen internet ^_^.

 

[bigshot]

At some point you have to make a decision on what to buy... that is going to be based on a preponderance of evidence, not all of which would relate to specs. I find that with the generally high quality performance of audio components, the features and value for the money are more important to making a decision than a dB or two one way or the other.

 

[jjss]

I'm with you on pragmatism and logic. There is a purpose to all of this. The trick is assigning the threshold of audibility. Audiophiles have a vested interest in hearing the unbearable. They'll even deliberately cheat tests to try to prove that. And sloppy, uncontrolled impressions are sloppy and uncontrolled. It reallyisn't that hard to determine a ballpark for the threshold of audibility. If you know that, you can look at the measurements and know what may or may not be audible and what definitely isn't. And the stuff that may or may not be audible might just not matter for the purposes of listening to music in the home.

The problem I have with the way some people interpret specs is that they take the most extreme threshold as the line of "good enough" and call that the just detectable difference. Yes, you can hear noise when you loop on a fadeout and crank the volume up to much louder than a normal listening level, but who listens to music in their living room like that? And perhaps you can barely hear some distortion in test tones in particular frequencies, but under music, you would never notice it in a million years.

Agree. Audibility and knowledge in the case of "why you hear what you hear" happens to be the 2 sides of the same coin. The 16- and 24- bits word length and their respective audibility in "hires" music listening, is another case in point. A 24-bit depth "hires" file doesn't automatically mean one gets this from a systems perspective and able to hear the diff audibly! So when one is equipped with this "bit depth" knowledge he/she would be more likely to question what they hear, or led to believe. As we know, a 24-bit system needs 256 times the accuracy of a 16-bit system in order to make full use of the additional 8 bits!, furthermore a bit depth of about 22-bits would begin to exceed the ave capabilities of human hearing!

 

[jjss]

I would argue that the over reliance on one source of measurement is easily explained by the scarcity of alternatives. Forget reliable, all online sources of audio measurement put together about a device still don't amount to much.
So I understand how easy it would be for me or like minded people to desperately hang onto ASR's graphs, like at some point in time, I did with Innerfidelity's graphs or Stereophile's. Even if you have a clear idea that you're not really trusting a source and that it's not conclusive of anything, you will still tend to rely on the source you distrust the least and draw conclusions from it. It's the magic of heuristics.

And that's not limited to objective data. We know the power that the first review has on the following ones and on the product's success. Last paper I read on this was about Amazon reviews but we can easily see evidence of it here, on IMDB, and just about anywhere. It's to a point where those who are consistently the first to review something are almost certain to work for the brand or benefit from it, "influencer" style. Because it works, and brands would be stupid not to abuse the system and let honest reviews be the first. I know that, but when some new stuff I'm interested in has a review out, I'll read it and get so very biased by that review that it's not even fun.
Like with most biases, knowing doesn't save me from it. I'm still curious, I still wish to make a decision as a consumer, and so I will inevitably rely on what I can find, no matter how I totally shouldn't trust it. The information I get will in practice tend to become "a fact" about the gear for me, if only deep down in my spongy brain. measurements and graphs in particular have the same power. the first graph tends to become law, not because we're gullible, but because there is all too often nothing more reliable to confirm or disprove it.
TBH, many domains of science do the same. The difference is that in the presence of too little data, a scientist will write "this could suggest ....", while the average Joe will write "it's 100% like that and I'll fight anybody who disagrees because he's evil incarnate", or something along those lines. I don't have to explain, you've seen internet ^_^.

Well said and so true. But then isn't it the case that those influential KOLs do have some "implied" responsibility to their followers and the communities that they should point out their limitations, as caveats, so as to shy away from any absolutism perceived or real.

 

[danadam]

furthermore a bit depth of about 22-bits would begin to exceed the ave capabilities of human hearing!

22-bits is about 132 dB. That's waaay beyond capabilities of human hearing. AFAIU it's even beyond the physical limits of what can be implemented in hardware.

 

[71 dB]

Agree. Audibility and knowledge in the case of "why you hear what you hear" happens to be the 2 sides of the same coin. The 16- and 24- bits word length and their respective audibility in "hires" music listening, is another case in point. A 24-bit depth "hires" file doesn't automatically mean one gets this from a systems perspective and able to hear the diff audibly! So when one is equipped with this "bit depth" knowledge he/she would be more likely to question what they hear, or led to believe. As we know, a 24-bit system needs 256 times the accuracy of a 16-bit system in order to make full use of the additional 8 bits!, furthermore a bit depth of about 22-bits would begin to exceed the ave capabilities of human hearing!

13 bits is enough in consumer audio when used optimally. That's why 16 bits is enough in consumer audio.

But human hearing goes from 0 dB (threshold of hearing around 1 kHz) to 120 dB (threshold of pain), so how can 13 bits (78 dB) be enough? Because at any given moment the dynamic range of human hearing is something like 70 dB. You can only hear 0 dB sounds if you have stayed in extreme silence for a while. That's one reason why you can't experience 120 dB dynamic range in music and it is why 13 bits in consumer audio is enough. Vinyl gives 10 bits (60 dB) at best, and many are happy with it. In fact, music with large dynamic variation is annoying to listen to. Also, the background noise level of a quiet listening room is 30 dB and good audio gear can give about 110 dB meaning technically you can have 80 dB of dynamic range at most. So, 13 bits is enough in consumer audio.

 

[hakunamakaka]

Came across the vid on a FB group and this is not the 1st time I saw this kind of posts. I'm neither a subjectivist nor a objectivist but one striving to be objective, rational, logical, pragmatic and realistic. But all this doesn't mean that I'm immune from infection of ignorance so I'm constantly reflecting, and learning. But then the phenomenon that in the name of science, half baked science itself has turned into a perceived antidote of ignorance and self-constructed myth of an ideal world, is in itself quite phenomenal! (Btw this also serves as a warning reminder to myself).

Disclaimer: I don't own nor have used the product in the vid, and its being pick merely serving as a convenient example to deliberate my points.

ASR and its founder AMR has been doing some great stuff but at the same time he's kinda inadvertently spreading "misinformation" that his measurement is a "be all and end all" judgement of a piece of gear, or at least being treated as such by his followers. But is it really so? Is he measuring all things I mean ALL things? Yes, no doubt he's measuring those common, salient aspects of the gears he's dealing with, but if a significant no of ppl in a community said they hear a diff, an audible diff, has he gone out to investigate like through a ABX? And see if the "audible diff" claimed really exists? Yes or No? If yes why?, and he then should go back to his drawing board and look further! If no, the conclusion is more complete! Many of the "measure well" audio equipment manufacturers place huge, equal emphasis on measuring AND listening, and ABX as well before concluding, including Benchmark. The ASR forum has evolved into a cult that AMR's measurement is a holy grail and to his followers it seems that no further discussion tolerated and points to those who've heard diff as fools!, and that's divisive! Is he then doing the audiophile community a blessing or a curse?

Cult based mentality is never healthy for any community. Amir got lot's of followers who blindly believes what he says, but many folks know how bias he is as well. Another issue is that Amir doesn't actually listen to music and simply blasts tracks on very high volumes.

I actually bought L30 amp based on their review and specs just to see that their claims were false. Received lots of negative feedback on how dare I say such things about this product and only later when amp was dissembled and tested by different member not on ASR, designer of an amp admitted that there can be an issue in certain scenarios and especially using a high gain.

Important thing to note is that it's easy to manipulate charts. Measurement in audio is a dynamic outcome which will shift towards one way or the other depending on use.

For my casual listening I'm not that bothered on how and what I use, but when I listen critically on high end gear I actually need to hear stuff to believe it. Seen false claims in subjective or measurements reviews and in some cases was amazed with stuff which I believed being a complete snake oil

 

[sonitus mirus]

Came across the vid on a FB group and this is not the 1st time I saw this kind of posts. I'm neither a subjectivist nor a objectivist but one striving to be objective, rational, logical, pragmatic and realistic. But all this doesn't mean that I'm immune from infection of ignorance so I'm constantly reflecting, and learning. But then the phenomenon that in the name of science, half baked science itself has turned into a perceived antidote of ignorance and self-constructed myth of an ideal world, is in itself quite phenomenal! (Btw this also serves as a warning reminder to myself).

Disclaimer: I don't own nor have used the product in the vid, and its being pick merely serving as a convenient example to deliberate my points.

ASR and its founder AMR has been doing some great stuff but at the same time he's kinda inadvertently spreading "misinformation" that his measurement is a "be all and end all" judgement of a piece of gear, or at least being treated as such by his followers. But is it really so? Is he measuring all things I mean ALL things? Yes, no doubt he's measuring those common, salient aspects of the gears he's dealing with, but if a significant no of ppl in a community said they hear a diff, an audible diff, has he gone out to investigate like through a ABX? And see if the "audible diff" claimed really exists? Yes or No? If yes why?, and he then should go back to his drawing board and look further! If no, the conclusion is more complete! Many of the "measure well" audio equipment manufacturers place huge, equal emphasis on measuring AND listening, and ABX as well before concluding, including Benchmark. The ASR forum has evolved into a cult that AMR's measurement is a holy grail and to his followers it seems that no further discussion tolerated and points to those who've heard diff as fools!, and that's divisive! Is he then doing the audiophile community a blessing or a curse?

When anyone is providing detailed measurements and sharing all the data so that others can repeat their findings, and if the data correlates with their claim that no difference could be heard, that person has no obligation to conduct an ABX test when people claim to hear differences. If the measurements indicate that no audible difference would be likely, it doesn’t matter if a thousand people believe they are hearing a difference. It only takes one of those listeners to perform an ABX and provide the results, procedures, and files for others to validate or repeat the test. That would be a good starting point for a rational discussion into finding the reason for any possible differences being heard, whether from potential faults in the original measurements or from an error with the method used to conduct ABX test itself.

I frequently see illogical arguments pop up to challenge Amir’s testing and what the results indicate with regards to audible transparency, but I’m not seeing too much scientific rigor with these debates.

 

[gregorio]

ASR and its founder AMR has been doing some great stuff but at the same time he's kinda inadvertently spreading "misinformation" that his measurement is a "be all and end all" judgement of a piece of gear, or at least being treated as such by his followers. But is it really so? Is he measuring all things I mean ALL things?

No, he’s not. Somethings, that affect perception, he just gives his subjective opinion but no measurements and somethings he might not mention at all, such as appearance or “brand” name, etc. “Judgement of a piece of gear” is a complex and at least partially subjective process, even if it doesn’t consciously appear so.

Yes, no doubt he's measuring those common, salient aspects of the gears he's dealing with, but if a significant no of ppl in a community said they hear a diff, an audible diff, has he gone out to investigate like through a ABX? And see if the "audible diff" claimed really exists?

A “significant no of ppl” doesn’t mean anything to science, even if it’s virtually everyone. The Earth didn’t stop being the centre of the universe when enough people no longer believed it, the Earth was never the centre of the universe and virtually everyone was mistaken.

Amir obviously hasn’t gone out and investigated (DBT/ABX). Each “review” would take months and cost a lot. However, a great deal of DBTs have been done, over the course of many decades, so we (science) have a very good idea of what’s audible and what isn’t. And, it’s been proven countless times, beyond any doubt, that a very “significant no of ppl” who are certain “they hear a difference” actually can’t/don’t.

Many of the "measure well" audio equipment manufacturers place huge, equal emphasis on measuring AND listening, and ABX as well before concluding, including Benchmark.

That depends on what audio equipment and what their specific design decisions were. No one would waste money performing a DBT/ABX test, testing for something that was obviously inaudible.

The ASR forum has evolved into a cult that AMR's measurement is a holy grail and to his followers it seems that no further discussion tolerated and points to those who've heard diff as fools!, and that's divisive! Is he then doing the audiophile community a blessing or a curse?

A cult is never a good thing. However, a lot of people like being part of a cult and if they're going to join one, much better that it's one based on objective measurements rather than one based on marketing falsehoods and perceptual error/bias!

[1] As we know, a 24-bit system needs 256 times the accuracy of a 16-bit system in order to make full use of the additional 8 bits!, [2] furthermore a bit depth of about 22-bits would begin to exceed the ave capabilities of human hearing!

1. No, it's the other way around. A 24bit system theoretically provides 256 times the "accuracy" of a 16bit system but does not "need" 256 times the "accuracy". The reason I've said "theoretically" is because in practice, there is no system (or even just a DAC) that does or can provide that level of "accuracy". Around 20-22bits is the most that can be achieved in practice. Also note that I've put quotes around your term "accuracy" because that's not really the correct term, a 24bit system theoretically provides 256 times the "precision" but not necessarily any increase in accuracy.

2. As mentioned, 20bits (120dB dynamic range) is the about the max capability of human hearing, assuming that "max" is around the threshold of pain. HOWEVER, listening to music/sound recordings is not a professional competitive sport, it's entertainment, so commercial music/sound recordings do not go anywhere near the threshold of pain. The furthest they go, is to push the boundary of "comfortable" and even then, only very rarely. The vast majority of music recordings have a dynamic range of only about 40dB or less. The most dynamic recordings, such as some large orchestra recordings, have a dynamic range of about 60dB, more than this is virtually unheard of.

But then isn't it the case that those influential KOLs do have some "implied" responsibility to their followers and the communities that they should point out their limitations, as caveats, so as to shy away from any absolutism perceived or real.

What are "KOLs"? Notwithstanding that, why would we want to "shy away from any absolutism"? Who would not want to be absolutist about 1+1=2 or that the Earth isn't flat? The only answer is someone who is insane or who doesn't know the facts. There are many similar cases with audio, the only difference is that we'd expect any adult to know the absolute fact that 1+1=2 but not so many are likely to know that the absolute limit of analogue audio is Johnson/Nyquist (thermal) noise, that 30kHz is inaudible or numerous other absolutes. Absolutism isn't the issue, the issue is people not knowing the facts! This leads them to not being absolutist when the facts dictate they should be or to being absolutist when the facts dictate there is no absolute.

G

 

[gregorio]

I actually bought L30 amp based on their review and specs just to see that their claims were false.

What was false, the subjective opinions/claims or the measurements? If it's the former, that's off-topic, if it's the latter then in what way were the measurements false?

later when amp was dissembled and tested by different member not on ASR, designer of an amp admitted that there can be an issue in certain scenarios and especially using a high gain.

That's not uncommon with amps. With amps, noise and distortion always increase as max gain is approached, so the only question and audible variability between the vast majority of amps is how close to max do we have to get before that increase in noise and distortion becomes an issue? This is the reason why it's always been a "rule of thumb" with amps to never run them higher than about 60-70% of max gain. So, when you say "using high gain", do you mean significantly more than about 60-70% of max gain or lower? If it's higher, then you're probably using the wrong amp for the task, if it's significantly lower then there is/was an issue, under conditions which I presume were not measured. In which case, what was "false" was your incorrect interpretation of the measurements and/or your false assumption that the measurement would still be applicable under different conditions.

Important thing to note is that it's easy to manipulate charts.

Of course they are, in many cases charts would be useless or near useless unless they could be easily manipulated. There's only a problem with that if the chart has been nefariously manipulated, for example if the scale has been removed or it's been deliberately falsified in some way. There can of course be a problem if the reader doesn't understand or mis-reads the chart, although of course that's not a problem with the chart but with the reader of the chart.

Measurement in audio is a dynamic outcome which will shift towards one way or the other depending on use.

No, an objective audio measurement is NOT "a dynamic outcome". Within the limits of the measuring device, the outcome is always the same/repeatable, it does NOT change! However, if you change the conditions of the test then OF COURSE you need a new measurement, which is extremely likely to result in anything from a somewhat to a hugely different measurement. What's the 0-100kph time of a Ferrari? What about if you do that test in a muddy field? ... The test conditions are standardised, for example: a dry, flat road, no/light wind, etc. Outside of those conditions the measurement is not applicable, you'd obviously need a new measurement and also obviously, it's impossible to test every possible set of conditions.

Although it's effectively off-topic (because now we're talking about a reader's interpretation of the measurement/s, rather than the trustworthiness of the measurement itself), if your personal conditions do not match the conditions of the measurement, then although still valid, the measurement/s may not be applicable to your personal use. Isn't this obvious?

[1] ... when I listen critically on high end gear I actually need to hear stuff to believe it. [2] Seen false claims in subjective or measurements reviews and in some cases was amazed with stuff which I believed being a complete snake oil

1. And how do you know what you "actually hear"? For example, how do you separate what you're actually hearing from some perceptual error?
2. That's again off-topic. The thread isn't "why you can't trust claims", it's "why you can't trust measurements" and so far there's been little/no reliable evidence to support the thread's title. What there has been is a lot of effectively off-topic misdirection, such as; why you can't trust people's ability to read or interpret objective measurements, or even to understand the basic rules of objective measurements in general, which should be obvious, such as test/use conditions.

G

 

[bigshot]

Amir refused to answer if he was looping on fade outs and cranking the volume on his listening test when I asked him, but later someone on his own site got him to admit to it. I trust his measurements a lot more than I trust his listening tests.

 

[jjss]

When anyone is providing detailed measurements and sharing all the data so that others can repeat their findings, and if the data correlates with their claim that no difference could be heard, that person has no obligation to conduct an ABX test when people claim to hear differences. If the measurements indicate that no audible difference would be likely, it doesn’t matter if a thousand people believe they are hearing a difference. It only takes one of those listeners to perform an ABX and provide the results, procedures, and files for others to validate or repeat the test. That would be a good starting point for a rational discussion into finding the reason for any possible differences being heard, whether from potential faults in the original measurements or from an error with the method used to conduct ABX test itself.

I frequently see illogical arguments pop up to challenge Amir’s testing and what the results indicate with regards to audible transparency, but I’m not seeing too much scientific rigor with these debates.

So what is science? If there is one data point and one uses this data point to draw a definitive conclusion to me this is not "too scientific". And this is what AMR has been doing. I'm not saying his work is "scientific" vs "not scientific" as his work wasn't 0 or 1. To me he's move a certain distance on his "scientific endevor" but not enough, not yet to definitive to conclude that a product is xxx say. The ABX thing is one way (of course not the only way) as an attempt to establish a more plausible, a more conclusive interpretation of the finding, in this case its the "why we hear what we hear". Again I'm talking this purely in the realm of a scientific endevor, not necessary in the role of a Hi-Fi gear reviewer. In the end its also about specificity (one experiment - in this case AMR's) and generality (concluding that a product is worthwhile or not). That's why you may see some reviewers has attempted putting things in context: meaning attempting to interpret their findings (one illustration is when Bob Katz was reviewing PassLabs HPA-1). Btw On record I'm not challenging or "attacking" what AMR has been doing so save your defence of AMR on his behalf. All I'm saying is that when it comes to science lets differentiate between specificity and generality. I would image that AMR might be better saying that this DAC for example sucks as it measure bad, and if you're a measurement only guy, don't buy it; rather than saying this unit's measurement result sucks , so this unit sucks. As going from measurement (btw his own measurement is one data point only) to saying the unit is problematic involves other considerations that many will incorporate but haven't been spelled out by AMR. Of course in a way I kinda get what AMR is doing (not necessarily by his intention - as I'd never know his intention - but by his result): that he wishes to build an audiophile community / followers that probably only / predominately care about measurements period. And to some degree he'e been doing it quite well.

 

[hakunamakaka]

Perception error is valid when there is little to no change, with significant improvements it is heard no just by me and my imagination. I just share my opinion of what worked/did not on my high end setup, won't gonna try to cut someones throat to defend my point.

L30 failed to deliver advised power output. With 2300mW x 2 @32Ω and about 40% of headroom sound totally collapsed on my Utopia's, while P1U didn't distort at all and played way louder with 2x 0.5 W/32 Ω on specs. L30 was dissembled and measured on a different forum, where it painted different picture than it did on ASR. It's easy to google it

As I said Amir doesn't listen to music, his headphone reviews are useless. The goal is to push cheap and well measured gear in his testing conditions. Of course you will pay premium because Amirs charts will send you into audio nirvana, but in reality you could get same type of gear from Aliexpress and pay just a fraction of the price compared to the products that they are hyping.

Either way in my eyes ASR servers little to no purpose to the community.

 

[bigshot]

The accuracy of the data is rarely the problem, the accuracy of the interpretation of the data is. But without interpretation, it's just numbers on a page. The interpretation is what makes the data useful.

 

[castleofargh]

So what is science? If there is one data point and one uses this data point to draw a definitive conclusion to me this is not "too scientific". And this is what AMR has been doing. I'm not saying his work is "scientific" vs "not scientific" as his work wasn't 0 or 1. To me he's move a certain distance on his "scientific endevor" but not enough, not yet to definitive to conclude that a product is xxx say. The ABX thing is one way (of course not the only way) as an attempt to establish a more plausible, a more conclusive interpretation of the finding, in this case its the "why we hear what we hear". Again I'm talking this purely in the realm of a scientific endevor, not necessary in the role of a Hi-Fi gear reviewer. In the end its also about specificity (one experiment - in this case AMR's) and generality (concluding that a product is worthwhile or not). That's why you may see some reviewers has attempted putting things in context: meaning attempting to interpret their findings (one illustration is when Bob Katz was reviewing PassLabs HPA-1). Btw On record I'm not challenging or "attacking" what AMR has been doing so save your defence of AMR on his behalf. All I'm saying is that when it comes to science lets differentiate between specificity and generality. I would image that AMR might be better saying that this DAC for example sucks as it measure bad, and if you're a measurement only guy, don't buy it; rather than saying this unit's measurement result sucks , so this unit sucks. As going from measurement (btw his own measurement is one data point only) to saying the unit is problematic involves other considerations that many will incorporate but haven't been spelled out by AMR. Of course in a way I kinda get what AMR is doing (not necessarily by his intention - as I'd never know his intention - but by his result): that he wishes to build an audiophile community / followers that probably only / predominately care about measurements period. And to some degree he'e been doing it quite well.

I think his point was more about how the purpose of a blind test is typically to challenge the null hypothesis. We get conclusive results when someone passes the test(proves difference), not so much when someone doesn’t. Leading to the obvious idea that those who think they can hear a difference between gears should be the ones doing controlled tests to substantiate their claims.