Why you can't trust audio measurements

[Giru]

No, what I actually said is that objective measurements, fortunately, don’t include minute to minute variations in perception driven by numerous reasons. And that the reason people frequently report differences between electronics that measure the same or have differences below audible level is that human variability.

The objective measurements are repeatable under the same conditions. People are not capable this. Not sure why this concept is so hard to grasp - it explains most of the audiophile “findings” - their human condition has changed (unless using a controlled testing methodology over short duration) and they can’t/won’t account for it

I implore you to kindly read all my posts carefully. It seems you're not reading but going on and on about defending the sanctity of objective measurements.

Nobody is questioning the validity of the accuracy of these measurements. I'm just pointing out that these measurements are not enough or are not ideal descriptors of sound quality and therefore have little real world value when ultimately the device that you'll be utilising to use these devices that you test: YOUR EARS; are subject to human "perception". How hard is this to understand?

I'm not promoting that subjective evaluations be the benchmark for quantifying how good the device performs either. I'm merely suggesting that since there is so much of variability in human perception and objective data is not of much value in real world application; the consumer is best served by listening and deciding for themselves.

You do realize that you just made my point, don’t you? Unless you believe the output of electronics changes with the moods/health/time of day of the listener.

Humans are variable. Measurements are consistent.

The suggestion that their aren’t universally good or bad electronics misses the point entirely. Again, the issue is with individual perception, not our ability to produce transparent electronics. Unless you believe the mood of the listener alters the output of electronic…

On the contrary I think you may have made mine. Especially when you say:

Humans are variable. Measurements are consistent.

If humans are indeed variable and our perceptions different, do these measurements matter??
Especially in regards to being the absolute criterion when making purchase decisions??

You're going around in circles buddy; trying to defend these objective measurements as the last word, while in reality most of these tests translate very poorly to human observations. These tests are only indicators of how good the device is performing electrically; not how good it will sound. The reason is that these tests are not perfect descriptors of sound quality; we can't quantify things like detail/resolution, timbre, dynamics, soundstage, etc with these tests: all of which are actual "real-world" perceivable differences and account for why two devices don't sound the same (and by extension of course to two different individuals).

 

[bfreedma]

I implore you to kindly read all my posts carefully. It seems you're not reading but going on and on about defending the sanctity of objective measurements.

Nobody is questioning the validity of the accuracy of these measurements. I'm just pointing out that these measurements are not enough or are not ideal descriptors of sound quality and therefore have little real world value when ultimately the device that you'll be utilising to use these devices that you test: YOUR EARS; are subject to human "perception". How hard is this to understand?

I'm not promoting that subjective evaluations be the benchmark for quantifying how good the device performs either. I'm merely suggesting that since there is so much of variability in human perception and objective data is not of much value in real world application; the consumer is best served by listening and deciding for themselves.


On the contrary I think you may have made mine. Especially when you say:



If humans are indeed variable and our perceptions different, do these measurements matter??
Especially in regards to being the absolute criterion when making purchase decisions??

You're going around in circles buddy; trying to defend these objective measurements as the last word, while in reality most of these tests translate very poorly to human observations. These tests are only indicators of how good the device is performing electrically; not how good it will sound. The reason is that these tests are not perfect descriptors of sound quality; we can't quantify things like detail/resolution, timbre, dynamics, soundstage, etc with these tests: all of which are actual "real-world" perceivable differences and account for why two devices don't sound the same (and by extension of course to two different individuals).

You seem to be missing the point. I’m not “defending the sanctity of objective measurements”, I’m stating that the human auditory system is too variable on a moment to moment basis to generate reliable analysis for any individual, and certainly not a reliable method for generating recommendation.

I think you may want to update your knowledge regarding current measurement capabilities and tools. Most of what you list absolutely can be measured. They’re components of the recording, not generated or altered by electronics unless we’re now discussing DSP.

But I’ll play along. Please post some examples of DACs and Amps that measure the same but when compared in a properly constructed blind test result in statistically significant differences being identified while operating within spec.

BTW, you’re brain is subject to perception, not your ears. And since this perception changes by the minute due to various physiological and psychological input, your brain is far less reliable than measurements. Today, if I eat well and have a low stress workday, a system might sound much better than that same system after a day of eating poorly, a highly stressful workday, and sinus issues due to allergies.

 

[bigshot]

I think all of us agree that measurements don't register subjective bias, the effect of fluctuations in mood on how we perceive music, or natural human perceptual error. All they measure is the qualities of sound. Even if we could accurately measure subjective perception, I don't know how we could alter an amp or DAC to cater to it.

It's a lot easier to focus your energy on achieving fidelity, and then color the sound with volume and tone controls on the fly to accommodate your subjective feelings. That's what those knobs on the box are for!

 

[71 dB]

Please post some examples of DACs and Amps that measure the same but when compared in a properly constructed blind test result in statistically significant differences being identified while operating within spec.

There are situations where measurements can be problematic. For example headphone amps with high output impedance. The measurement might have been done connecting the headphone output to a high impedance (say 47 kΩ) input of the measuring instrument while in subjective listening the impedance of the load (headphones) can be not only much smaller (say nominally 32 Ω), but also strongly frequency dependent. The measurements are not wrong. They just do not represent real life listening situations. Comparing two headphone amps with 1 Ω and 20 Ω output impedances using 32 Ω headphones is kind of rubbish. Such tests tell more about the impedance mismatch than the headphone amps compared.

 

[bigshot]

Impedance isn’t an issue with the DAC, it’s with mismatched headphones. It’s a given that you need to pair things that pair properly.

 

[PROblemdetected]

After kz issues on the latest month, famous youtubers being exposed, and dunu confirming that est drivers dont appear on measurements...

Dont worry, I dont trust u, mate.

 

[gregorio]

I think human variance should not be ignored, but at the same time humans are not as varied as some might think.

Effectively, that’s off topic. The topic is audio measurements, which are measurements of audio ONLY and have nothing to do with humans, neither the variance between them nor anything else about humans. So human variance absolutely SHOULD be ignored.

If we’re going to talk about humans, that’s an entirely different subject and we obviously cannot use audio measurements to quantify humans, human perception or the variance between humans. We would obviously need to test humans, not the devices they are listening to, and we would obviously need entirely different measurements, psychoacoustic measurements instead of audio measurements.

It's not about inter-subject variance per say, as much as it is about the translation of most objective measurements to humans.

As above, it’s not about “subjects” at all. Objective audio measurements are measurements of audio, they are not measurements of anything to do with human subjects, including human perception or how humans translate/interpret (or apparently in this case misinterpret) objective measurements.

This is what I'm trying to emphasize here: we know how to measure a device's technical "performance" (quite accurately infact) but do these metrics "always" translate to the real world performance while listening to music?

Of course they do, how is that not obvious? What sort of performance do you think say a DAC or amp can possibly have other than “technical” performance? You think maybe they use magic or have a human brain and use human perception to make judgements about who is listening and change their output accordingly? Of course not, that would be ridiculous! Therefore, the ONLY thing a device can do is output a signal (real world performance) according to it’s “metrics”, that’s it, nothing more, REGARDLESS of who or even if anyone is listening to it, or whether that signal represents music or any other sort of sound.

How a listener perceives and responds to that “real world performance” (after it’s converted to sound) is an entirely different question. That is obviously a question of the listener’s ears/brain and therefore a question of psychoacoustics and psychoacoustic measurements (where they exist), NOT a question of audio measurements!!

What I'm trying to add is that there are so many variations in human perception (dependant on a lot of factors) that these tests are moot.

And that is exactly why your assertions are false! Why are you “trying to add” human perception to audio measurements of devices that do not have any human perception? When we’re objectively measuring the performance of a device, then we’re obviously measuring the performance of that device, we are NOT measuring the performance of whoever is listening to that device!

Moreover as I've been asserting in the last few posts: human perception (and variations therein) make things far more complicated

And for the same reason, this assertion is also false!

Personal pot-shots at me ain't gonna bring anything productive to the debate buddy.

Then why are you the one who started with the “personal pot-shots”?

These tests are only indicators of how good the device is performing electrically; not how good it will sound.

Duh of course! A DAC or amp for example ONLY perform electrically, they don’t produce any sound. So what other objective measurement of their performance do you suggest, miles per gallon or calories maybe?

[1] The reason is that these tests are not perfect descriptors of sound quality; [2] we can't quantify things like detail/resolution, timbre, dynamics, soundstage, etc with these tests: [3] all of which are actual "real-world" perceivable differences …

1. Again, duh! A frequency response measurement for example, is obviously just a measurement of frequency response, it’s not a measurement of anything else (such as sound quality) and a jitter measurement is only a measurement of jitter, etc. How is this not obvious?

2. So in regard to my previous question, you are effectively claiming that digital audio recordings never have any resolution, timbre or dynamics! It’s only because we CAN quantify these things that digital audio exists!

3. No! Firstly, they’re only perceivable differences if they are of sufficient magnitude to fall within the realm of audibility. Secondly, “soundstage” is an exception to your list. It’s not an audio property, it’s effectively an aural illusion and would therefore require a measurement of perception, a psychoacoustic measurement, not an audio measurement.

It’s shocking that some people obviously don’t seem to understand the difference between the objective measured performance of something and the human perception/experience of it, despite actually posting videos demonstrating that difference! Shocking though not entirely surprising, as that’s exactly what audiophile snake oil salesman have been misrepresenting for many years.

G

 

[bfreedma]

There are situations where measurements can be problematic. For example headphone amps with high output impedance. The measurement might have been done connecting the headphone output to a high impedance (say 47 kΩ) input of the measuring instrument while in subjective listening the impedance of the load (headphones) can be not only much smaller (say nominally 32 Ω), but also strongly frequency dependent. The measurements are not wrong. They just do not represent real life listening situations. Comparing two headphone amps with 1 Ω and 20 Ω output impedances using 32 Ω headphones is kind of rubbish. Such tests tell more about the impedance mismatch than the headphone amps compared.

Agreed and exactly why I added “operating within spec” to my post.

I know you and the regular posters here don’t need that clarification but suspect others do. It’s certainly possible to use headphones mismatched to an amp, but that’s user error, not a problem with hardware.

 

[Giru]

You seem to be missing the point. I’m not “defending the sanctity of objective measurements”, I’m stating that the human auditory system is too variable on a moment to moment basis to generate reliable analysis for any individual, and certainly not a reliable method for generating recommendation.

Here we go again.
Brother, I'm advocating not recommending anything at all. Neither these objective tests and/or subjective evaluations (in the form that they currently exist) are solid enough to generate an absolute certainty that a person will/will not like a product. I'm just recommending that people trust their own ears; no matter how good/bad the product is in these "tests" (objective or subjective).

I think you may want to update your knowledge regarding current measurement capabilities and tools. Most of what you list absolutely can be measured. They’re components of the recording, not generated or altered by electronics unless we’re now discussing DSP.

Okay, let me entertain the idea that I'm a complete n00b and have no idea how AET is done. Will you provide links and detail these "measurement capabilities and tools" that you speak of which can assess all descriptors of sound fidelity (soundstage, timbre, etc.). I'm sure you'll add something of value to this discussion apart from making unsubstantiated claims and pot-shots.

BTW, you’re brain is subject to perception, not your ears.

The ear is just a sensory organ that sends signals to the brain to be perceived; ergo it is subject to perception. Not the other way round.

All they measure is the qualities of sound.

Only some and under ideal conditions. Hence my contention that these "Tests" give only a report on how a device performs electrically. Minimally indicating how it may sound. Not at all indicating how it will be perceived (as I've been asserting and you rightly agreed).

Even if we could accurately measure subjective perception, I don't know how we could alter an amp or DAC to cater to it.

It is actually measured using blind testing on trained/untrained subjects and the results are used to "tune" the "device" according to a preferred subjective pattern. This is a part of R&D and manufacturing process at most quality audio equipment manufacturers.
However we are discussing perception at the end user and how a variance here makes "hobbyist" audio equipment testing using lab test equipment moot and useless; a point constantly being ignored by fellow forum members here, getting off-topic and beating the same dead horse over and over.

It's a lot easier to focus your energy on achieving fidelity, and then color the sound with volume and tone controls on the fly to accommodate your subjective feelings. That's what those knobs on the box are for!

The idea of "achieving fidelity" and then "color the sound with volume and tone controls" is paradoxical. You want a device that measures perfectly and then introduce all sorts of deviations by applying "tone controls"? What gives man?

 

[SoundAndMotion]

Okay, let me entertain the idea that I'm a complete n00b and have no idea how AET is done. Will you provide links and detail these "measurement capabilities and tools" that you speak of which can assess all descriptors of sound fidelity (soundstage, timbre, etc.). I'm sure you'll add something of value to this discussion apart from making unsubstantiated claims and pot-shots.

Try this. I think there's a 2017 edition, but this (2005) is the newest I found.
Audio Measurement Handbook

 

[Giru]

Try this. I think there's a 2017 edition, but this (2005) is the newest I found.
Audio Measurement Handbook

Thanks for the link but buddy the words "timbre", "soundstage" etc are absent from the manual. It seems you just went on the intenet and in your hurry to appear half-smart, regurgitated. My original request:

Will you provide links and detail these "measurement capabilities and tools" that you speak of which can assess all descriptors of sound fidelity (soundstage, timbre, etc.).

 

[castleofargh]

TBH if people were truly all about soundstage, timbre, etc they wouldn't bother wondering which DAC or amp to reject and why. They would long ago have rejected headphones as a playback solution.

 

[SoundAndMotion]

Thanks for the link but buddy the words "timbre", "soundstage" etc are absent from the manual. It seems you just went on the intenet and in your hurry to appear half-smart, regurgitated. My original request:

Well buddy, you know nothing about me, except that I tried to help you with a first step. I'm perfectly capable of helping you much further,but you lack a lot of foundation, e.g. definitions of the terms you use. But we won't be going in that direction.

I do know quite a bit about you, buddy. You have a huge, unwarranted arrogance, you are aggressively rude, and you lack an understanding about critical parts of each step from recorded medium to brain. Your blade, buddy, is in urgent need of grinding.

 

[71 dB]

Agreed and exactly why I added “operating within spec” to my post.

I know you and the regular posters here don’t need that clarification but suspect others do. It’s certainly possible to use headphones mismatched to an amp, but that’s user error, not a problem with hardware.

Unfortunately it is a "user error" easily made while the awareness to avoid it takes quite a lot of knowledge into electronic circuits. What's even worse is that often the "specs" aren't even given by the manufacturer! I have had to measure headphone output impedances myself to know what they are. Doesn't help "operating within spec."

 

[Giru]

TBH if people were truly all about soundstage, timbre, etc they wouldn't bother wondering which DAC or amp to reject and why. They would long ago have rejected headphones as a playback solution.

Nice. Why contribute when you can cancel

Well buddy, you know nothing about me, except that I tried to help you with a first step. I'm perfectly capable of helping you much further,but you lack a lot of foundation, e.g. definitions of the terms you use. But we won't be going in that direction.

I do know quite a bit about you, buddy. You have a huge, unwarranted arrogance, you are aggressively rude, and you lack an understanding about critical parts of each step from recorded medium to brain. Your blade, buddy, is in urgent need of grinding.

Yes. Let it all out. I'm sure you have more to regurgitate.

Gents I think we're heading in the wrong direction. Trolling and pedestrian squabble is adding nothing to this discussion.

Again: Just trying to point out that traditional benchmarking tests are of little value to prospective buyers coz they don't completely reflect all attributes of sound quality. Moreover, perceptional variances are far more predominant in the end making such objective tests of little value in describing what will sound "good" to the end user.

About 40 posts have passed by and not one cogent argument has been made instead the thread is being derailed again and again by personal attacks, unsubstantiated claims and the regular subjective vs objective argle-bargle.

I pleade fellow members to abstain from rhetorical dialogue, cynicism, crude commentary and unwarranted insults. Elitism and a gatekeeper mentality will definitely not win you any argument.