Objectivist/subjectivist is to me very baffling. Much of my work over the last 20 years involves building systems that respond usefully to human typed or spoken inputs. We use lots of metrics, but there's a big gap between what we can measure and how the systems perform in the field, because of the long-tailed distributions and constant drift of human behavior. Still, measurements are critical in getting us to the right ballpark. I can't imagine that the situation is different for audio gear. Designers need measurements to avoid egregious mistakes, but once the gear is feeding customer ears, many uncontrollable factors take hold, from expectations to listening conditions to shifts source material mix over time to aging of the listener.
I know the discussion has moved on since Jason's inflammatory common-sense and outrageous level-headedness, but this is a point worth highlighting...
If you take a step back, take a cold shower and pour yourself a hard-liquor treat of your choosing, you may realize that "measurements" are remarkably limited in scope.
From the off, the term "measurement" is somewhat of a misnomer in the context and has a halo effect of suggesting that it's really easy to "measure" things: ye know, like take a ruler and, well,
measure things. (Jason rightly points out that
"Getting consistent measurements isn’t as easy as you might think." in his
Measurements chapter, yet another outrageously balanced and considered piece of prose. How dare he!)
While I did no such tests and my understanding is largely theoretical, I believe that common measurements in audio are remarkably complex beasts, some depending on quite a bit of stats (weighting schemes, calibration, etc.). From what I see "measurements" are more like (stress-)
testing an audio device, by blasting a (simplistic and uncharacteristic of real-life usage) sine wave through it and (yes!)
measure how the system responds.
In other parts of the engineering world this is known as laboratory
simulation, which will not always yield similar results in real-world conditions (i.e. sonic performance as perceived by human listening). We can take an analogy from the F1 world, where teams will stress test their designs using 60% scale prototypes in a windtunnel, and then cross fingers REAL REAL hard that the lab measurements and estimations
correlate with real-life on-track performance. (Some lose by the millions and play years of catch up when they don't.)
This is why, when queried "when will specs become available for the latest and greatest Schiit gizmo?", Jason
often suggests that published specs will largely be uncorrelated with sonic performance...
And as any "estimator"---is there a statistician in the room?---, it's often an imperfect quantifier of a phenomenon (say, distortion), and there will be various estimators / estimating schemes that could be used. You could take something that tries to bundle it all in one measure (e.g. THD), or look at bits and pieces (e.g. distortion of 2nd, 3rd order harmonics, or THD graphs, etc.). This makes headline numbers without book-long methodological explanations meaningless.
Put differently, estimations often may not mean what people think they do, and the only people who could put any sense into them is those who actually did the measuring (and fingers crossed, actually knew what they were doing and understand what they mean!).
And remarkably few relevant measurements seem to actually be published on DACs and amps... As this RMAF presentation on
"What The Specs Don’t Tell You… And Why" rightly points out, single D/A chips may come with a full booklet of specs and graphs, whereas an infinitely more complex full-blown DAC often comes with something like half a dozen published measurements...
What's more, in my understanding most audio measurements will usually indicate what's wrong with a given gear (cf "avoid egregious mistakes"), but have little to say about what's right. So THD, IMD, and even Frequency response which indicates
deviations from flat FR, or SNR which highlights noisy gear.
This can also be thought of as "diseases". If we take the human body as an analogy, doctors can never --- well, they always do, but that's a discussion for another day --- tell someone "you're perfectly healthy" (i.e. the device is "audibly transparent"). The most a doctor can say is that "given the symptoms you describe, and all the tests we've performed, and given the state of science today and our limited understanding of it, we cannot see anything wrong with you, but this doesn't really mean that you're necessarily in good health".
Take cancer: if you want to know whether someone is afflicted by cancer, you need to run several tens or hundreds tests on them. If one comes back positive, there is a good chance the person has cancer. If none do, that's lack of scientific evidence for cancer, but not scientific evidence for lack of cancer. So doctors don't know, i.e. they're in the black. The person may be having cancer in very early stages, so not detected by usual tests (that's why e.g. HIV testing is recommended at least 6 months after suspected infection). Or the person may have a very rare form of cancer that wasn't tested for; or a variation of a known type of cancer but which evades the usual tests; or even that as of yet unknown form of toe-nail cancer. Same holds for virus infections, bacterial infections and the like. And same holds for doping in sports: witness Lance Armstrong who has never officially tested positive for performance-enhancing drugs, yet has admitted to being guilty of it and stripped of all Tour de France achievements.
In other words, to know if someone is "perfectly healthy" one needs to run ALL known tests and ALL unknown tests, for all known and unknown diseases. That's a tall order.
And for a device being certifiably "audibly transparent", audio measurements don't get anywhere near that. (Jason nicely highlights in his
“Improving on ‘Perfection’” chapter how even a device that "is" "perfect" can be improved upon...)
And this is ignoring the fact that most audio measurements usually under discussion generally aren't 'dynamic' tests that would reflect actual real-life music, but instead are 'static' exercises using simplistic (if, helpfully, calibrated) sine waves --- as they stand, and until the advent and wide adoption of dynamic tests
and complex pattern-analysis algorithms accounting for the non-linearities of the Human Auditory System, audio measurements can be thought of as... err... primitive. And yes, the human ear exhibits non-linearities (cf
Critical band), whereas measurements are in effect linear and fall short of this level of complexity... (Metrum
discusses this some in the context of FL NOS R2R tech.)
So while it's obvious that measurements are an incredibly useful tool and
indispensable for building high-quality gear (again, needed to "avoid egregious mistakes"), it is HIGHLY unlikely that they can possibly be---in their current form and application---the final arbiter in sonic performance and the teller of absolute (or even conditional) truth in things audio. After all, we appreciate a Rembrandt by looking at it with our own eyes, and not by inspecting a color histogram or an IR representation... Similarly, we don't appreciate wine by looking at the chemical breakdown of the thing.
Sure, those tools could hold clues and highlight some patterns, but that's just about as far as it gets. Until we develop algorithms complex and intelligent enough to
shame an art connoisseur (e.g. by definitively settling attribution disputes or correcting them on style similarities), the audio world can only rely on human skills for assessing ultimate sonic performance: ear-assisted, brain-performed pattern analysis...