Three points in this post.
First, there is a common misconception about measurements. Let me explain that, as audio buffs, we are interested in the behavior of an audio device. Under certain conditions, with a given input signal, a device will behave a certain way. Measurements are an attempt to characterize the device's behavior. Note that any real-world device has complex behavior and is not truly linear-time-invariant. Measurements evaluate its behavior under just a few conditions/inputs. But there's an entire universe of behavior, and measurements just a peephole into that universe, not a complete description of a device's behavior, as some believe. These same people reinforce this misconception by saying that we can measure very very small signals. Well, it really doesn't matter if we can measure microscopic signals, if we are only evaluating a tiny fraction of the possible behaviors.
Second, broadly speaking, a device is more accurate if it has less distortion. But the effect of distortion on perception can only be evaluated by listening. Therefore accuracy is subjective.
Third: we are often reminded that perception is fallible, subject to illusion. That's true! A lot of stuff can skew your perception of something, including your belief systems. Which leads me to wonder if people who believe that measurements are a good characterization of accuracy, and that device X measures well, end up perceiving device X as accurate entirely due to their belief system.
I'm not claiming that anyone has infallible perception. But one cannot get around the fact that accuracy is subjective. The fallibility of perception complicates the matter, but does not and cannot change the basic facts.
Great! You explain this very well. I always read in wonder at attempts being made to look at audio on completely objective terms.
.... and yes, this is a very appropriate post for head-fi related sound science. It's not about fully discussing the science involved, it's limitations and how we may misuse it.
Some people claim the concept "accuracy" is inherently empirical (measurable or numerical). But one of the definitions of accuracy is "true to a standard." No one said that standard had to be empirical. I would wager that in English, the word is used more often non-empirically.
Someone might say, "Your description of that birthday party was accurate." Someone describes the events at a party (not using any numbers, mind you), including the reactions of people present, etc. Someone else says: "That's an accurate description."
Or consider a class of art students told to paint a copy of the Mona Lisa. Twenty students stuggle; twenty different images result. The students are given a grade based on the accuracy of their copy.
How is the accuracy evaluated? By measurements? We could scan each image and check the color and brightness differential at each pixel location. That would give us millions of numbers. What would we do with these numbers?
This whole idea is nonsense. The teacher evaluates accuracy by observing each painting, and making a subjective judgment. Not only is accuracy subjective, it may have multiple meanings---she may have several criteria in mind. How close are the shapes and forms? How close are the colors?
One interesting criteria: which student really nailed the expression on her face?
(Try to calculate that with numbers.)
It is entirely possible that a student gets the emotion on her face right, while getting shapes and colors less right than other students.
Since art is about the experience of looking at the painting, the accuracy of a copy can only be judged by experiencing it. If someone wants to say: "Look, measure the shapes and colors and find the student who is closest," I would point out that every student deviates from the original more in some places and less in others. How do you quantify each deviation (turn it into a single number)? How do you weight the series of numbers that results? And if you find one student that is generally closer than others, you still have to ask: does that student capture the feeling of the painting? We don't know without looking at the painting and making a subjective judgment. If a student is the best one at copying shapes but doesn't create an artistic experience in the viewer, then I'm not very interested in that student's work. I would say he's on the wrong track.
This is an excellent analogy.
Additionally, controlled measurements all too often don't represent real world behaviour. As a serious F1 fan, it's fascinating to read how engineers come up with ways to make the cars faster through the use of their very sophisticated models and measurements. However, during race conditions, results can be quite variable. It can be quite the dickens for them to figure out what else is amiss or that they missed. Quite often, it's the driver who figures out what's wrong through his hopelessly flawed perception of what's happening with the car. Successful teams learn to take the input of great drivers very seriously. As listeners, we're like the drivers. Our input may contradict the measurements, but out on the track or in the listening room, it's a very dynamic situation and that listening session is what really counts because as flawed as our perception may be in some ways, we do have an uncanny way of coming up with an overall impression of what's going on. Afterall, aren't these measuring devices trying to mimic this very ability?
