Head-Fi.org › Forums › Equipment Forums › Sound Science › A misconception about measurements
New Posts  All Forums:Forum Nav:

A misconception about measurements - Page 3

post #31 of 52
Quote:
Originally Posted by mike1127 View Post

Three points in this post.

 

First, there is a common misconception about measurements. Let me explain that, as audio buffs, we are interested in the behavior of an audio device. Under certain conditions, with a given input signal, a device will behave a certain way. Measurements are an attempt to characterize the device's behavior. Note that any real-world device has complex behavior and is not truly linear-time-invariant. Measurements evaluate its behavior under just a few conditions/inputs. But there's an entire universe of behavior, and measurements just a peephole into that universe, not a complete description of a device's behavior, as some believe. These same people reinforce this misconception by saying that we can measure very very small signals. Well, it really doesn't matter if we can measure microscopic signals, if we are only evaluating a tiny fraction of the possible behaviors.

 

Second, broadly speaking, a device is more accurate if it has less distortion. But the effect of distortion on perception can only be evaluated by listening. Therefore accuracy is subjective.

 

Third: we are often reminded that perception is fallible, subject to illusion. That's true! A lot of stuff can skew your perception of something, including your belief systems. Which leads me to wonder if people who believe that measurements are a good characterization of accuracy, and that device X measures well, end up perceiving device X as accurate entirely due to their belief system.

 

I'm not claiming that anyone has infallible perception. But one cannot get around the fact that accuracy is subjective. The fallibility of perception complicates the matter, but does not and cannot change the basic facts.

 

 


Great!  You explain this very well.  I always read in wonder at attempts being made to look at audio on completely objective terms.
 

.... and yes, this is a very appropriate post for head-fi related sound science.   It's not about fully discussing the science involved, it's limitations and how we may misuse it.

 
Quote:
Originally Posted by mike1127 View Post

Some people claim the concept "accuracy" is inherently empirical (measurable or numerical). But one of the definitions of accuracy is "true to a standard." No one said that standard had to be empirical. I would wager that in English, the word is used more often non-empirically.

Someone might say, "Your description of that birthday party was accurate." Someone describes the events at a party (not using any numbers, mind you), including the reactions of people present, etc. Someone else says: "That's an accurate description."

Or consider a class of art students told to paint a copy of the Mona Lisa. Twenty students stuggle; twenty different images result. The students are given a grade based on the accuracy of their copy.

How is the accuracy evaluated? By measurements? We could scan each image and check the color and brightness differential at each pixel location. That would give us millions of numbers. What would we do with these numbers?

This whole idea is nonsense. The teacher evaluates accuracy by observing each painting, and making a subjective judgment. Not only is accuracy subjective, it may have multiple meanings---she may have several criteria in mind. How close are the shapes and forms? How close are the colors?

One interesting criteria: which student really nailed the expression on her face?

(Try to calculate that with numbers.)

It is entirely possible that a student gets the emotion on her face right, while getting shapes and colors less right than other students.

 

Since art is about the experience of looking at the painting, the accuracy of a copy can only be judged by experiencing it. If someone wants to say: "Look, measure the shapes and colors and find the student who is closest," I would point out that every student deviates from the original more in some places and less in others. How do you quantify each deviation (turn it into a single number)? How do you weight the series of numbers that results? And if you find one student that is generally closer than others, you still have to ask: does that student capture the feeling of the painting? We don't know without looking at the painting and making a subjective judgment. If a student is the best one at copying shapes but doesn't create an artistic experience in the viewer, then I'm not very interested in that student's work. I would say he's on the wrong track.
 


This is an excellent analogy.

 

Additionally, controlled measurements all too often don't represent real world behaviour.  As a serious F1 fan, it's fascinating to read how engineers come up with ways to make the cars faster through the use of their very sophisticated models and measurements.  However, during race conditions, results can be quite variable.  It can be quite the dickens for them to figure out what else is amiss or that they missed.  Quite often, it's the driver who figures out what's wrong through his hopelessly flawed perception of what's happening with the car.  Successful teams learn to take the input of great drivers very seriously.  As listeners, we're like the drivers.  Our input may contradict the measurements, but out on the track or in the listening room, it's a very dynamic situation and that listening session is what really counts because as flawed as our perception may be in some ways, we do have an uncanny way of coming up with an overall impression of what's going on. Afterall, aren't these measuring devices trying to mimic this very ability?

post #32 of 52
Thread Starter 
Quote:
Originally Posted by aimlink View Post


Additionally, controlled measurements all too often don't represent real world behaviour.  As a serious F1 fan, it's fascinating to read how engineers come up with ways to make the cars faster through the use of their very sophisticated models and measurements.  However, during race conditions, results can be quite variable.  It can be quite the dickens for them to figure out what else is amiss or that they missed.  Quite often, it's the driver who figures out what's wrong through his hopelessly flawed perception of what's happening with the car.  Successful teams learn to take the input of great drivers very seriously.  As listeners, we're like the drivers.  Our input may contradict the measurements, but out on the track or in the listening room, it's a very dynamic situation and that listening session is what really counts because as flawed as our perception may be in some ways, we do have an uncanny way of coming up with an overall impression of what's going on. Afterall, aren't these measuring devices trying to mimic this very ability?



Thanks for the F1 analogy, that is interesting.

 

One comment on your statement above about perception contradicting the measurements. I know what you mean, but if you think about it, you can only say that perception contradicts the measurements if you conflate the measurement with perception. For example, if I perceive a Stax headphone as "fast," and an engineer measures the transient response and finds it "slow", those only contradict each other if you have conflated the concepts. It's only an assumption that perceived "speed" has anything to do with "transient response."

post #33 of 52
Thread Starter 

 

 

Quote:

 

We have people that claim Stax are "dramatically faster" than dynamics when they're actually slower in quite a few cases.  Can we trust their perception?  Not really.

 

You're making the mistake of thinking perceived speed is the same concept as measured speed. The perception is the perception; the measurement is the measurement. They don't contradict each other.

 

 

Quote:

 

Quote:

And, what's audible will differ from person to person! Oh no! Accuracy is subjective!

 

Audibility has nothing to do with accuracy in this context (Nick's post).

 

Of course it does. I asked him which of two devices, with two different sets of distortion measurements, is more accurate. He rightly pointed out that depends on the audibility of the distortion.

post #34 of 52

Quote:

Originally Posted by mike1127 View Post


you can only say that perception contradicts the measurements if you conflate the measurement with perception. For example, if I perceive a Stax headphone as "fast," and an engineer measures the transient response and finds it "slow", those only contradict each other if you have conflated the concepts. It's only an assumption that perceived "speed" has anything to do with "transient response."


That's the problem though, perception is often different from the reality.  There's a reason we have standards of measurements.

 

 

Quote:

Of course it does. I asked him which of two devices, with two different sets of distortion measurements, is more accurate. He rightly pointed out that depends on the audibility of the distortion.

 

No, he didn't.  Here's what he wrote:

 

 

Quote from Nick

Without the scales and actual parameters it is difficult to speculate (some distortions are more audible than others) , are you measuring % or levels viz how many dbs the distortion is down on signal.

 

The fact that some distortions are more audible means that depending on the measurement it may be more audible than the other.  Of course there's also the chance that both are below the level of audibility too.

 

Regardless, audibility isn't relevant to accuracy.


Edited by Shike - 9/1/10 at 7:57pm
post #35 of 52
Quote:
Originally Posted by mike1127 View Post

Thanks for the F1 analogy, that is interesting.

 

One comment on your statement above about perception contradicting the measurements. I know what you mean, but if you think about it, you can only say that perception contradicts the measurements if you conflate the measurement with perception. For example, if I perceive a Stax headphone as "fast," and an engineer measures the transient response and finds it "slow", those only contradict each other if you have conflated the concepts. It's only an assumption that perceived "speed" has anything to do with "transient response."


Let me explain further with the analogy.  The car is modified based on simulations, experiments and measurements.  Under race conditions things don't work out as expected.  The driver proposes why.  Some teams actually ignore the drivers in this regard, being more prepared to trust their instruments.  OTOH, there are a few drivers in the paddock whose feedback is very much respected along those lines.  With such an arrangement, the simulators, measurements and calculations provide the 'prototype'.  What remains is based on what actually works with car and driver.   Further modifications or interpretation of the model may change as a result of highly skilled driver input.

 

So for assessing accuracy, a skilled listener can assess in ways that simple parameter measurements cannot.  The listener has the ability to synthesize and listen to all.  What's the effect of all together in not only one way, but in a lot of ways.  That final, overall impression is very difficult to measure in a lab.  It is way too often erroneous to extrapolate that one measurement reflects the whole, or that the measurement correlates even with consistently reproducible perceptual appreciation by the same or among several listeners.

post #36 of 52
Thread Starter 

 

 

Quote:
Originally Posted by Shike View Post
That's the problem though, perception is often different from the reality.

Perception is real. It can't be "different from the reality."

 

Quote:
Regardless, audibility isn't relevant to accuracy.

 

Let me try this again. I put one minute of music signal into two devices, A and B, and measure the input to output differential during that one minute of duration. Sampling at 44.1 KHz, that gives me two lists of 4.7 million numbers. I hand these two lists to you and ask you to tell me which device is more accurate. How do you answer?

post #37 of 52
Quote:
Originally Posted by mike1127 View Post

 

Perception is real. It can't be "different from the reality."


Yes it can.  Remember the example of the optical illusion in the other thread?  You perceive the lines aren't straight, but in fact they are.  Or are you going to argue they're bent because that's how you see them?

 

If perception is always real then there would be no such thing as an illusion.

 

 

Quote:

Let me try this again. I put one minute of music signal into two devices, A and B, and measure the input to output differential during that one minute of duration. Sampling at 44.1 KHz, that gives me two lists of 4.7 million numbers. I hand these two lists to you and ask you to tell me which device is more accurate. How do you answer?

 

You use a null differencing method you will be able to specifically see which has more distortion than another transducer at which frequency.  The ultimate goal is to have the lowest distortion across all frequencies.  The one that has the highest peak between the two is more inaccurate between the two being compared.

 

Generally any good headphone (many cheap ones like the Porta Pro included) offer extremely low THD (1% or lower, though I've seen some ho-hum ones hit 3%) at 90dB.  I'm much more inclined to believe you'll notice differences in dispersion, FR, or possibly decay before THD on most well received headphones here.


Edited by Shike - 9/1/10 at 9:08pm
post #38 of 52

Re:  Stax vs. fast dynamic headphones:

 

I think it's much simpler, precise, and easier to understand if you just go with the mathematical definitions of everything.  My posts are already too imprecise and difficult to understand already without having to deal with further ambiguity!  You could define "perceived speed" as some type of sonic quality that some people (but not others) perceive, and go on and create an alternate universe of parallel definitions for words like "speed" where "higher speed" really means "qualities X and Y but maybe not quite faster" in some contexts, where the word means "faster" in others.  And these may or may not conflate with your perception or mine.  Yet if you had some really slow headphones, almost everyone would be able to agree they are slower.  Having the same word mean multiple things impedes our communication of those concepts.

 

The same goes with accuracy.  If somehow you had a 100% distortion-free empirically accurate sound system, calling it "not accurate" because of one person's perception at a given time sounds silly.  This recent audibility relating to accuracy example (the difference between A and the source; and B and the source) highlights how different people using different definitions of accuracy.  If you want to determine the actual accuracy to the original, you don't need to worry about perceptions, audibility, etc.  You could specify some measures you're interested in and run the statistical calculations on both global and local scales.  Out pop out some numbers, and they have a precise definition that others can look at and compare.  But if you want to know how they sound differently (also "accuracy"...?), you would want the context of the original source.  If you want to know about how this difference is perceived, doing listening tests with a lot of people would not be a bad idea to supplement the statistics derived.

 

A perception is a real occurrence in the sense that it happened, but please just call it wrong if it's wrong.  I think we all agree that the perception is real in that sense and not real in the sense that it perfectly describes what exists in the real world.


Edited by mikeaj - 9/1/10 at 9:30pm
post #39 of 52
Thread Starter 

 

 

Quote:
Originally Posted by Shike View Post


Yes it can.  Remember the example of the optical illusion in the other thread?  You perceive the lines aren't straight, but in fact they are.  Or are you going to argue they're bent because that's how you see them?

 

If perception is always real then there would be no such thing as an illusion.

 

To me this is a different issue. You are talking about attempting to correlate perception to the external world. For example, trying to judge whether the lines are straight by eye.

 

That's the same mistake as thinking that a subjective description of a headphone as "fast" is the same thing as the measured transient response.

 

 

Quote:

You use a null differencing method you will be able to specifically see which has more distortion than another transducer at which frequency.  The ultimate goal is to have the lowest distortion across all frequencies.  The one that has the highest peak between the two is more inaccurate between the two being compared.

 

Generally any good headphone (many cheap ones like the Porta Pro included) offer extremely low THD (1% or lower, though I've seen some ho-hum ones hit 3%) at 90dB.  I'm much more inclined to believe you'll notice differences in dispersion, FR, or possibly decay before THD on most well received headphones here.

 

What I'm trying to get you to do is give me a completely empirical, numerical, and universally complete way of judging which of two devices is more accurate. What you have written is not even close. It looks like you are guessing at things.

 

Let's try it this way. Consider these two plots of FR and THD vs frequency for two different transducers. Tell me which one, green or red, is more accurate.

 

 

which_more_accurate.jpg

 

post #40 of 52

Is there a point in continuing to go back and forth without first agreeing on the meaning of the words in question?  Or, in the very least, understanding what the other person is referring to?  Usually I'm not that big on semantics, but this is getting ridiculous.  (That is, unless I've grossly misunderstood what is going on.)

post #41 of 52
Thread Starter 
Quote:
Originally Posted by mikeaj View Post

Is there a point in continuing to go back and forth without first agreeing on the meaning of the words in question?  Or, in the very least, understanding what the other person is referring to?  Usually I'm not that big on semantics, but this is getting ridiculous.  (That is, unless I've grossly misunderstood what is going on.)

 

To me it's not just semantics. It's more about epistemology.. that is, what do we consider valid knowledge, and how do we know what we think we know? Some people here claim that accuracy can be defined with no reference to perception. Then, when device A is declared to be "less accurate" than device B, and someone comes along and says "Device A is truer to the live reference," that someone is told, "No, you prefer a distortion." I'm pointing out this is tautological reasoning. That is, it's not knowledge at all, and it has no meaning. It does not advance our knowledge about audio.

 

In general, on this forum is a lot of equivocation between accuracy in a "measurement sense" and accuracy in a "perceived sense." Or worse, it is denied entirely that accuracy has any meaning in the context of perception.

 

Optical illusions are presented to us to "prove" that perception is unreliable, when the real analogy from these illusions is not even relevant to perceptual accuracy.

 

And so forth.

 

There's just a lot of invalid reasoning, lack of critical thinking, and strange statements that are irrelevant to audio reproduction. And a lot of these logical mistakes result from the way people define and use language. So yes, semantics is relevant.

 

post #42 of 52
Quote:
Originally Posted by mike1127 View Post



 

To me it's not just semantics. It's more about epistemology.. that is, what do we consider valid knowledge, and how do we know what we think we know? Some people here claim that accuracy can be defined with no reference to perception. Then, when device A is declared to be "less accurate" than device B, and someone comes along and says "Device A is truer to the live reference," that someone is told, "No, you prefer a distortion." I'm pointing out this is tautological reasoning. That is, it's not knowledge at all, and it has no meaning. It does not advance our knowledge about audio.

 

In general, on this forum is a lot of equivocation between accuracy in a "measurement sense" and accuracy in a "perceived sense." Or worse, it is denied entirely that accuracy has any meaning in the context of perception.

 

Optical illusions are presented to us to "prove" that perception is unreliable, when the real analogy from these illusions is not even relevant to perceptual accuracy.

 

And so forth.

 

There's just a lot of invalid reasoning, lack of critical thinking, and strange statements that are irrelevant to audio reproduction. And a lot of these logical mistakes result from the way people define and use language. So yes, semantics is relevant.

 


Wow, nothing I have said or read on the issue has ever been versed so clearly and eloquently.  If you don't mind it's going in my file as QFT.   

post #43 of 52

Very interesting discussion. Except for this post, I will remain a reader.

 

I do find the wish to have an accurate reproduction of the CD quite sympathetic. I love reading and I want to read the most well know works of Dostoyevsky. However, wanting to read his own words and not an approximation of what he has written, this means that I have to learn Russian. Also it is best to search out the early printings since later ones might remove or even change parts due to certain ideological currents depending on when they were printed.

H*ll of a lot of work just to read some books, but some people might find it rewarding. It does however not make the experience more fulfilling in an "objective" sense.

 

I agree with Anaxilus, very clear and well written, mike1127.


Edited by Danneq - 9/3/10 at 1:52am
post #44 of 52
Quote:
Originally Posted by mike1127 View Post

To me it's not just semantics. It's more about epistemology.. that is, what do we consider valid knowledge, and how do we know what we think we know? Some people here claim that accuracy can be defined with no reference to perception. 

 

You've hit the nail on the head here.
 

post #45 of 52
Quote:

Originally Posted by mike1127 View Post

 

There's just a lot of invalid reasoning, lack of critical thinking, and strange statements that are irrelevant to audio reproduction. And a lot of these logical mistakes result from the way people define and use language. So yes, semantics is relevant.

 

 

I don't disagree. But I think that semantics can be dealt with if listenings tests are well written. If the listener writes how the test took place, there is already much information. And if something is not completely clear, the fact that this is a forum allows to ask.
 

New Posts  All Forums:Forum Nav:
  Return Home
  Back to Forum: Sound Science
Head-Fi.org › Forums › Equipment Forums › Sound Science › A misconception about measurements