Head-Fi.org › Forums › Equipment Forums › Sound Science › A misconception about measurements
New Posts  All Forums:Forum Nav:

A misconception about measurements - Page 2

post #16 of 52
Thread Starter 

Some people claim the concept "accuracy" is inherently empirical (measurable or numerical). But one of the definitions of accuracy is "true to a standard." No one said that standard had to be empirical. I would wager that in English, the word is used more often non-empirically.

Someone might say, "Your description of that birthday party was accurate." Someone describes the events at a party (not using any numbers, mind you), including the reactions of people present, etc. Someone else says: "That's an accurate description."

Or consider a class of art students told to paint a copy of the Mona Lisa. Twenty students stuggle; twenty different images result. The students are given a grade based on the accuracy of their copy.

How is the accuracy evaluated? By measurements? We could scan each image and check the color and brightness differential at each pixel location. That would give us millions of numbers. What would we do with these numbers?

This whole idea is nonsense. The teacher evaluates accuracy by observing each painting, and making a subjective judgment. Not only is accuracy subjective, it may have multiple meanings---she may have several criteria in mind. How close are the shapes and forms? How close are the colors?

One interesting criteria: which student really nailed the expression on her face?

(Try to calculate that with numbers.)

It is entirely possible that a student gets the emotion on her face right, while getting shapes and colors less right than other students.

 

Since art is about the experience of looking at the painting, the accuracy of a copy can only be judged by experiencing it. If someone wants to say: "Look, measure the shapes and colors and find the student who is closest," I would point out that every student deviates from the original more in some places and less in others. How do you quantify each deviation (turn it into a single number)? How do you weight the series of numbers that results? And if you find one student that is generally closer than others, you still have to ask: does that student capture the feeling of the painting? We don't know without looking at the painting and making a subjective judgment. If a student is the best one at copying shapes but doesn't create an artistic experience in the viewer, then I'm not very interested in that student's work. I would say he's on the wrong track.
 


Edited by mike1127 - 8/30/10 at 9:36pm
post #17 of 52
Thread Starter 

 

Quote:
Originally Posted by Pio2001 View Post

As a conclusion, I would say that measurements can completely describe the behaviour of an audio device, but that measuring completely an audio device is sometimes far from easy.

Let me put it this way. If you have completely described the behavior of a device, then you can predict the output for any input. I mean exactly, not an approximation of the output.

 

 

Quote:

According to Ethan Winer [1], audio measurements can be divided into four categories :

 

-Frequency response

-Distortion - THD, IMD, aliasing "birdies"

-Noise - hiss, hum & buzz, vinyl crackles

-Time-Based errors - wow, flutter, jitter

 

These are all models. This doesn't change the fact that models are only approximations of reality. On top of that, the essence of the problem is how a distortion is experienced. We have only rough correlations of measurements to experience. Many measurements don't correlate to experience at all.

 

 

Quote:

However, there is one measurement that encompass all of them : signal cancellation. Take the output and substract it from the input. What's left is the difference between the two.

 

 

This seems like the most useless measurement of all, because it gives you an infinity of numbers. What do you do with those numbers? As soon as you try to collapse them into a meaningful single number, then you've used a model, which is not reality.
 

 

 

Quote:

Sometimes, the new model is based on completely different ideas than the old. That's what is called a paradigm change. But it never contradicts the old model.

 

The old Newtonian gravitation model said that all massive objects attract each others. As a result, on earth's surface, objects fall down with an acceleration of 9.8 m/s2.

The new general relativity says that mass and energy curve space-time. As a result, on earth's surface, objects fall down with an acceleration of 9.8 m/s2.

 

 

New observations definitely contradict old models. I think what you are trying to say is that new models don't contradict old observations. An object falls at a certain rate of acceleration: that was measured long ago. Adopting general relativity doesn't make that measurement invalid.

 

However, certain phenomena, like the progression of Mercury's orbit, contradict Newton's model.

 

And the fact that science continually makes observations that contradict old models should be sufficient proof for anyone that models are not reality.

 

 


Quote:

 

By the way, Mike, some time ago, you started in-depht long-term blind listening tests. What was the outcome ? Are you still working on this ?

 

I got as far as doing 8 or so trials in an experimental protocol, but the protocol was probably insensitive. I didn't prove any cable differences. After that, it was impossible to find anyone that cared enough to help me do the tests.

 

The practical result of my experiments was this: I prioritize spending more on electronics and less on cables than I did before, because I can't say for sure that cables make a difference.

post #18 of 52

Mike,

 

This may be hard for you to swallow, but I said this once and I will say this again:  this is sound science not sound impressions.  Here the most common use of accuracy will most certainly be empirical to reproduction.

 

 

Quote:

Let me put it this way. If you have completely described the behavior of a device, then you can predict the output for any input. I mean exactly, not an approximation of the output.

 

There's always going to be a margin of error, it's a matter of degrees though.  In same cases extremely small differences can lead to drastic results, other times not.  In this case the margin of error is generally quite low which gives us a picture that is easily predictable.  Especially with cables.

 

Quote:

These are all models. This doesn't change the fact that models are only approximations of reality. On top of that, the essence of the problem is how a distortion is experienced. We have only rough correlations of measurements to experience. Many measurements don't correlate to experience at all.

 

 

We don't need to correlate to experience at this point, and that's your misconception.  As for them being models - let's say they are.  The tests done using these models are usually substantially harder to pass than using music.  Square waves are notorious for showing overshoot or ringing in drivers that you may not experience at all with music.  It's pushing the equipment to find its inherent limits under an extreme condition which wouldn't occur in reality for a reason.  It's a form of benchmarking so to speak.

 

When engineers test the limits of a cars engines do they drive it or do they push it as hard as they can in a lab?  The lab.  If it dies at a certain speed that will never be achieved on a road, does it make their test invalid?  No.

 

 

Quote:

This seems like the most useless measurement of all, because it gives you an infinity of numbers. What do you do with those numbers? As soon as you try to collapse them into a meaningful single number, then you've used a model, which is not reality.

 

We're not looking for a single number in that method - it's looking for multiple differences in pre and post waveforms.  We can average it out yes, but that'd be pretty silly and ruin the point overall.

 

 

Quote:

And the fact that science continually makes observations that contradict old models should be sufficient proof for anyone that models are not reality.

 

Of course it's not reality, if it was you could never control variables.  That's really the point - to remove outside influence as much as possible to eventually make a theory or law based on the outcome of an observation.  It's also a good thing that old models can be disproved - it shows progress.

 

As it stands now though, we have no reason to consider the current models disproved.  Without evidence contrary to the way we currently know things work then it becomes an exercise in futility.  You'll find many here would happily accept there being differences in cables if it could be proven with some method beyond strictly anecdotal difference, but it hasn't happened yet as there is no empirical evidence.

post #19 of 52
Thread Starter 

 

Quote:

Originally Posted by Shike View Post

 

this is sound science not sound impressions.  

 

 

A sub-forum of a forum called Head-fi, which is about music. There are lots of uses for sound other than music. Let's say as a weapon to break windows by vibrating them violently. A forum called "Sound Weapons" would then be a good place to discuss, say, the pitch accuracy of a transducer as related to its ability to nail the resonance frequency of a window. An entirely empirical definition of accuracy would be appropriate in that forum.

 

 

Quote:

 

Here the most common use of accuracy will most certainly be empirical to reproduction.

 

Those who use a notion of accuracy stripped of the experience of music are on the wrong forum, then. This is a forum about people who pay a lot of money for equipment that lets them enjoy music.

 

 

Quote:
As it stands now though, we have no reason to consider the current models disproved.

What models are you talking about, and what observed phenomena?

post #20 of 52

First off, interesting read, thanks, Anaxilus.
 

Quote:
Originally Posted by Pio2001 View Post


Is IMD with three frequencies more than the sum of IMD for each pair of frequencies, and if so, does it make a significant difference, or is it just 0.1 % of 0.1 %, and IMD with four frequencies 0.1 % of 0.1 % of 0.1 % ?

 

Transient IMD asks the same question. If you measure IMD on a fixed frequency sine plus a transient burst, will you get more than the total IMD for all frequencies contained in the burst associated with your sine ? If so, is it significantly more, or just 0.1 % more ?

 

I don't know.

 

And just in case you couldn't tell, I was being facetious with the 4D, 5D, etc., but I think it got the point across.  In general for a linear system, if you have input frequencies A, B, C, and D, the output will only have frequencies A, B, C, and D (may have different amplitude and phase, but frequencies do not change).  For a nonlinear system, intermodulation results in output frequencies at different sums and differences of the inputs.  Sums and differences of A, B, C, and D as seen in an IMD graph for those four inputs would be would include IMD frequencies seen whenever any pair of A, B, C, and D (e.g. A with D only) are input as well as others from triple and quadruple combinations.  The distortion from A, B, C, and D should be at least the distortion with A and B plus the distortion with C and D.  I'm not quite sure what the order of magnitude of the distortion in these cases would be, though.

 

For one example, ryumatsuba has IMD graphs with test tones of 100, 1000, and 10000 Hz.  http://www.geocities.jp/ryumatsuba/review.html.  Here is the graph for the AKG K101.

k101_ds.gif

 

Note that the IMD graphs from test tone measurements will include other harmonic distortion too.  What's being measured is all frequencies, which include the test tones, IMD, harmonic distortion in multiples of the test tones, harmonic distortion in multiples of the resultant intermod products, other distortion, noise, etc.  For characterizing IMD in general, the Volterra series model is used and the parameters estimated, but I couldn't tell you much more than that about the process.  There are different numerical methods to solve a problem like that, and I don't even know if people use this tool in audio.  About transient IMD and the rest, I'm not sure either.  I'm leaving it out there as an open question.

 

 

Quote:

Originally Posted by mike1127 View Post

Some people claim the concept "accuracy" is inherently empirical (measurable or numerical). But one of the definitions of accuracy is "true to a standard." No one said that standard had to be empirical. I would wager that in English, the word is used more often non-empirically.

I think "accuracy" in the sound science forum when discussing measurements would imply the numerical definition by default, but maybe that's just me.  I would use "perceived accuracy" or "subjective accuracy" for the other definition.  The Mona Lisa example explained what you really meant, thanks.

 

 

Quote:

Originally Posted by mike1127 View Post

Since art is about the experience of looking at the painting, the accuracy of a copy can only be judged by experiencing it. If someone wants to say: "Look, measure the shapes and colors and find the student who is closest," I would point out that every student deviates from the original more in some places and less in others. How do you quantify each deviation (turn it into a single number)? How do you weight the series of numbers that results? And if you find one student that is generally closer than others, you still have to ask: does that student capture the feeling of the painting? We don't know without looking at the painting and making a subjective judgment. If a student is the best one at copying shapes but doesn't create an artistic experience in the viewer, then I'm not very interested in that student's work. I would say he's on the wrong track.


On a side note, I must mention that a lot of people in DSP, video/image transmission, etc. would point out that they do scan the image and "turn it into a number(s)."  There are many algorithms for this that a lot of people work with.  Applications include robot vision, detection of objects from photographs, and the like.  However, I do agree that from an artistic point of view, the "best" (which is subjective, though experts may be able to agree on this) portrayal may not have the highest "closeness" score from whatever algorithm you apply.  You could say that the algorithms are not measuring the right thing.

 

Often in evaluating image compression, coding, transmission, and the like, researchers use both empirical measures from algorithms as well as human responses to "rate how good picture B is compared to picture A" or "is there a difference between A and B", or something like that.  Both are important if the end image or audio signal is meant to be processed by humans.  Machine algorithms are more useful at determining absolute differences, especially when differences are smaller.  If the goal is for the end product to be processed by a human, you want to know what differences can be discerned by humans.  And this is determined through the usual blinded testing.  Well, blind in the sense of eliminating biases wherever possible, not that you test people blindfolded when asking them to compare images. 

 

 

Quote:
Originally Posted by mike1127 View Post

This seems like the most useless measurement of all, because it gives you an infinity of numbers. What do you do with those numbers? As soon as you try to collapse them into a meaningful single number, then you've used a model, which is not reality.

 

Cancellation is most useful when the difference is zero, or at least close to it:  hence canceling.  Then you don't need to do anything with those numbers.  If you record the output using interconnect X and record it again with interconnect Y and find that the difference between the two recordings is zero, there was no difference between the two interconnects (that could be picked up by the measuring equipment, which is more empirically accurate than human hearing).

 

 

 

Let me wrap up this post by reiterating that the models used are close enough to reality for our purposes in audio.  Particularly, they are great for interconnect cables, which are very linear.  I think one thing you're trying to say is that perceived accuracy in music reproduction is more important than empirical accuracy.  e.g. a higher THD score from a system that only produces second harmonics is better than a lower THD score from a system that produces all sorts of unrelated frequencies -- this being the extreme case.  However, the most obvious and relevant measures can be related to experience.  You just need to interpret the measurement data differently in a known way to conclude what might have been determined subjectively.  Even if you focus on what is perceived as best to you (not a bad choice), I think that a reasonably comprehensive set of measurements would give a better understanding of the system than you give it credit for.


Edited by mikeaj - 8/30/10 at 11:12pm
post #21 of 52

Quote:

Originally Posted by mike1127 View Post

A sub-forum of a forum called Head-fi, which is about music. There are lots of uses for sound other than music. Let's say as a weapon to break windows by vibrating them violently. A forum called "Sound Weapons" would then be a good place to discuss, say, the pitch accuracy of a transducer as related to its ability to nail the resonance frequency of a window. An entirely empirical definition of accuracy would be appropriate in that forum.

 


Or maybe we rather reproduce music without distortion, time based errors, etc.  Thus an empirical definition makes perfect sense.  Taking us out of context to further a pseudo-philosophical point isn't going to impress or change the way things are discussed here because they displease you.

 

Quote:

Those who use a notion of accuracy stripped of the experience of music are on the wrong forum, then. This is a forum about people who pay a lot of money for equipment that lets them enjoy music.

 

 

This is a sub-forum for those that want to discuss what actual differences exist in equipment are and the phenomena that can impact it for the purpose of listening to music.  If we want transducers that are empirically accurate so we're sure our music is being reproduced accurately (according to the signal, obviously) then there's no problem and it's not contradictory to the goal of head-fi in the least.

 

Why this is hard for you to understand I don't know.  Some of us don't judge our music by how we "think" it should sound, but want it to be presented as intended (an assumption that the final product is indeed what was intended).  For this empirical accuracy in absolutely necessary in regards to discussion of equipment.  If music sounds bad when empirically accurate, then to me the master or music is just plain bad and I move on.  I don't play around ultimately trying to fix a bad product, but that's me.

 

Quote:

What models are you talking about, and what observed phenomena?

 

 

The ones brought up by Pio that you quoted, basically ones currently related to audio and measures of accuracy in terms of empirical reproduction.

post #22 of 52
Thread Starter 


 

 

 

 

Quote:
Originally Posted by Shike View Post

Or maybe we rather reproduce music without distortion, time based errors, etc. 

 

There is no such thing as reproduced music without distortion.

 

Quote:

Quote:

What models are you talking about, and what observed phenomena?

 

 

The ones brought up by Pio that you quoted, basically ones currently related to audio and measures of accuracy in terms of empirical reproduction.

 

You have to be more specific about the observed phenomena.

post #23 of 52

Quote:

Originally Posted by mike1127 View Post

 

There is no such thing as reproduced music without distortion.

 

No, but there's nothing wrong with wanting to get as close to it as possible now is there?

 

Quote:

You have to be more specific about the observed phenomena.


Really beyond the point I was making, but for example audibility of noise (SNR).  We can observe generally where noise becomes inaudible when there's a signal.  I was referring to the models more than anything else, the phenomena is just was related in reality to the model.

post #24 of 52
Thread Starter 
Quote:
Originally Posted by Shike View Post

Quote:

Originally Posted by mike1127 View Post

 

There is no such thing as reproduced music without distortion.

 

No, but there's nothing wrong with wanting to get as close to it as possible now is there?

.

My point is this: if you had a perfect recorder/playback system, then it seems reasonable to say it is accurate. But because no audio system is perfect (the mics and speakers being the most easily measurable non-perfections), then "getting as close as possible" becomes a matter of listening, perhaps supplemented with some measurements, but with listening as primary.

 

Suppose you have two devices, X and Y, and you measure distortion parameters A, B, and C. X gives values of 0.1, 1.2, and 2.1 respectively. Y gives values of 0.2, 1.1, and 1.5. Which is more accurate? There's no way to answer that without a model of the perceptual effects of A, B, and C. That requires listening. And because your model is fallible, you'd better just give a listen to X and Y anyway.



 

post #25 of 52
Quote:
Originally Posted by mike1127 View Post

 

Suppose you have two devices, X and Y, and you measure distortion parameters A, B, and C. X gives values of 0.1, 1.2, and 2.1 respectively. Y gives values of 0.2, 1.1, and 1.5. Which is more accurate? There's no way to answer that without a model of the perceptual effects of A, B, and C. That requires listening. And because your model is fallible, you'd better just give a listen to X and Y anyway.



 



Without the scales and actual parameters it is difficult to speculate (some distortions are more audible than others) , are you measuring % or levels viz how many dbs the distortion is down on signal.

 

In fact unlesss the differences were astounding (orders of magnitude on one parameter) you would probably not give a unified answer you would say X is better on A and worse on B and so on.

 

Based on research you can however with decent confidence state whether distortion at such and such a level is likely to be audible.

 

Someone listening to X and Y and stating that X is more accurate is not helpful, as it is just one opinion and even if everyone who listens to them says that X is more accurate that still does not help. What reference point do the listeners have to make that judgment, perhaps they are just unconsciously expressing a preference, not the same thing, but very human. Since Hi Fi originally meant high fidelity (accuracy) we can easily be fooled into thinking our preference is higher fidelity, witness the fact that many folks think vinyl is more accurate despite it being objectively rather less accurate on most measured parameters.

 

Let us say everyone is shown a visual illusion that two lines are different lengths (arrows and crows-feet) and everyone perceives this difference the same, the lines are not different lengths our perception is just plain wrong. No amount of convergenece of opinion overcomes this fact.

 

Rather than asking which is more accurate it is more interesting to ask are the differences audible, in DBT of course. 

 

If they are then you need to ask why. Then you control variables until you ascertain which variable is most likely responsible for the audible differences. So for instance you can add selectively distortions so that X and Y measure the same on your 3 parameters. Confirm that they are audibly identical with this tweak. Then alter one variable to see the point at which they become different again. return to equal and repeat for each variable then you see how each variable influences audible differences.


Edited by nick_charles - 9/1/10 at 5:29am
post #26 of 52

Quote:

Originally Posted by mike1127 View Post



My point is this: if you had a perfect recorder/playback system, then it seems reasonable to say it is accurate. But because no audio system is perfect (the mics and speakers being the most easily measurable non-perfections), then "getting as close as possible" becomes a matter of listening, perhaps supplemented with some measurements, but with listening as primary.


We aren't going by your definition of accurate - do you actually understand this?  We want accurate to the signal as defined by the source material.  What happened in mastering is of no concern when our objective is to reproduce the material accurate to the source (CD for example).

 

Most "accurate" by subjective listening doesn't tell us anything as there's no fixed scale for comparison and will vary from person to person based on their experience.

post #27 of 52

I don't see the point in arguing wether a system is absolutely perfect or not absolutely perfect. This has been done recently in a french forum, and it lasted for about 100 pages. There was not a single relevant information in them. Just a few people arguing about the possible meanings of "absolute", "relative", "objective" and "subjective". Everyone standing with his own exact meaning.

 

More interesting are frequency response measurments of interconnects vs listening position in a room.

 

Here are the frequency responses of some interconnects, substracted from a standard 2.30 € interconnect, that would be completely flat. Mind the vertical scale : -1 dB to +1 dB.

 

RG179-Std.png

 

Taralabs-Std.png

 

Audioquest-Std.png

 

Standard-6metres-Std.png

 

The last one is the worst, see the 0.02 dB roll-off at 20 kHz ? That's a 5 meters standard extension cable for interconnects.

 

Now, let's look at some speakers measurements. The room is a professional auditorium made according to IEC 268-13 (1985) specifications. The speaker is the same for all curves. Model unknown. The measurements of the first set of curves are done from 4 different listening positions, 1 meter apart, in a 4.1 x 6.7 meters room.

 

fr.jpg

 

Taken from Sound Reproduction - Loudspeakers and Rooms, by Floyd E.Toole, Focal Press. (horizontal scale is 20 Hz to 20 kHz)

 

The second set of curves are the same speaker measured from one of the four previous locations, but with the speaker positionned at three slightly different locations in front of the listeners.

 

The main difference between the top and bottom curves is that the bottom curves are smoothed with a 1/4 octave resolution, while the top curves are more accurate. The smoothing is a 20th of octave only.

 

And this representation is still far from the reality. In fact, there are hundreds of peaks and dips on the right side of the graph.

 

Have you seen the vertical scale ? Moving your seat one meter away can change the frequency response more than 10 dB.


Edited by Pio2001 - 9/1/10 at 12:44pm
post #28 of 52
Thread Starter 
Quote:
Originally Posted by nick_charles View Post





Without the scales and actual parameters it is difficult to speculate (some distortions are more audible than others) ...

 

 

Rather than asking which is more accurate it is more interesting to ask are the differences audible, in DBT of course. 

 

If they are then you need to ask why. Then you control variables until you ascertain which variable is most likely responsible for the audible differences. So for instance you can add selectively distortions so that X and Y measure the same on your 3 parameters. Confirm that they are audibly identical with this tweak. Then alter one variable to see the point at which they become different again. return to equal and repeat for each variable then you see how each variable influences audible differences.

My point exactly. You can't talk about accuracy without talking about the human experience; i.e., what's audible.

 

And, what's audible will differ from person to person! Oh no! Accuracy is subjective!

 

Your idea for an experiment isn't as objective as you think it is. You will get a different result for each person.

 

Nor is it even feasible. You can't arbitrarily change distortion levels on an arbitrary transducer or amplifier. You are thinking of some kind of signal generator. That doesn't help me evaluate two speaker designs, A and B.



 

post #29 of 52
Thread Starter 

 

Quote:
Originally Posted by Shike View Post

Quote:

Originally Posted by mike1127 View Post



My point is this: if you had a perfect recorder/playback system, then it seems reasonable to say it is accurate. But because no audio system is perfect (the mics and speakers being the most easily measurable non-perfections), then "getting as close as possible" becomes a matter of listening, perhaps supplemented with some measurements, but with listening as primary.


We aren't going by your definition of accurate - do you actually understand this?  We want accurate to the signal as defined by the source material. 

Of course I understand. I'm just saying your definition of "accurate" is silly.

 

And I agree that sometimes a live reference is not available. But your devices still distort the signal as recorded on the master tape, and you are stuck with developing some way to relate those distortions to perception. You are stuck with the fact that your measurements are a model and can never completely represent reality.

 

Quote:
Most "accurate" by subjective listening doesn't tell us anything as there's no fixed scale for comparison and will vary from person to person based on their experience.

It's complicated. But being complicated doesn't make it wrong.

post #30 of 52

Quote:

Originally Posted by mike1127 View Post

 

Of course I understand. I'm just saying your definition of "accurate" is silly.

 


Really?  Unfortunately AES and any engineering community would probably disagree with you.

 

 

Quote:

And I agree that sometimes a live reference is not available. But your devices still distort the signal as recorded on the master tape, and you are stuck with developing some way to relate those distortions to perception.

 

So what?  Does a band adjust their performance to "your perception"?  Are they going to equalize their presentation to your ears?  No.

 

We have people that claim Stax are "dramatically faster" than dynamics when they're actually slower in quite a few cases.  Can we trust their perception?  Not really.

 

 

Quote:

You are stuck with the fact that your measurements are a model and can never completely represent reality.

 

And you're stuck that your opinion will never represent fact on an empirical scale making them ultimately useless to anyone but you or those that follow your opinion.  Which has more weight in this sub-forum?

 

Your opinion of accurate does not make it accurate.  In mathematical terms we need evidence and numbers which is around this sub-forum is built upon.  If you don't like it why do you post here?

 

Quote:

It's complicated. But being complicated doesn't make it wrong.

 

No, since it's subjective (your opinion) it can never be proven as fact either though - empirical evidence and claims based upon them can be proven as wrong or right.

 

 

As for this line:

 

 

Quote:

And, what's audible will differ from person to person! Oh no! Accuracy is subjective!

 

Audibility has nothing to do with accuracy in this context (Nick's post).


Edited by Shike - 9/1/10 at 6:35pm
New Posts  All Forums:Forum Nav:
  Return Home
  Back to Forum: Sound Science
Head-Fi.org › Forums › Equipment Forums › Sound Science › A misconception about measurements