Head-Fi.org › Forums › Equipment Forums › Sound Science › A misconception about measurements
New Posts  All Forums:Forum Nav:

A misconception about measurements

post #1 of 52
Thread Starter 

Three points in this post.

 

First, there is a common misconception about measurements. Let me explain that, as audio buffs, we are interested in the behavior of an audio device. Under certain conditions, with a given input signal, a device will behave a certain way. Measurements are an attempt to characterize the device's behavior. Note that any real-world device has complex behavior and is not truly linear-time-invariant. Measurements evaluate its behavior under just a few conditions/inputs. But there's an entire universe of behavior, and measurements just a peephole into that universe, not a complete description of a device's behavior, as some believe. These same people reinforce this misconception by saying that we can measure very very small signals. Well, it really doesn't matter if we can measure microscopic signals, if we are only evaluating a tiny fraction of the possible behaviors.

 

Second, broadly speaking, a device is more accurate if it has less distortion. But the effect of distortion on perception can only be evaluated by listening. Therefore accuracy is subjective.

 

Third: we are often reminded that perception is fallible, subject to illusion. That's true! A lot of stuff can skew your perception of something, including your belief systems. Which leads me to wonder if people who believe that measurements are a good characterization of accuracy, and that device X measures well, end up perceiving device X as accurate entirely due to their belief system.

 

I'm not claiming that anyone has infallible perception. But one cannot get around the fact that accuracy is subjective. The fallibility of perception complicates the matter, but does not and cannot change the basic facts.

 

 

post #2 of 52

Quote:

Originally Posted by mike1127 View Post

Three points in this post.

 

First, there is a common misconception about measurements. Let me explain that, as audio buffs, we are interested in the behavior of an audio device. Under certain conditions, with a given input signal, a device will behave a certain way. Measurements are an attempt to characterize the device's behavior. Note that any real-world device has complex behavior and is not truly linear-time-invariant. Measurements evaluate its behavior under just a few conditions/inputs. But there's an entire universe of behavior, and measurements just a peephole into that universe, not a complete description of a device's behavior, as some believe. These same people reinforce this misconception by saying that we can measure very very small signals. Well, it really doesn't matter if we can measure microscopic signals, if we are only evaluating a tiny fraction of the possible behaviors.

 

Second, broadly speaking, a device is more accurate if it has less distortion. But the effect of distortion on perception can only be evaluated by listening. Therefore accuracy is subjective.

 

Third: we are often reminded that perception is fallible, subject to illusion. That's true! A lot of stuff can skew your perception of something, including your belief systems. Which leads me to wonder if people who believe that measurements are a good characterization of accuracy, and that device X measures well, end up perceiving device X as accurate entirely due to their belief system.

 

I'm not claiming that anyone has infallible perception. But one cannot get around the fact that accuracy is subjective. The fallibility of perception complicates the matter, but does not and cannot change the basic facts.

 

 

 

This is sound science, you're going to have to cite what we're measuring wrong.  We're not trying to gauge individual perception, just whether something makes a measurable difference most of the time.  No one can tell you how you perceive distortion as it will vary from person to person, just to what level it exists and to what level of accuracy equipment is following the signal.  I suggest you look at the mathematical definition of accuracy, because I doubt other definitions are used here often.

post #3 of 52
Thread Starter 
Quote:
Originally Posted by Shike View Post

 

This is sound science, you're going to have to cite what we're measuring wrong.  

 

No, it's the other way around. A measurement is meaningless until someone can show its relation to perception.

 

We're not trying to gauge individual perception, just whether something makes a measurable difference most of the time.

 

If that's all you're doing, then it's irrelevant to whether a device reproduces music accurately.

 

 

 

I suggest you look at the mathematical definition of accuracy, because I doubt other definitions are used here often.

 

I'm not sure what you mean; there's no "mathematical definition" of accuracy devoid of context. Part of my point is that definitions "used here often" aren't very relevant.

 

 



 

post #4 of 52

Quote:

Originally Posted by mike1127 View Post

No, it's the other way around. A measurement is meaningless until someone can show its relation to perception.

 


We already have our own perceptions, and perceptions will vary as they are based mostly on opinion.

 

 

Quote:

If that's all you're doing, then it's irrelevant to whether a device reproduces music accurately.

 

 

If we want to listen to music (a signal) accurately then it's kind of important.

 

 

 

Quote:

I'm not sure what you mean; there's no "mathematical definition" of accuracy devoid of context. Part of my point is that definitions "used here often" aren't very relevant.

 

 

Accuracy refers to precision that's quantifiable in a mathematical context.

post #5 of 52
I'm not sure what could possibly be going on that cannot be measured or quantified. There are three electrical qualities that define cables: resistance, inductance and capacitance. They interact with each other in elegant, simple equations. Not only that, but they've been backed up with nearly 150 years of applied and theoretical practice.

If you're saying that three feet of wire possess qualities that have gone unnoticed in tens of millions of experiments, real-world use and theory, well, that's remarkable.

Whatever this fourth force is would have to have no effect whatsoever on known values, which is also remarkable. Possible, yes, but it would completely rewrite what's known. That has happened before, but it is difficult to believe that it is possible.

To me, the claims are like getting a phone call from a telemarketer promising me guaranteed returns of 30% a year on an investment with no risk or downside. Possible, yes, but it throws up every red flag and leads me to start asking a lot of questions.
post #6 of 52

@mike1127

 

Great post!

 

@Uncle Erik

 

Great post as well.  However, it deviates from the topic at hand by restricting itself to cables.  It is a great response to the notion of accuracy and the collection of evidence for sure.  But, w/ respect to the OP, it turns his argument into a bit of a straw man because how an earphone sounds to us is a far more complex system to examine.  I will never be of the school that any measured data is useless.  I do prefer it to be taken in context w/ an appropriate level of scientific humility.  

post #7 of 52
Thread Starter 

Hey Shike, as an aside, how do you split the divs? This newer forum system is driving me crazy with the difficulty in editing quote blocks. It used to be so simple.

post #8 of 52

I think the point being made is this:

(1) headphones/speakers are not perfectly linear

(2) measurements are very precise and accurate at what they are measuring, but because of (1) only actually give "perfect" information about what they are measuring -- extrapolation works mostly but not absolutely

(3) there are many different kinds of audio signals (i.e. many possible inputs), so they are not all described perfectly by a given finite set of measurements performed

 

e.g. put in an input of a 60 Hz + 10000 Hz signal and see if there is any measurable IMD on the output.  Now you know the IMD very precisely given those inputs.  Now, can you predict exactly what will happen if you do 125 Hz + 8000 Hz + 9000 Hz?  (Not really.)  Granted, the original measurement probably gives a decent idea of the behavior of the device.

 

 

As for the other part about accuracy, I am confused.  Accuracy is closeness to the actual or intended value.  As a serious question, what else do you think it means?  Or are you talking about perceived accuracy?


Edited by mikeaj - 8/29/10 at 11:20pm
post #9 of 52
Thread Starter 
Quote:
Originally Posted by Uncle Erik View Post

I'm not sure what could possibly be going on that cannot be measured or quantified. There are three electrical qualities that define cables: resistance, inductance and capacitance. They interact with each other in elegant, simple equations. Not only that, but they've been backed up with nearly 150 years of applied and theoretical practice.
 



I actually answered this in my first paragraph.

 

R, L and C are models.

 

Let's assume a cable is linear-time-invariant (LTI). It's not; no real system is. But even if it were, R, L, and C would be an insufficient model at higher frequencies where the cable begins to act as a transmission line.

 

Please don't respond that "higher frequencies don't matter to audio." That's not the point. The point is that you invoked a model which is not complete, which demonstrates my very point that people tend to conflate their models with reality.

 

Now we realize that a cable is not LTI and the "R, L and C" model gets further from reality. Simply put, a cable is a universe of behaviors.

 

The history of science demonstrates that models are not reality--this happens every time evidence is discovered that contradicts an established model. This has happened through scientific history and there's no reason to think it will stop now.

 

So it's not that I'm invoking a mysterious "fourth force"; it's that I'm pointing out our models (and measurements, which are a model) are not complete descriptions of reality.

 

A lot of people understand that models aren't reality, but different people respond to this in different ways. When I listen and hear something that isn't explained by the models (I'm not just talking about cables) I find it reasonable to conclude the models are insufficient. Other people find other explanations.

post #10 of 52
Thread Starter 
Quote:
Originally Posted by mikeaj View Post

I think the point being made is this:

(1) headphones/speakers are not perfectly linear

(2) measurements are very precise and accurate at what they are measuring, but because of (1) only actually give perfect information about what they are measuring

(3) there are many different kinds of audio signals (i.e. many possible inputs), so they are not all described perfectly by a given finite set of measurements performed

 

e.g. put in an input of a 60 Hz + 10000 Hz signal and see if there is any measurable IMD on the output.  Now you know the IMD very precisely given those inputs.  Now, can you predict exactly what will happen if you do 125 Hz + 8000 Hz + 9000 Hz?  (Not really.)  Granted, the original measurement probably gives a decent idea of the behavior of the device.

 

 

As for the other part about accuracy, I am confused.  Accuracy is closeness to the actual or intended value.  As a serious question, what else do you think it means?  Or are you talking about perceived accuracy?



Yeah, thanks for putting it that way. A little simpler than what I wrote. I tend to get abstract.

 

I love the experience of music. Audio exists to reproduce the experience of music. In many contexts, the reproduction can be compared to the original.

 

Comparison can be done very concretely during a recording session in a single acoustic space---step into the space and listen to the music, then step into the control room and check how it sounds similar or different.

 

Accuracy is then how much the experience in the control room resembles the experience in the concert hall.

 

A lot of people complain at this point that experiences are nebulous, hard to define and remember, etc. Well, not to a musician. They spend a lot of time creating and studying musical effects. Someone who has spent little time doing this can't relate to what a professional musician does.

 

People also complain that different people experience different things----yup, accuracy is subjective!

post #11 of 52
Quote:
Originally Posted by mike1127 View Post

Yeah, thanks for putting it that way. A little simpler than what I wrote. I tend to get abstract.

 

I love the experience of music. Audio exists to reproduce the experience of music. In many contexts, the reproduction can be compared to the original.

 

Comparison can be done very concretely during a recording session in a single acoustic space---step into the space and listen to the music, then step into the control room and check how it sounds similar or different.

 

Accuracy is then how much the experience in the control room resembles the experience in the concert hall.

 

A lot of people complain at this point that experiences are nebulous, hard to define and remember, etc. Well, not to a musician. They spend a lot of time creating and studying musical effects. Someone who has spent little time doing this can't relate to what a professional musician does.

 

People also complain that different people experience different things----yup, accuracy is subjective!


I think I see what you're saying now, thanks.  You're talking about perceived accuracy, and in the example, the relative difference between two audio experiences.  First there is listening in the room and then there is the recording.  But here we get into issues of recording/microphone placement/mastering and the fact that the sound actually going into your ears is dependent on exactly where you're sitting in the auditorium or other space.  However, those recording issues are not relevant to measuring playback devices.  The goal of audio (playback) may be to reproduce an experience, but the goal of audio equipment is to reproduce what the input is.  The input is what's on the recording--in a CD, already stored in bits.

 

As a pretty decent student musician (no, it wasn't my major--EE is) that knows many musicians, I doubt that aural experiences are so easy to remember exactly, even for a musician.  e.g. please hum an A 440 within 1 cent without any reference.**  Or if humming is too hard, could you pick out which one was the correct 440 Hz if 438.5, 439, 439.5, 440, 440.5, etc. were all played and you had no reference other than past memories?  It's something you've heard countless times before, right?  I'm sure there are many other details that are forgotten or misremembered slightly.  If you're listening for certain details as opposed to others, what you remember will change.  The way you listen impacts the perception of the experience.

 

**Actually, some people can do this probably, but definitely not all professional musicians, even good ones.

 

edit: I now just skimmed through the last couple pages of the cable thread and am now confused again by what you mean.  I also didn't realize that some of what I stated about accuracy was a rehash of others' statements.


Edited by mikeaj - 8/30/10 at 12:06am
post #12 of 52

 

 

 

Quote:

Originally Posted by mike1127 View Post

 

Measurements evaluate its behavior under just a few conditions/inputs. But there's an entire universe of behavior, and measurements just a peephole into that universe, not a complete description of a device's behavior, as some believe.


The problem is not that measurement are incomplete. They are complete. The problem is that, willingly or nor, no one performs the complete measurements.

 

According to Ethan Winer [1], audio measurements can be divided into four categories :

 

-Frequency response

-Distortion - THD, IMD, aliasing "birdies"

-Noise - hiss, hum & buzz, vinyl crackles

-Time-Based errors - wow, flutter, jitter

 

Let's add that phase response can be included into frequency response, and that reverberation fits in the time-based errors.

To be complete, measurements should include THD at all frequencies (2D plot) (for a speaker, it depends strongly on the frequency), and IMD at all pairs of frequencies (3D plot).

 

Adding on top of that that for amplifiers and speakers, measurements should be done at various levels, this is not very convenient.

 

However, there is one measurement that encompass all of them : signal cancellation. Take the output and substract it from the input. What's left is the difference between the two.

 

[1] http://www.youtube.com/watch?v=BYTlN6wjcvQ


 

Quote:
Originally Posted by mike1127 View Post

Hey Shike, as an aside, how do you split the divs? This newer forum system is driving me crazy with the difficulty in editing quote blocks. It used to be so simple.


Select the text in the quote box from the end to the beginning. Go on outside the quote box until the "Quote" word. Copy that into your clipboard. Insert some carriage returns, and paste your clipboard in the middle of the carriage returns. Then, remove from the pasted quote boxes the text that you don't want to quote.

Quote:
Originally Posted by mikeaj View Post

I think the point being made is this:

(1) headphones/speakers are not perfectly linear

(2) measurements are very precise and accurate at what they are measuring, but because of (1) only actually give "perfect" information about what they are measuring -- extrapolation works mostly but not absolutely

 

 

Indeed cancellation might not work every time. For example if a system aliases past 18 kHz, and the test song has no frequency over 18 kHz.

But the other measurements cover all that can happen in a real-life system. Aliasing will appear in the IMD measurements with pairs of frequencies above 18 kHz.

On the other hand, don't take the above list as a guide. I'm no expert, and I can see that in some cases, special measurements should be performed. For example, to detect quantization noise, noise should be measured with a signal.

And mp3 encoders are an example of extremely non-linear devices. They can be tweaked in order not to alter frequency response, and still produce very distorded results. In this very special case, a measurement that can tell if the sound is distorded is the signal cancellation.

 

Despite these examples, I stand with my initial view : saying that measurements -in general- are only a partial description of the behaviour of a system is very far from reality.

The problems lies in

-Manufacturers don't perform or don't publish relevant measurements.

-Sometimes measurements show that the system is not perfect, but give no useful information : headphone frequency response, mp3 signal cancellation...

-Sometimes very special measurements are needed : aliasing, loose connection, piece of hair rattling inside a headphone driver...

 

As a conclusion, I would say that measurements can completely describe the behaviour of an audio device, but that measuring completely an audio device is sometimes far from easy.

 

Quote:
Originally Posted by mike1127 View Post

 

Let's assume a cable is linear-time-invariant (LTI). It's not; no real system is.

 

 

The case where a cable is not time-invariant is when there is a loose connection. Sometimes it works, sometimes not. Of course, it can happen extremely fast, so that the cable transmits a crackling sound.

We may mention another case : heating. But for that to be significant, the cable should become bright red. This occurs in speaker coils, but not in inteconnects or even speaker cables.

 

Wether a cable is linear or not depends on the definition of the input. If you include external electromagnetic interference as an input, then a cable is completely and totally linear. If you don't include EMI as an input, then sensitivity to EMI is a non-linearity of the cable, as an audio device.

 

I recall the definition of linearity : output is an algebraic linear application of the input. An equalizer is a linear device. A reverberation device too. A dynamics compressor is not.

 

Quote:
Originally Posted by mike1127 View Post

 

But even if it were, R, L, and C would be an insufficient model at higher frequencies where the cable begins to act as a transmission line.

 

 

On the contrary, R, L and C are a bit difficult to handle at audio frequencies, while they become a more effective way of describing the cable behaviour as a transmission line. They become ineffective at hyperfrequencies, above several GHz, when copper ceases to behave as a conductor.

 

Quote:
Originally Posted by mike1127 View Post

 

Please don't respond that "higher frequencies don't matter to audio." That's not the point. The point is that you invoked a model which is not complete, which demonstrates my very point that people tend to conflate their models with reality.

 

Why not ? This is a valid point. We are trying to modelize audio behaviour. We are not interested in the taste that a cable gives to beefsteak when added in the pan near the end of the cooking.

 

Quote:

Originally Posted by mike1127 View Post

 

The history of science demonstrates that models are not reality--this happens every time evidence is discovered that contradicts an established model. This has happened through scientific history and there's no reason to think it will stop now.

 

This is a misconception. Science always builds something new on top of something old.

Sometimes, the new model is based on completely different ideas than the old. That's what is called a paradigm change. But it never contradicts the old model.

 

The old Newtonian gravitation model said that all massive objects attract each others. As a result, on earth's surface, objects fall down with an acceleration of 9.8 m/s2.

The new general relativity says that mass and energy curve space-time. As a result, on earth's surface, objects fall down with an acceleration of 9.8 m/s2.

 

Today, the output of interconnects cancels the input down to -110 dB. Later, if a new theory of cables is found, that won't change the fact that the output of interconnects cancels the input down to -110 dB.

 

 

By the way, Mike, some time ago, you started in-depht long-term blind listening tests. What was the outcome ? Are you still working on this ?


Edited by Pio2001 - 8/30/10 at 7:00am
post #13 of 52
Quote:
Originally Posted by Pio2001 View Post

Let's add that phase response can be included into frequency response, and that reverberation fits in the time-based errors.

To be complete, measurements should include THD at all frequencies (2D plot) (for a speaker, it depends strongly on the frequency), and IMD at all pairs of frequencies (3D plot).

 

Adding on top of that that for amplifiers and speakers, measurements should be done at various levels, this is not very convenient.


Shouldn't there be a 4D IMD graph with sets of three frequencies, and a 5D IMD graph with sets of four frequencies, etc.?  And you need each graph at different volume levels too.  And how about transient intermod?  Non-transient IMD when there are also other transients occuring at the same time?  If we agree that a system is nonlinear and you don't have a complete formula that perfectly describes the output for any input, there are still some things you do not know even given full 2D IMD, full THD, etc. graphs.  Whether or not these effects are audible or worth considering down to a certain point is a different matter.

Well, if we're talking about standard interconnects, those behave linearly in the conditions we are concerned with at least down to the point of the noise floor of the best measuring equipment.  I agree that If cancellation works, there is no measurable difference in actual music and thus you can be sure that there is no difference to hear.  Differences may still be heard, but that's a property of the listener.  I originally thought that the argument was intended only to cover those devices that are known to be significantly nonlinear.

Measuring significantly nonlinear systems is tricky, so there is a lot of research on the subject.  Headphones and speakers are even more difficult to measure than nonlinear electronics where you're just dealing with input and output voltages and currents, but this has to do with the practicalities of recording sound waves.  However, very good descriptions can still be made even with these kinds of devices. 

Granted, I'm neither an expert in measure theory nor instrumentation, so I will leave it at that for now.  Please correct or debate any points you take issue with.

post #14 of 52

Maybe you'll find this relevant, maybe not.  Just happened on an article that did a great job of discussing the difficulty of perceiving accuracy.  The article revealed how a set of Stax Signatures were more accurate is revealing the recorded source while the Koss ESP950s sounded more accurate to the actual live Piano.

 

http://www.stereophile.com/headphones/koss_esp950_electrostatic_stereophones/index3.html

 

I think its a good read for many on Head-Fi to get a better understanding of what you personally should be looking for and understanding what others might be searching for.  Neutrality and accuracy usually get bandied about as this universal monolithic dogma without any context or point of reference.  Sometimes coming at odds w/ musicality in certain cases. 

post #15 of 52

 

Quote:
Originally Posted by mikeaj View Post


Shouldn't there be a 4D IMD graph with sets of three frequencies, and a 5D IMD graph with sets of four frequencies, etc.?  And you need each graph at different volume levels too.  And how about transient intermod?  Non-transient IMD when there are also other transients occuring at the same time? 


Is IMD with three frequencies more than the sum of IMD for each pair of frequencies, and if so, does it make a significant difference, or is it just 0.1 % of 0.1 %, and IMD with four frequencies 0.1 % of 0.1 % of 0.1 % ?

 

Transient IMD asks the same question. If you measure IMD on a fixed frequency sine plus a transient burst, will you get more than the total IMD for all frequencies contained in the burst associated with your sine ? If so, is it significantly more, or just 0.1 % more ?

 

I don't know.

New Posts  All Forums:Forum Nav:
  Return Home
  Back to Forum: Sound Science
Head-Fi.org › Forums › Equipment Forums › Sound Science › A misconception about measurements