Quantitative measurements of audio performance
Jan 13, 2008 at 6:03 PM Thread Starter Post #1 of 14

P_A_W

New Head-Fier
Joined
Jan 13, 2008
Posts
7
Likes
0
I guess there are two cliques who post here. There are those that believe that audio can (and should) be quantitatively reviewed and those that think that only what they hear is relevant. I have no interest in hearing a rehash of this discussion. I am only writing to the former group who think that measurements are useful.

Is there a stash of meaningful, quantitative measurements on audio component performance on the web? Ie measurements of the transfer function/linearity for amplifiers/DACs/Speakers/ (what the hell) cables. I have seen surprisingly few measurements on this forum. Is there even agreement over what the right metrics for performance are?

Please accept my apologies if this has been covered in depth here or elsewhere.

Thx,

PAW
 
Jan 13, 2008 at 6:33 PM Post #2 of 14
Quote:

Originally Posted by P_A_W /img/forum/go_quote.gif
I guess there are two cliques who post here. There are those that believe that audio can (and should) be quantitatively reviewed and those that think that only what they hear is relevant. I have no interest in hearing a rehash of this discussion. I am only writing to the former group who think that measurements are useful.


That would be me to a limited extent. My feeling is that some measurements can sometimes to some extent tell you if something really strays from a transparent rendering of its inputs.

Quote:

Is there a stash of meaningful, quantitative measurements on audio component performance on the web? Ie measurements of the transfer function/linearity for amplifiers/DACs/Speakers/ (what the hell) cables. I have seen surprisingly few measurements on this forum.


I have never found one, if you scratch around you can find some stuff, but it is patchy. Stereophile for instance do some measurements but a global database, not as far as I know.

Quote:

Is there even agreement over what the right metrics for performance are?


Not here.
 
Jan 13, 2008 at 7:33 PM Post #3 of 14
Quote:

Originally Posted by P_A_W /img/forum/go_quote.gif
Is there a stash of meaningful, quantitative measurements on audio component performance on the web? Ie measurements of the transfer function/linearity for amplifiers/DACs/Speakers/ (what the hell) cables. I have seen surprisingly few measurements on this forum. Is there even agreement over what the right metrics for performance are?


There is no centralized one-stop shop for measurements. You'll have to do a lot of digging through various research papers, hunting down manufacturer and parts component datasheets, and hoping that someone has measured something similar to what you're looking for. There's also a lot of what's known as "lost technology", for example, internal company R&D information which has vanished after the company was bought out or went bankrupt. Some Western Electric amplifiers would fall into this category, we still have working examples but very few people, if any, have a full understanding of what makes them work.

Quote:

Originally Posted by hciman77 /img/forum/go_quote.gif
I have never found one, if you scratch around you can find some stuff, but it is patchy. Stereophile for instance do some measurements but a global database, not as far as I know.


Stereophile measurements should be taken with a bucket full of salt. The worst offender is their speaker measurements, specifically, the frequency response and the CSD (waterfall) plot. The frequency response chart should be identical to the back line (T=0) of the CSD plot, for they measure the exact same thing, which is to say the frequency response of the speaker before the signal is shut off to begin the decay. You'll see that there's often large discrepancies between the two, meaning either there's a significant measurement error or someone is doctoring the results. Either way, the measurements can't be trusted.
 
Jan 13, 2008 at 10:24 PM Post #5 of 14
Quote:

Originally Posted by achristilaw /img/forum/go_quote.gif
With Headpones. Everything is laid bare. My Ears Measure fine.


Please%20do%20not%20feed%20the%20trolls.jpg
 
Jan 13, 2008 at 11:03 PM Post #6 of 14
Quote:

Originally Posted by P_A_W /img/forum/go_quote.gif
I guess there are two cliques who post here. There are those that believe that audio can (and should) be quantitatively reviewed and those that think that only what they hear is relevant. I have no interest in hearing a rehash of this discussion. I am only writing to the former group who think that measurements are useful.

Is there a stash of meaningful, quantitative measurements on audio component performance on the web? Ie measurements of the transfer function/linearity for amplifiers/DACs/Speakers/ (what the hell) cables. I have seen surprisingly few measurements on this forum. Is there even agreement over what the right metrics for performance are?



This is a great question. First, there is no universally agreed metric for judging performance. Generally, absolute THD+N at full power is meaningless because it does not correspond to how the human auditory system functions (and it also doesn't take into account variations at different power levels). For some experimental results demonstrating this, see here:
Results
There have been numerous attempts to develop a quantitative metric for distortion, going back to the tube days. For some modern work in this area, Dr. Geddes' work is good:
http://www.gedlee.com/downloads/dist...perception.ppt
(warning PowerPoint)
and this article isn't bad either:
Stereophile: Euphonic Distortion: Naughty but Nice?

The main academic journal in this area is the JAES (Journal of the Audio Engineering Society). If you have access to it, it's universally good, and it will answer your questions more than most resources on the web. There's nonsense being spread by both sides ("objectivists" and "subjectivists") on the web, and while the subjectivist nonsense is more overt, you have to be careful with the objectivists as well. For instance, while it is true that cables make very little difference, anyone who insists that a 100ft run of unbalanced RCA cable does not have an audible effect on sound is just repeating a quasi-religious belief not based on listening tests.

Generally, you want to look for the following things in measurements:
- for speakers, the power response (frequency response is next to meaningless)
- for amplifiers, the distribution of odd-order distortion products, particularly higher order distortion products, as well as the amplifier's distortion performance at low power levels (e.g. 1 watt) because this is where you spend most of your listening time, even with speakers
- for anything, distortion (both linear and nonlinear) within the 1kHz-2kHz range because this is where the human ear is most sensitive... a spike in third harmonic distortion in this range is very audible

There are various other things to look for. Some things, such as the effect of various choices of digital filter in a DAC are hard to correlate with measurements. For those you basically do have to listen.

The best way to get a handle on what matters, and in particular what matters to you, is to get into DIY. Even with speakers, a decent measurement rig is inexpensive now and you'll learn more than you will by just reading, by correlating various factors with how they affect the sound.

In terms of gear measurements, Stereophile is good. They publish a full set of measurements for about half the gear they review, and it's comprehensive, including things like distortion at various power levels, jitter sensitivity, and power response. You can ignore the verbal reviews, or try to correlate them with the measurements (this is often very easy with speakers, harder to do with amplifiers).
 
Jan 14, 2008 at 12:16 AM Post #7 of 14
Quote:

Originally Posted by Wodgy /img/forum/go_quote.gif
Generally, you want to look for the following things in measurements:
- for speakers, the power response (frequency response is next to meaningless)
- for amplifiers, the distribution of odd-order distortion products, particularly higher order distortion products, as well as the amplifier's distortion performance at low power levels (e.g. 1 watt) because this is where you spend most of your listening time, even with speakers
- for anything, distortion (both linear and nonlinear) within the 1kHz-2kHz range because this is where the human ear is most sensitive... a spike in third harmonic distortion in this range is very audible



I am not sure what you mean by frequency response.

The power response (as a function of frequency) will tell you about how uniform the amplitude response is, but it gives you no information about the phase. To what extent can you actually hear a phase shift (between frequencies)?

Assuming a linear component, the impulse response (or step function response) would give you both amplitude and phase at all frequencies. Of course, it is unclear exactly how to interpret this data when comparing two different components. The power response has a more straightforward interpretation. On-the-other-hand, products of the fourier transforms of the transfer functions could predict which components in the system were limiting. (ie Does that $3k interconnect really significantly affect the signal or is the dominant source of degradation the amplifier or the DAC. Sure, you could demo it, but who has the time?)

It appears from the length at which you wrote about non-linearities that you believe this to be the dominant source of audible distortion. Is this universally accepted?

For instance, for a DAC what is the dominant source of distortion?

When these components were designed, I guarantee that the design strategy wasn't twenty golden-eared audiophiles ABXing every component choice. Sure, one could get into DIY, but these are commercial products and people have already made the measurements on better equipment than I am ever likely to buy. These companies won't tell us what the specs are because it would allow us to compare equipment without bringing it home. And... no one would ever spend more than fifty bucks on a cable again.

By not demanding to know the specs in detail, we are begging the snake-oil salesmen to stop by...
 
Jan 14, 2008 at 1:15 AM Post #8 of 14
Good question.

I personally believe it's a matter of both. I use specs and measurement as a guide to help lead me to a purchase. If one component doesn't sound good to my ears, chances are another using the same chips won't either. That doesn't always hold true, but different chips usually have different house sounds. Dynamic range, SNR, THD+N... measurement like those are all important.

In the end though, your ears should be the final judge. If it sounds good to you, great. But your ears can be deceiving. A long time ago I liked EQing everything. It took me many years of training my brain and ears to appreciate a flat frequency response. What you think may sound like a flat frequency curve may not be since your brain naturally hones in on the human vocal range first (around 1kHz). Measurements can serve as a useful guide for helping you learn to appreciate a non-EQ'd sound.
 
Jan 14, 2008 at 1:42 AM Post #9 of 14
Quote:

Originally Posted by P_A_W /img/forum/go_quote.gif
I am not sure what you mean by frequency response.

The power response (as a function of frequency) will tell you about how uniform the amplitude response is, but it gives you no information about the phase. To what extent can you actually hear a phase shift (between frequencies)?



With loudspeakers, "power response" typically has a different meaning than you're thinking. It refers to the amount of total power emitted into the room (i.e. not just straight ahead), while "frequency response" refers to the amplitude response measured on-axis (straight ahead). Eventually most radiated sound reaches your ear through reflections, so the power response is a much more accurate gauge of how a speaker sounds. This was the subject of an enormous amount of research in the 70s and 80s.

Any multi-driver speaker has a non-uniform power response, the extent of the non-uniformity being determined mostly by the size of the drivers, the crossover point, and the steepness of the crossover. Different issues affect the bass region.

There is a debate about whether phase response is audible. Some believe it is audible, but only in the midrange, others believe it is only audible in the bass region.

Quote:

Assuming a linear component, the impulse response (or step function response) would give you both amplitude and phase at all frequencies. Of course, it is unclear exactly how to interpret this data when comparing two different components.


Yes, precisely. It is always possible to measure, but the difficulty is in determining what is audible. Unfortunately, the measures which are the easiest to quantify (linear distortion and THD+N) are the least useful. Get used to reading nonlinear distortion spectra and correlating them with sound quality.

Quote:

(ie Does that $3k interconnect really significantly affect the signal or is the dominant source of degradation the amplifier or the DAC. Sure, you could demo it, but who has the time?)


No, $3k interconnects and power cords typically don't make any significant measurable difference and aren't worth spending any time with. That said, the impact of a long run of unbalanced cable will be measurable.

Quote:

It appears from the length at which you wrote about non-linearities that you believe this to be the dominant source of audible distortion. Is this universally accepted?


For the most part, yes. It is not difficult to get the linear behavior of a system right (the exception being headphones actually, where their linear behavior is all over the map and there is no universally agreed upon ideal linear response, because the head has been subtracted out). There are some cases where the linear behavior has an impact, such as the power response of speakers and the shape of a DAC's digital filter.

Quote:

For instance, for a DAC what is the dominant source of distortion?


Linear distortion? The digital filter. Non-linear distortion? The analog output stage and I/V conversion.

Quote:

When these components were designed, I guarantee that the design strategy wasn't twenty golden-eared audiophiles ABXing every component choice.


When you get into the true high end ($5,000+ gear), they often don't do any measurements at all, they just design by ear. In most cases this produces poor sounding gear, which is why you should avoid the high end.

If you're talking sources, you won't get any better than the Benchmark DAC1 or the Lavry DA10, no matter how much you spend. Those two are pushing the state of the art in digital to analog converters, and both are reasonably priced and neither uses boutique components. As you would expect from high quality, well-measuring gear, those two sound more similar than different, though they both have a slightly different flavor due to their use of different DAC chips and different output stages. Despite the advertising, recent revisions of the DA10 use exactly the same jitter recovery scheme as the DAC1, which I find a bit of a letdown, but they're both good.

Quote:

Sure, one could get into DIY, but these are commercial products and people have already made the measurements on better equipment than I am ever likely to buy. These companies won't tell us what the specs are because it would allow us to compare equipment without bringing it home.


Yes and no. You'll never be able to compare equipment just by looking at measurements, because there are too many variables. For instance, in a speaker, whether you prefer a slightly more uneven power response over a peak in odd-harmonic distortion around 5kHz is going to be a matter of personal preference. There are many engineering tradeoffs in designing a quality speaker. It's the same with amplifiers, which is why a well-designed tube amp is competitive with a well-designed solid state amp, despite THD+N numbers suggesting otherwise. What measurements are good for is winnowing out the garbage from the quality gear, so you know what to audition. Once you become familiar with your own sonic preferences (e.g. can you hear a peak in a speaker's frequency response at 18 kHz?), you can do this a lot more effectively.
 
Jan 14, 2008 at 1:56 AM Post #10 of 14
i find this amusing.

Quote:

Originally Posted by P_A_W /img/forum/go_quote.gif
Is there a stash of meaningful, quantitative measurements on audio component performance on the web?


Quote:

Originally Posted by Roam /img/forum/go_quote.gif
Stereophile measurements should be taken with a bucket full of salt. ... Either way, the measurements can't be trusted.


Quote:

Originally Posted by Wodgy /img/forum/go_quote.gif
In terms of gear measurements, Stereophile is good.


 
Jan 14, 2008 at 6:27 PM Post #11 of 14
I have very little faith in advertiser funded media for really objective reviews. Audio magazines (and websites) are among the worst offenders.

See ya
Steve
 
Jan 15, 2008 at 6:19 AM Post #12 of 14
This would be similar to measuring a beautiful lady or a handsome man.
 
Feb 2, 2008 at 3:38 PM Post #14 of 14
Quote:

Originally Posted by melomaniac /img/forum/go_quote.gif
treasure trove of info in this regard on the AES website and elsewhere


I guess what I had in mind was more competent reviews of existing equipment than an engineering journal.

I did, at last, find a source of quantitative information and reviews: The Audio Critic. What a sad world we inhabit that Stereophile remains in print while The Audio Critic is relegated to online obscurity. Wow... these writers and reviewers actually understand electrical engineering AND the scientific method!

Clearly such competence was NOT an effective business model with respect to ad revenue but this publication was a spectacular value to the reader and consumer. Perhaps the editor's habit of referring to the high-end of the industry, various audio charlatans and the uninformed audiophile consumer as tweakos was as persuasive a strategy for increasing readership as name calling is a part of professional journalism. But, what editor lacked in judgement and business acumen was more than made up for by his professional approach to audio. These writers had a passion for quantitation and honesty that is unmatched in any other magazine on audio that I have seen.

--P_A_W
 

Users who are viewing this thread

Back
Top