- Joined
- Mar 20, 2013
- Posts
- 16,644
- Likes
- 12,458
Measurements vis-à-vis sound quality - part 1
At AMR/iFi, we go against the tide to some degree as we tend to refrain from publishing reams and reams of test data. There are those who like to pour over pages upon pages of test reports and get excited seeing 0.003% THD figures or 126dB SNR. We aren’t in that category.
This is because:
1. Making reliable measurements
As John Atkinson of Stereophile will attest, using an Audio Precision 2 measurement platform is not trivial. Complex measurements executed on such precise systems are significantly more challenging than checking DC voltage on a fully automatic multimeter.
Take our previous iPOWER article as an example; while internally we measure constantly, we don't publish the specs until we have everything out of the way. To share with an outcome this specific, it takes at least one full day of proper test rig setup, calibration and recalibration. It’s as far from just sticking two probes somewhere and reading the display as it gets. The takeaway is that the AP2 takes time and effort and then needs to be checked and re-checked.
![color]](http://www.computeraudiophile.com/attachments/f30-abbingdon-music-research-ifi-audio-sponsored/31289d1480876265-ifi-audio-micro-idsd-black-label-released-30th-november-2016-2.jpg[color=rgb(51, 51, 51)][/color])
2. Processing raw data takes time
Each test or measurement provides more or less precise data but only provides a singular answer to one very specific question above all else; the Volt DC, THD&N or SNR figures…
Now, measuring DC (ideally under nominal load - which complicates things) tells one if a power supply operates on the most basic level. It tells one nothing about how much noise the power supply generates, which is far more informative. Point being, a basic digital multimeter is unable to measure the noise of a good quality power supply.
However, the results of many different measurements performed on a device, inform us about the device’s performance in strictly one specific test. The only conclusion that can be drawn from these procedures is if the device works as intended within the limits of each test.
If we aimed for 0.005% THD and 0.05% THD is measured, we know that something is off as the product doesn’t work as intended. Said measurement result is needed in order to seek out the problem and correct it, therefore it is useful in this specific case. Yet what it does not provide is any indication of sound quality, none at all.
3. Measurements don't track sound quality
Whole audio systems’ (headphones based and full-sized stereo) frequency response measurement is the only ‘tool’ designers currently have that shed light on audible quality. There is no other which would reliably correlate with what most of us would call ‘good sound’. And even though its usefulness has been proven to a reasonable degree, there are often exceptions. Therefore even this measurement is far from perfect.
There are strong indications that time-domain’s performance impact is far greater than previously thought. Yet its measurements methods aren’t standardised thus far and research with proper correlation has generally low statistical significance.
Past frequency and time-domain performance, as long as - for example - harmonic and intermodulation distortion is lower than its audibility limit (something that is incidentally not captured by the common ‘THD&N’ measurement), the distortion itself is inaudible.
Similar qualifications of "as low as it's low enough" apply to noise and pretty much any other current audio measurement. So once a reasonable basic level of performance has been established in terms of traditional audio measurements, improving these parameters – ceteris paribus - does not provide improved sound quality.
As to what is audible, this is a complex multi-dimensional picture and contains material for several doctorates and a professorship (at least).
"To illustrate the limits of distortion audibility and the utter uselessness of THD as a measure of quality please consider this.
For a 100Hz tone played at 100dB (arguably very loud) 2nd harmonic distortion lower than 3% and 3rd harmonic distortion lower than 1% will be completely inaudible. Yet if the 30th harmonic exceeds 0.003% this will be audible and has been shown to be objectionable.
So in one case a system with 3% THD at 100dB SPL would pass the ‘no audible distortion’ criteria. While in another case a system with 0.003% THD would not pass the ‘no audible distortion’ criteria!"
This is of course hardly news, distortion audibility discussions are already known from dawn days of high fidelity in the 1950's. Yet this article provides a summary of the Audio of Distortion Audibility ca. 1957 state of knowledge and sadly, not much progress has been made since:
How Little Distortion can we hear
This extreme ambiguity in terms of measured performance with subjective sound quality correlation is one of the reasons why we do not generally publish extensive measurement results. It simply would be a waste of time, bits, electricity and paper.
And realistically, while we have our own views how “good sound” may be measured, to develop and promote such a system of tests is far outside the scope of a commercial operation and even if we were inclined to do so anyway, it would be likely as much of a fool’s errand as the work of D.E.L. Shorter of the BBC, Harry F. Olson of RCA and ongoing to Earl & Linda Geddes has been treated as by the audio engineering profession in general. ‘Chocolate fireguard’ springs to mind!
This is where our story's part one ends. Stay tuned, in the next one we'll shed some light on stuff we prefer to do.
At AMR/iFi, we go against the tide to some degree as we tend to refrain from publishing reams and reams of test data. There are those who like to pour over pages upon pages of test reports and get excited seeing 0.003% THD figures or 126dB SNR. We aren’t in that category.
This is because:
- Making reliable measurements is not a ‘5 minute job.’ It takes hours if not a full day or two to properly execute.
- Processing the raw test data for understandable publication takes even more time.
- Most importantly, measurements shed little (if any) light on sound quality.
1. Making reliable measurements
As John Atkinson of Stereophile will attest, using an Audio Precision 2 measurement platform is not trivial. Complex measurements executed on such precise systems are significantly more challenging than checking DC voltage on a fully automatic multimeter.
Take our previous iPOWER article as an example; while internally we measure constantly, we don't publish the specs until we have everything out of the way. To share with an outcome this specific, it takes at least one full day of proper test rig setup, calibration and recalibration. It’s as far from just sticking two probes somewhere and reading the display as it gets. The takeaway is that the AP2 takes time and effort and then needs to be checked and re-checked.
2. Processing raw data takes time
Each test or measurement provides more or less precise data but only provides a singular answer to one very specific question above all else; the Volt DC, THD&N or SNR figures…
Now, measuring DC (ideally under nominal load - which complicates things) tells one if a power supply operates on the most basic level. It tells one nothing about how much noise the power supply generates, which is far more informative. Point being, a basic digital multimeter is unable to measure the noise of a good quality power supply.
However, the results of many different measurements performed on a device, inform us about the device’s performance in strictly one specific test. The only conclusion that can be drawn from these procedures is if the device works as intended within the limits of each test.
If we aimed for 0.005% THD and 0.05% THD is measured, we know that something is off as the product doesn’t work as intended. Said measurement result is needed in order to seek out the problem and correct it, therefore it is useful in this specific case. Yet what it does not provide is any indication of sound quality, none at all.
3. Measurements don't track sound quality
Whole audio systems’ (headphones based and full-sized stereo) frequency response measurement is the only ‘tool’ designers currently have that shed light on audible quality. There is no other which would reliably correlate with what most of us would call ‘good sound’. And even though its usefulness has been proven to a reasonable degree, there are often exceptions. Therefore even this measurement is far from perfect.
There are strong indications that time-domain’s performance impact is far greater than previously thought. Yet its measurements methods aren’t standardised thus far and research with proper correlation has generally low statistical significance.
Past frequency and time-domain performance, as long as - for example - harmonic and intermodulation distortion is lower than its audibility limit (something that is incidentally not captured by the common ‘THD&N’ measurement), the distortion itself is inaudible.
Similar qualifications of "as low as it's low enough" apply to noise and pretty much any other current audio measurement. So once a reasonable basic level of performance has been established in terms of traditional audio measurements, improving these parameters – ceteris paribus - does not provide improved sound quality.
As to what is audible, this is a complex multi-dimensional picture and contains material for several doctorates and a professorship (at least).
"To illustrate the limits of distortion audibility and the utter uselessness of THD as a measure of quality please consider this.
For a 100Hz tone played at 100dB (arguably very loud) 2nd harmonic distortion lower than 3% and 3rd harmonic distortion lower than 1% will be completely inaudible. Yet if the 30th harmonic exceeds 0.003% this will be audible and has been shown to be objectionable.
So in one case a system with 3% THD at 100dB SPL would pass the ‘no audible distortion’ criteria. While in another case a system with 0.003% THD would not pass the ‘no audible distortion’ criteria!"
This is of course hardly news, distortion audibility discussions are already known from dawn days of high fidelity in the 1950's. Yet this article provides a summary of the Audio of Distortion Audibility ca. 1957 state of knowledge and sadly, not much progress has been made since:
How Little Distortion can we hear
This extreme ambiguity in terms of measured performance with subjective sound quality correlation is one of the reasons why we do not generally publish extensive measurement results. It simply would be a waste of time, bits, electricity and paper.
And realistically, while we have our own views how “good sound” may be measured, to develop and promote such a system of tests is far outside the scope of a commercial operation and even if we were inclined to do so anyway, it would be likely as much of a fool’s errand as the work of D.E.L. Shorter of the BBC, Harry F. Olson of RCA and ongoing to Earl & Linda Geddes has been treated as by the audio engineering profession in general. ‘Chocolate fireguard’ springs to mind!
This is where our story's part one ends. Stay tuned, in the next one we'll shed some light on stuff we prefer to do.
and loving the stuff that you guys are sharing and posting
