[garbulky]

The main reason is that measurements are intended to verify a design, not to test how something sounds, and verification requires testing performance to certain specs. Generating a known, calibrated test signal is the only way to verify if a design is objectively performing to spec.

Listening tests should always be done with music, and preferably with known music, and the results are always subjective to the listener

This is why test measurements are not necessarily the best guide to how something sounds, and they can never be used as more than predictive. speculative metrics for how something might sound to a wide audience. Measurements give a designer an idea that they have created a proper design and that it "should" sound right, but it is not the end-all.

Something I've been curious about - since measurements describe the actual signal coming out of the piece of gear and can also be used to measure impedance and interactions, what would make it sound different other than subjective biases?

 

[Ableza]

Something I've been curious about - since measurements describe the actual signal coming out of the piece of gear and can also be used to measure impedance and interactions, what would make it sound different other than subjective biases?

We don't listen to music the same way a piece of gear handles test signals. It's not about bias, it's about perception. Music has subtlety, characters that emerge from the combination of tones, rhythms, overtones, harmonics, dynamics, pacing, etc. that simply cannot be duplicated with a signal generator. At the same time, creating a "standard" using a complex signal like music would be impossible as the possible variety is too great. What music should we use as a standard? Mahler? The Monkeys? Judas Priest?

 

[MisterMoJo]

In other words, and to quote @Baldr, things can test well and still sound like ass.

And maybe visa versa? Test bad and sound great?

 

[garbulky]

We don't listen to music the same way a piece of gear handles test signals. It's not about bias, it's about perception. Music has subtlety, characters that emerge from the combination of tones, rhythms, overtones, harmonics, dynamics, pacing, etc. that simply cannot be duplicated with a signal generator. At the same time, creating a "standard" using a complex signal like music would be impossible as the possible variety is too great. What music should we use as a standard? Mahler? The Monkeys? Judas Priest?

I think subjective bias was a misnomer. The brain doesn't hear bias, it hears perception. But my question is the combination of tones rhythms etc. wouldn't they be adequately captured in the minute variations in the signals we can measure? So for instance two dacs with measurements that are identical to below 100 db, what would cause a change in the sound?
As for creating a standard, my standard has always been an unprocessed non-close mic'd piece of acoustic music/vocals captured in stereo on a natural soundstage. That's the closest imitation of real life music that takes into account that most of our music gear is two channel. But it doesn't have to be that one.

 

[Ableza]

I think subjective bias was a misnomer. The brain doesn't hear bias, it hears perception. But my question is the combination of tones rhythms etc. wouldn't they be adequately captured in the minute variations in the signals we can measure? So for instance two dacs with measurements that are identical to below 100 db, what would cause a change in the sound?
As for creating a standard, my standard has always been an unprocessed non-close mic'd piece of acoustic music/vocals captured in stereo on a natural soundstage. That's the closest imitation of real life music that takes into account that most of our music gear is two channel. But it doesn't have to be that one.

Nope, that's not a standard. That's a subjective choice. A standard is something that is 100% uniform, can be calibrated on a piece of test equipment, and is the same no matter who is using it. A recording of an instrument is not a standard, nor can it be.

 

[garbulky]

Nope, that's not a standard. That's a subjective choice. A standard is something that is 100% uniform, can be calibrated on a piece of test equipment, and is the same no matter who is using it. A recording of an instrument is not a standard, nor can it be.

Sure choose one recording as described and then use it across the devices. (Make digital copies to keep the recording the same). If you wish to know how loud it needs to be for subjective listening, record a singular test tone at the beginning of the master for volume measurement. Do you have any thoughts on the other segment? DACs that measure below 100 db in distortions and identically above and yet sound different?

 

[Ableza]

Sure choose one recording as described and then use it across the devices. (Make digital copies to keep the recording the same). If you wish to know how loud it needs to be for subjective listening, record a singular test tone at the beginning of the master for volume measurement. Do you have any thoughts on the other segment? DACs that measure below 100 db in distortions and identically above and yet sound different?

That would be a great plan for personal comparisons, or even measurements if you have the tools. But it is not a standard that could be used world-wide and industry-wide. And yes, equipment - DACs, amps, preamps, whatever - that seemingly measure the same can and do sound different. If that was not the case, why have so many different designs on the market? If there was one design that always met the standard and sounded "perfect," then everyone would simply do that. It's about how the parts work together to create a perception in the mind of the listener, and that's where preference comes in. Perception. It's not bias, it's unconscious in each of us.

 

[garbulky]

That would be a great plan for personal comparisons, or even measurements if you have the tools. But it is not a standard that could be used world-wide and industry-wide. And yes, equipment - DACs, amps, preamps, whatever - that seemingly measure the same can and do sound different. If that was not the case, why have so many different designs on the market? If there was one design that always met the standard and sounded "perfect," then everyone would simply do that. It's about how the parts work together to create a perception in the mind of the listener, and that's where preference comes in. Perception. It's not bias, it's unconscious in each of us.

My conundrum here is that if they measure the same but sound different, then it's our perception that's changing. I have listened to plenty of gear that I have a hard time believing it's all in my head. But I can't reconcile that identically measured gear should sound the same, and (logically) it must be us hearing differences from things not related to the actual output of the device. It sounds logical, but the differences are so noticeable that I believe they MUST be measurable, being easily over -100 db in magnitude (I'm guessing in the -20 and -40 db). But yet, they are not. So I don't understand - either I'm imagining it or being easily influenced by things not related to the actual sound in the air, or there is something that we are not clearly measuring to correlate the two.
It doesn't help that when doing DBT's all these big differences I hear just sort of dissapear.... so it's just confusing. But I can't bring myself to say that it's all in my head because the experience is repeatable in regular use.

 

[Ableza]

Welcome to the wonderful world of audio, where things are not always logical. Buy what you think sounds good and enjoy the music.

 

[earnmyturns]

Something I've been curious about - since measurements describe the actual signal coming out of the piece of gear and can also be used to measure impedance and interactions, what would make it sound different other than subjective biases?

Standard audio measurements are averages of many data points from subjecting the equipment under test to simple inputs. Such a low-dimensional representation of the complex behavior of a complex physical/perceptual system cannot possibly do justice to the very high-dimensional nature of real musical signals and system response.

 

[johnjen]

Another factor is music is dynamic while test tones and measurements are not.
Measurements need repetition of the test signal to 'validate' the measured response as being 'real' and not an 'artifact'.
This need for duplication and repetition is where music simply doesn't and can't apply.

We don't hear perceptions, we create them based upon previous experience(s), along with expectation(s), and many other variables (mood, set and setting etc.) so there is no sameness from person to person, nor situation to situation, on top of the dynamic nature of music itself.

Yes you can hear differences between gear that measurements can't reveal, but does anyone else hear those differences, and in the same ways?

Measurements, as mentioned, are used as a 'standard' to make operational comparisons even possible, and while they are operationally dependent, that doesn't mean they are acoustically (as in the re-creation of the dynamics of music) related.

JJ

 

[earnmyturns]

My conundrum here is that if they measure the same but sound different, then it's our perception that's changing

No. The standard measurements are insufficient to capture all the subtleties of how we hear, and the variation among humans, and listening conditions. As was noted before by others, standard measurements are mainly error checks on design or implementation. Once measurements are close enough to ideal, the variation between systems that is left is still there, but no longer measurable. However, human hearing under the right conditions and listener training can detect differences that are beyond the "skill" of standard measurements.

When I was a kid in college, I helped civil engineers with coding. Back then, computers, numerical methods, or test rigs were not capable enough to measure and model anything but linear dynamics. But the engineers I worked with still did not ignore nonlinear plastic deformation in their designs, often by trial and error with scale models and drawing from the practical wisdom of their elders. Two designs that "measured the same" could definitely behave differently under conditions beyond the quantitative tools available then. Quite a few of the infamous disasters in civil engineering came from people believing their limited math and measurements too much,

My work over the last several decades has been on systems for search, speech recognition, pattern recognition where the work is all driven by measurement, A/B tests, ... Still we understand very well that A and B may measure similarly in all the standard metrics, and still behave very differently in practice.

 

[garbulky]

No. The standard measurements are insufficient to capture all the subtleties of how we hear, and the variation among humans, and listening conditions. As was noted before by others, standard measurements are mainly error checks on design or implementation. Once measurements are close enough to ideal, the variation between systems that is left is still there, but no longer measurable. However, human hearing under the right conditions and listener training can detect differences that are beyond the "skill" of standard measurements.

When I was a kid in college, I helped civil engineers with coding. Back then, computers, numerical methods, or test rigs were not capable enough to measure and model anything but linear dynamics. But the engineers I worked with still did not ignore nonlinear plastic deformation in their designs, often by trial and error with scale models and drawing from the practical wisdom of their elders. Two designs that "measured the same" could definitely behave differently under conditions beyond the quantitative tools available then. Quite a few of the infamous disasters in civil engineering came from people believing their limited math and measurements too much,

My work over the last several decades has been on systems for search, speech recognition, pattern recognition where the work is all driven by measurement, A/B tests, ... Still we understand very well that A and B may measure similarly in all the standard metrics, and still behave very differently in practice.

I would like to see some documentation or research that shows that our instruments are insufficient in capturing the subtleties that the ear can tease out. Our instruments can dig way down into the noise floor with greater accuracy than our ears can. While our ears may give out at 60-70 db of variation realistically, we have instruments getting well beyond 110 db. Having said that, yes I do hear differences in electronic gear (in subjective evaluation) when the instruments are both providing distortion at well below what I can hear. So it is a conundrum for me.

 

[Ableza]

I would like to see some documentation or research that shows that our instruments are insufficient in capturing the subtleties that the ear can tease out. Our instruments can dig way down into the noise floor with greater accuracy than our ears can. While our ears may give out at 60-70 db of variation realistically, we have instruments getting well beyond 110 db. Having said that, yes I do hear differences in electronic gear (in subjective evaluation) when the instruments are both providing distortion at well below what I can hear. So it is a conundrum for me.

In all seriousness, I humbly suggest you don't worry about it. Let the audio engineers and professionals worry about it as it affects their designs. Listen to gear and choose what appeals to you. Be subjectively smart with your money. You don't buy specs, you don't listen to test tones, you buy enjoyment. Enjoy the music.

 

[gdhal]

@garbulky measurements/data of audio hardware do matter and at some level (sometimes large, sometimes small) make a contribution to the overall sound; and that is irrespective of whether or not the sound is measured/heard objectively, subjectively or both. Measurements can also be used to validate/invalidate marketing claims by manufacturers that may or could entice potential customers or existing customers to purchase products.