[Bio-Rhythm]

A “back of the envelope” method to consider if the aim is to measure a headphones neutrality. E.g., not over emphasizing any aspect of the frequencies response, not adding a signifiant amount of distortion. The methods to measure & reproduce (synthesize) this less cognitively biased method already exist, the scientific method.

As an example I infer the reader to the measurement of temperature. “Absolute zero is the lowest possible temperature where nothing could be colder and no heat energy remains in a substance. “ https://www.sciencedaily.com/terms/absolute_zero.htm

Or the measurement of timing using the benchmark of atoms.
https://www.timeanddate.com/time/international-atomic-time.html

Therefore, a temperature measuring device, e.g., thermometer can be calibrated for accuracy of temperature. This method mitigates the human cognitive bias effect of, to paraphrase, “I will always trust my own subjective sense of touch to get a more accurate temperature reading”. As it can be calculated why an accurate thermometer is more precise than human subjective experience & demonstrated using experiment to show a thermometer is more accurate than the human sense of touch for the measurement of objective temperature. Of course, devices that produce temperature don’t tend to cause such a fuss compared to headphones, due to behavioural psychological traits that are beyond the scope of this brief example. And many humans will simply not be able to accept being taken out of the “what’s the most accurate audio production” equation. However, they are missing out on the aim (point).

The question is, for those that really are interested in being able to develop a more accurate system to measure headphones audio accuracy, what can a headphones audio quality be calibrated against?. We’d need to use the physics of sound to produce an accurate audio measuring device to measure an objects sound, just as a calibrated thermometer can comparably accurately (compared to human touch) measure temperature. e.g., A microphone that measures (records),just for an example, a drum (using physics to quantify the measurements) . Once this is achieved, we have, to the best of our current understanding, an accurate benchmark to test any audio production devices sound reproduction accuracy. Of course taking into consideration the relevant limitations (acoustic finger print) of the audio device. However, we can then develop headphones & measure them using the same calibrated microphone.

The above is just a very brief outline & I’m sure mathematicians, physicists & audio engineers could provide more details.

Some will say, but that method does not take into account the many variables of human hearing. Yep, and that’s the whole point of the scientific methods, i.e., to remove human bias when trying to analyze objective reality. i.e., In general, science is about discovering objective reality, not trying to do the impossible & make objective reality conform to human individual & highly variable subjective experience (even though a less calibrated subjective experience believes that's possible). The physics of objective sound, e.g., the bang of a drum is also not taking human subjective perception into consideration (before it enters our ears). Therefore the human variable is actually irrelevant if we want to have a high degree of confidence that headphone “X” is producing more accurate “real world” audio . E.g., Subjectively we’d all hear (perceive environmental information) the accurate “X” headphone differently (within the subjective variance of our species subjective perception), just as we hear the bang of a drum differently ( environmental\ecological objective sounds differently). But at least we’d be more confident that the “neutral headphone” was producing the sound of the drum more accurately, & then our ears (biology) would analyze that sound using our own subjective experience. i.e., a more natural sounding headphone. Of course there are many acoustic variables to consider when recording the bang of a drum, e.g., environment the drum is in, location to the recording device. However, due to the reason why we hear, i.e., evolutionary psychological functional adaptions, the objective audio should be a sound from our natural environment as we are already more adapted to hear those sounds more accurately.

Of course the above method will not be a “cash cow” in a consumerist driven market place, in the context of different headphone manufactures maintaining the “the next product is better than the last” marketing strategy & headphone manufactures competing with one another for “the best headphone ever” strategy. Maybe they could, ultimately, do us all a favour & compete to produce the headphones with the “X sound signature” using more ecologically sustainable manufacturing methods.

Fortunately headphone manufacturers don’t tend to sell scientific instruments as marketing gimmicks and spin (lies if it’s intentional). i.e., science needs accurate instruments. Though personal bias has significantly corrupted the publics perception of the sciences that are related to anthropogenic climate change. e.g., ecolology, climatology. Thanks “humanity as usual” , you’ll make losers of us all if your not regulated to behave more ecologically sustainably. i.e., regulate yourselves for the good of yourselves.

 

[sander99]

If you were talking about any other audio component, even loudspeakers, but not headphones then everything you say sounds reasonable.
Only with headphones there is one complication: there is no objective flat or neutral for everyone.
If you are listening to sounds coming from your environment, from a certain distance, for example to the drum you mentioned then the sound is subjected to your personal head related transfer function (hrtf) filtering. (The sound is bouncing off your torso, bending round your head, bouncing of your pinnae, bending into your ear canal, etc.)
The resulting filtering of the audio differs per person. If you use headphones a part of this filtering is skipped. Because the part that is skipped differs per person one and the same headphone will sound objectively different to different people.
Also you would have to consider what you use the headphones for, or what end result you want from the headphones.
If you are listening to traditional stereo recordings intended for loudspeakers then it is not so clear what headphones should make from that. The hrtf filtering depends on what direction the sound is coming from. What direction is it intended to come from when you listen to headphones? With traditional stereo recordings for most people the sound will seem to be inside or close to the head anyway, because of all sorts of localisation clues will be missing or not consistant and correct for localisation of sounds at a distance.

 

[KeithPhantom]

A “back of the envelope” method to consider if the aim is to measure a headphones neutrality. E.g., not over emphasizing any aspect of the frequencies response, not adding a signifiant amount of distortion. The methods to measure & reproduce (synthesize) this less cognitively biased method already exist, the scientific method.

As an example I infer the reader to the measurement of temperature. “Absolute zero is the lowest possible temperature where nothing could be colder and no heat energy remains in a substance. “ https://www.sciencedaily.com/terms/absolute_zero.htm

Or the measurement of timing using the benchmark of atoms.
https://www.timeanddate.com/time/international-atomic-time.html

Therefore, a temperature measuring device, e.g., thermometer can be calibrated for accuracy of temperature. This method mitigates the human cognitive bias effect of, to paraphrase, “I will always trust my own subjective sense of touch to get a more accurate temperature reading”. As it can be calculated why an accurate thermometer is more precise than human subjective experience & demonstrated using experiment to show a thermometer is more accurate than the human sense of touch for the measurement of objective temperature. Of course, devices that produce temperature don’t tend to cause such a fuss compared to headphones, due to behavioural psychological traits that are beyond the scope of this brief example. And many humans will simply not be able to accept being taken out of the “what’s the most accurate audio production” equation. However, they are missing out on the aim (point).

The question is, for those that really are interested in being able to develop a more accurate system to measure headphones audio accuracy, what can a headphones audio quality be calibrated against?. We’d need to use the physics of sound to produce an accurate audio measuring device to measure an objects sound, just as a calibrated thermometer can comparably accurately (compared to human touch) measure temperature. e.g., A microphone that measures (records),just for an example, a drum (using physics to quantify the measurements) . Once this is achieved, we have, to the best of our current understanding, an accurate benchmark to test any audio production devices sound reproduction accuracy. Of course taking into consideration the relevant limitations (acoustic finger print) of the audio device. However, we can then develop headphones & measure them using the same calibrated microphone.

The above is just a very brief outline & I’m sure mathematicians, physicists & audio engineers could provide more details.

Some will say, but that method does not take into account the many variables of human hearing. Yep, and that’s the whole point of the scientific methods, i.e., to remove human bias when trying to analyze objective reality. i.e., In general, science is about discovering objective reality, not trying to do the impossible & make objective reality conform to human individual & highly variable subjective experience (even though a less calibrated subjective experience believes that's possible). The physics of objective sound, e.g., the bang of a drum is also not taking human subjective perception into consideration (before it enters our ears). Therefore the human variable is actually irrelevant if we want to have a high degree of confidence that headphone “X” is producing more accurate “real world” audio . E.g., Subjectively we’d all hear (perceive environmental information) the accurate “X” headphone differently (within the subjective variance of our species subjective perception), just as we hear the bang of a drum differently ( environmental\ecological objective sounds differently). But at least we’d be more confident that the “neutral headphone” was producing the sound of the drum more accurately, & then our ears (biology) would analyze that sound using our own subjective experience. i.e., a more natural sounding headphone. Of course there are many acoustic variables to consider when recording the bang of a drum, e.g., environment the drum is in, location to the recording device. However, due to the reason why we hear, i.e., evolutionary psychological functional adaptions, the objective audio should be a sound from our natural environment as we are already more adapted to hear those sounds more accurately.

Of course the above method will not be a “cash cow” in a consumerist driven market place, in the context of different headphone manufactures maintaining the “the next product is better than the last” marketing strategy & headphone manufactures competing with one another for “the best headphone ever” strategy. Maybe they could, ultimately, do us all a favour & compete to produce the headphones with the “X sound signature” using more ecologically sustainable manufacturing methods.

Fortunately headphone manufacturers don’t tend to sell scientific instruments as marketing gimmicks and spin (lies if it’s intentional). i.e., science needs accurate instruments. Though personal bias has significantly corrupted the publics perception of the sciences that are related to anthropogenic climate change. e.g., ecolology, climatology. Thanks “humanity as usual” , you’ll make losers of us all if your not regulated to behave more ecologically sustainably. i.e., regulate yourselves for the good of yourselves.

Even though some say it has some defects, for headphones you can buy a miniDSP EARS (only $199) and for electrical equipment, Audio Precision just released the APx500 Flex (albeit $3,000);

 

[bigshot]

Bias is only half of the problem. The other half is perceptual error. Our heads and ear canals are all shaped a little differently. That affects how we hear. You can establish a perfect target curve for headphones, but if the size of the person's noggin makes the cups fit a little different, or if someone has a particular bend in their ear canal that another person doesn't have, the perfect target might be different for them.

Everyone points to the Harman Curve as if it is an absolute thing... it isn't. It's the average of the bell curve of a whole bunch of differing responses from different people with different skulls. It's a starting point, not a destination. You start there, and add a little of this and a little of that until it hits your own personal target curve. Is all this subjective or objective? I'm not going to get into that silly argument any more. I'm just going to say that the "perfect target" varies from person to person.

Whenever you've got something intensely personal like this in a chain, you make that the "wild card". You calibrate everything else that you possibly can, and you provide the options to satisfy the wild card at the end of the chain. In cooking, the chef carefully balances all of the herbs and flavors. When it's absolutely perfect, the waiter takes it to the table... and on the table, right in front of the guest is a set of salt and pepper shakers, and sugar and cream for his coffee. With audio, you calibrate all of the electronics- the player, the DAC, the amp... to pass signal perfectly, like a wire with gain. Then you let people choose a set of headphones that fits their anatomical uniqueness. If it isn't quite perfect, you tweak with EQ.

I had the opportunity to interact with an engineer who designed high end headphones and speakers. He told me that the difference between a high end set of headphones and a less expensive set isn't so much in the materials and manufacturing. It's in the manufacturing tolerances. One copy of a particular brand and model of headphones might sound quite different than a different copy of the same brand and model simply because of the vagaries of manufacturing. So even if it was possible to create the perfect set of headphones, it would still vary from copy to copy.

It's better to focus on things that matter. Don't worry about established standards of perception when it comes to transducers. Find the transducers that work best for you. Fine tune them with EQ if you need to. But job one in this case is pleasing yourself, not striking some fixed criteria that doesn't take your individuality into account.

 

[castleofargh]

A “back of the envelope” method to consider if the aim is to measure a headphones neutrality. E.g., not over emphasizing any aspect of the frequencies response, not adding a signifiant amount of distortion. The methods to measure & reproduce (synthesize) this less cognitively biased method already exist, the scientific method.

As an example I infer the reader to the measurement of temperature. “Absolute zero is the lowest possible temperature where nothing could be colder and no heat energy remains in a substance. “ https://www.sciencedaily.com/terms/absolute_zero.htm

Or the measurement of timing using the benchmark of atoms.
https://www.timeanddate.com/time/international-atomic-time.html

Therefore, a temperature measuring device, e.g., thermometer can be calibrated for accuracy of temperature. This method mitigates the human cognitive bias effect of, to paraphrase, “I will always trust my own subjective sense of touch to get a more accurate temperature reading”. As it can be calculated why an accurate thermometer is more precise than human subjective experience & demonstrated using experiment to show a thermometer is more accurate than the human sense of touch for the measurement of objective temperature. Of course, devices that produce temperature don’t tend to cause such a fuss compared to headphones, due to behavioural psychological traits that are beyond the scope of this brief example. And many humans will simply not be able to accept being taken out of the “what’s the most accurate audio production” equation. However, they are missing out on the aim (point).

The question is, for those that really are interested in being able to develop a more accurate system to measure headphones audio accuracy, what can a headphones audio quality be calibrated against?. We’d need to use the physics of sound to produce an accurate audio measuring device to measure an objects sound, just as a calibrated thermometer can comparably accurately (compared to human touch) measure temperature. e.g., A microphone that measures (records),just for an example, a drum (using physics to quantify the measurements) . Once this is achieved, we have, to the best of our current understanding, an accurate benchmark to test any audio production devices sound reproduction accuracy. Of course taking into consideration the relevant limitations (acoustic finger print) of the audio device. However, we can then develop headphones & measure them using the same calibrated microphone.

The above is just a very brief outline & I’m sure mathematicians, physicists & audio engineers could provide more details.

Some will say, but that method does not take into account the many variables of human hearing. Yep, and that’s the whole point of the scientific methods, i.e., to remove human bias when trying to analyze objective reality. i.e., In general, science is about discovering objective reality, not trying to do the impossible & make objective reality conform to human individual & highly variable subjective experience (even though a less calibrated subjective experience believes that's possible). The physics of objective sound, e.g., the bang of a drum is also not taking human subjective perception into consideration (before it enters our ears). Therefore the human variable is actually irrelevant if we want to have a high degree of confidence that headphone “X” is producing more accurate “real world” audio . E.g., Subjectively we’d all hear (perceive environmental information) the accurate “X” headphone differently (within the subjective variance of our species subjective perception), just as we hear the bang of a drum differently ( environmental\ecological objective sounds differently). But at least we’d be more confident that the “neutral headphone” was producing the sound of the drum more accurately, & then our ears (biology) would analyze that sound using our own subjective experience. i.e., a more natural sounding headphone. Of course there are many acoustic variables to consider when recording the bang of a drum, e.g., environment the drum is in, location to the recording device. However, due to the reason why we hear, i.e., evolutionary psychological functional adaptions, the objective audio should be a sound from our natural environment as we are already more adapted to hear those sounds more accurately.

Of course the above method will not be a “cash cow” in a consumerist driven market place, in the context of different headphone manufactures maintaining the “the next product is better than the last” marketing strategy & headphone manufactures competing with one another for “the best headphone ever” strategy. Maybe they could, ultimately, do us all a favour & compete to produce the headphones with the “X sound signature” using more ecologically sustainable manufacturing methods.

Fortunately headphone manufacturers don’t tend to sell scientific instruments as marketing gimmicks and spin (lies if it’s intentional). i.e., science needs accurate instruments. Though personal bias has significantly corrupted the publics perception of the sciences that are related to anthropogenic climate change. e.g., ecolology, climatology. Thanks “humanity as usual” , you’ll make losers of us all if your not regulated to behave more ecologically sustainably. i.e., regulate yourselves for the good of yourselves.

As it's been pointed out, the purpose of a headphone being to listen to them, it's hard to just dismiss the severe variations that can manifest in the sound reaching the eardrum of one person compared to the eardrum of another. Or even more important, the variations that the listener will not get(because of the headphone bypass) that the brain knows as "real sound". At the very least the concept of flat doesn't exist on headphones/IEMs. We can still measure and care about other variables, and of course a purely objective approach should exist and sort of does, but it still brings up questions that will inevitably involve humans and make it complicated.

Let's say I measure a headphone, what variable would I give priority to?
-Noise isolation? If so, what would be the standard ambient noise reference?
-THD?
-IMD?
-Some other multi-tone measurement to better simulate music? <= probably a better choice subjectively
-Some specific piece of music? which one?

The straightforward approach would be to just go with measuring amplitudes of the variations. But how do we quantify the value of each type of variation? If headphone A changes one frequency at higher amplitude but headphone B changes many frequencies at a slightly lower amplitude, which one measures better? which one is better?

Then there is frequency vs time. We have sine waves, so both axis have the same objective worth. But listeners usually don't notice phase shift from the transducer, while amplitude makes for most of our subjective impressions and preferences. knowing that, should we treat them equally anyway?
those are the types constant problems we have to face. The purpose of a headphone isn't to peel potatoes in the most effective way, its purpose is to feel right(or maybe just pleasing) to a user. Ignoring body variations or subjective preferences might rapidly lead to objective but useless measurements in this particular context.

Harman proposes a standard for frequency response that pleases most people. I for one would agree to replace the usual diffuse field compensated target we see everywhere that almost nobody likes or feel to be neutral, with the Harman curve that at least some people(allegedly most people) prefer. But that's still just the frequency response and while it has been shown to be a major variable in term of how listeners judge sound quality, it's a small part of the data from music played into the headphone. Objectively it seems wrong to only rely on that. Subjectively, it works rather well most of the time, although each user need to get somewhat familiar with a specific measurement rig, and learn by listening to a bunch of the headphones measured so he can find some correlations with the graphs.


TLDR: it's complicated.






Ps: Among my cheap little toys, I have the miniDSP E.A.R.S mentioned by @KeithPhantom. To check variations between headphones(or placement of the same headphone), it does the job. But as some sort of objective standard, the "calibrated" compensations are

It's pretty convenient for headphone comparisons, and pretty bad for IEMs.