[aaf evo]

Fwiw, this was also my conclusion, but I also don't listen super loudly, so wasn't too limited by the dongle's limited output power.

I don't either, perhaps thats what makes it work. Either way I won't be blasting my ears to be shown otherwise

 

[Doltonius]

I'm referring to science in general, and here the conclusions that are being drawn by folks based on the available research clearly exhibits a similar bias.

Also, wild guess but I gather that the folks writing papers for B&K are more closely aligned with academia than audiophile forum gossip.

B&K just produces the measuring rig, which only does one job, i.e., to measure the FR. I don't think B&K having a quantitive bias is problematic; they only work with objective data after all.

Thus I think it is totally appropriate to talk about bias in the community, i.e., audiophiles, reviewers, and manufacturers. Then the dominant bias in decades is one against quantitative data.

 

[Chang]

B&K just produces the measuring rig, which only does one job, i.e., to measure the FR. I don't think B&K having a quantitive bias is problematic; they only work with objective data after all.

Thus I think it is totally appropriate to talk about bias in the community, i.e., audiophiles, reviewers, and manufacturers. Then the dominant bias in decades is one against quantitative data.

They publish research around how standards like 5128 are created. Most of the research that's referenced in the objective side of the audio space are all white papers that are made by institutions and organisations that are very much more academically inclined than hobby focused. What I mean to say is the the bias towards subjectivity in the hobby spaces likely has next to no impact on the current pool of research in this field, despite it's commonality in discourse in our spaces.

 

[Doltonius]

I'll chime in just once. We have been following the research as well and though there are indicators that FR at DRP is a large contributing factor to the listening experience, the notion that it is the be-all and end-all is far from conclusive.

This is something we have tested internally because if it was in fact the reality we would just close shop. No point investing this amount of time and energy if that level of performance can be achieved purely through FR. By that notion the most effective means of achieving TOTL audio would be TWS with a reasonably low distortion driver and full DSP. Or get a Q5k with any budget chi-fi and EQ with sweeps to your personal HRTF and be done with it.

And in fact that's the exact thing I did with several IEMs both budget and high-end. EQ them with sweeps, white noise and music to as close as I could get to STORM, while cross-referencing measurements on a both a 711 and a KB501x equivalent. The counterclaim is that this can't be done without measurements with the specific IEMs with my personal HRTF, but this is as close as I can get with the resources on hand. The listened experience tonally was within margin of error closer than 1db variance compared to when we're doing listened QC after doing measured QC when we batch our IEMs. Either way, what myself as well as the rest of the team heard when we did this led us to continue our development. I think a quick test anyone can do is just EQ a low pass on STORM with a 5db reduction on everything after 10k as suggested by Resolve's 5128 measurement and you'll find that a lot of the resolving ability is retained, still surpassing most IEMs with more upper treble energy, sustained or otherwise. Curiously enough our internal definition of true resolution is ability to render detail sans frequency response. Meaning to say that if the detail can be EQed in or EQed away then that perceived "detail" was artificially induced to begin with or brought to attention via FR emphasis as Resolve described.

I think my two takeaways from the objective space of the hobby in general are 1) I like that it's pushing for more research, because research as it pertains to better understanding our psychoacoustic system with regards to experienced sound quality and understanding of the correlation between in-ears, earbuds, headphones and speakers has been sorely lacking. The 5128 standard has been huge leap in this aspect but I still don't think averaging a population sample in the way that it was done is particularly effective for IEMs. I also like that the objective space better aligning the research space with industry in aspects to index for. And in general the more quality data we have available the more we are able to improve what we make.

And 2) I really don't like the dismissal of qualitative inputs, just as I don't like the dismissal of quantitative representations and data. I think it's quite foolish to presume that millions of years of evolution in our auditory system and our brain's ability to do signal processing can be captured in a single metric. While the current research does support this idea to an extent and point in the general direction of FR at DRP being a huge contributing factor to perceived sound quality. I think it's far from conclusive that it's the only relevant metric, and to presume so is quite foolish and arrogant in my opinion. I think that research in any area, and especially in understudied niche areas like audio as pertains to sound quality, has a tendency to go through cycles of accepted schools of thoughts before new paradigms are introduced. Historically research in any field is based on research that has previously been done and as such has a precedented tendency of having some inbuilt confirmation bias, so that's also something that is worth considering. Bias against qualitative data as inferior to quantitative data is also a particular sore spot for me as it has delayed research in mental health by decades, something that has affected me personally, and has also resulted in things like reports of pain and symptoms by minorities being more commonly dismissed, resulting in systematic level corrective changes taking far too longer than needed to be implemented because qualitative evidence wasn't taken seriously. Of course that's not to say that all subjective opinions are equally valuable, relevant or valid, but simply that those that considered and deliberate in controlling for variables should not be so simply dismissed.

I think a simple question we can ask is what we want out of this hobby, and by extension why we make the purchase decisions that we do. If you do buy into the idea that FR at DRP is everything, then great, the natural conclusion to this assumption is buy a $300 Topping stack or Q5k with any cheap, low distortion transducer and EQ with sweeps to your preferences and quit. Why buy Susvara when there are cheaper Hifimans that essentially measure the same. Or why buy any modern amp, DAC or cable when all of them have essentially zero impact on frequency response. Soundstage is also not a thing so just buy a cheaper IEM for the most bang for buck. Congratulations, you've just saved yourself a ton of money and concluded audio, and 95% of the headphones.com store is now irrelevant to you and spending a dime more is a fool's errand.

But for the rest of us that's not enough, and we'll have our fun continuing to explore the space. With our money. That we earned. That we are spending. You have finished audio and that's great, feel free to see yourself out. But for the rest of us there is still plenty we'd like to explore and continue to chase our unicorns of "perfect" and "better", whether anyone else sees it as placebo or not, my listening enjoyment and the peace and happiness it brings is enough for me.

Also mods, I think this discussion is an interesting and healthy one to have but I'd like to humbly request it be moved to a separate thread if it continues so as not to detract from the STORM discussion

I think as a whole this is slightly (maybe not just slightly) misrepresenting the other side of the debate. I don't think the divide is even quantitative vs qualitative bias. The other side has never challenged differences in subjective perception, i.e., resolution is real, soundstage is real, slam is real, etc, just because there is no data that directly tell us about them; or at least that is not the central issue for them. But "how far one abides by the Occam's Razor", or what explanations we give to these differences.

We know that FR has a large impact on basically all properties of subject perception, this is established beyond doubt. Now the question is, "are there other factors that are not yet measured also playing a part in subjective perception". One of the sides will answer, "no, at least for now we don't have such evidence", because we have not been able to eliminate FR as a confounding factor. That doesn't mean we don't believe that these subjective differences are real.

In the case of "Storm eq-ed to ie900 is still more resolving than a real ie900", we don't want to claim that this subjective perception is placebo or made up. We simply say, "by the technology of today, you cannot ensure that Storm eq-ed to ie900 really matches the real ie900 in FR at the eardrum". Thus, the difference in the subjective perception of detail might still be due to small differences in FR at the eardrum.

 

[Doltonius]

They publish research around how standards like 5128 are created. Most of the research that's referenced in the objective side of the audio space are all white papers that are made by institutions and organisations that are very much more academically inclined than hobby focused. What I mean to say is the the bias towards subjectivity in the hobby spaces likely has next to no impact on the current pool of research in this field, despite it's commonality in discourse in our spaces.

The 5128 standard, or other industrial standards, is just to make measurements commensurate and comparable with one another, isn't it? it does not try to say anything about preferences or subjective evaluation at all. that they are not hobby inclined is not a problem. they don't need to be. they never dismissed "soundstage/resolution/slam/dynamics/etc" in their papers. they just discuss, "how best to measure FR". whether or not those descriptions are real or represented in the FR is a completely separate issue. I am puzzled why you see a problem in that.

On the other hand, harman's research clearly incorporates subjective evaluations. Their target's are literally average preference targets. Though indeed i don't think they are very successful at least in the iem realm; looks like it just doesn't sound that good to many people. something must be wrong with their methodology. the point is that research that incorporates subjective evaluation does exist, successful or not.

 

[Chang]

I think as a whole this is slightly (maybe not just slightly) misrepresenting the other side of the debate. I don't think the divide is even quantitative vs qualitative bias. The other side has never challenged differences in subjective perception, i.e., resolution is real, soundstage is real, slam is real, etc, just because there is no data that directly tell us about them; or at least that is not the central issue for them. But "how far one abides by the Occam's Razor", or what explanations we give to these differences.

We know that FR has a large impact on basically all properties of subject perception, this is established beyond doubt. Now the question is, "are there other factors that are not yet measured also playing a part in subjective perception". One of the sides will answer, "no, at least for now we don't have such evidence", because we have not been able to eliminate FR as a confounding factor. That doesn't mean we don't believe that these subjective differences are real.

That's a sentiment I totally agree with, however, other statements in this thread indicate otherwise, in that all of those aspects are encoded in FR, which is a very different statement from we haven't been able to eliminate FR as a confounding factor.

I'm very much in agreement with your position now that you have elaborated, just that for now my experiences in doing my best to eliminate FR as a variable has me leaning towards believing what you suggested in that there are other factors not yet quantified that play a part in the listening experience. In this instance I very much believe Occam's Razor leads to the wrong conclusion, but I neither have the time, resources or expertise to run research to determine what that might be. Which is why in general all the recent research that has been done is very very interesting and I hope to see more.

The 5128 standard, or other industrial standards, is just to make measurements commensurate and comparable with one another, isn't it? it does not try to say anything about preferences or subjective evaluation at all. that they are not hobby inclined is not a problem. they don't need to be. they never dismissed "soundstage/resolution/slam/dynamics/etc" in their papers. they just discuss, "how best to measure FR". whether or not those descriptions are real or represented in the FR is a completely separate issue. I am puzzled why you see a problem in that.

On the other hand, harman's research clearly incorporates subjective evaluations. Their target's are literally average preference targets. Though indeed i don't think they are very successful at least in the iem realm; looks like it just doesn't sound that good to many people. something must be wrong with their methodology. the point is that research that incorporates subjective evaluation does exist, successful or not.

Not a problem with the systems in and of themselves but the conclusions drawn from the data they are now generating, specifically in reference to the assumption that FR at DRP encodes all aspects of sound quality, and the writing off of a whole ocean of subjective experiences (much of which is useless admittedly), but where certain trends can still be noticed and have a decent chance of being statistically significant because they run contrary to this assumption. I think in the current state of things it would still be best to pursue this assumption more as it will most efficiently lead to the answers we are looking for, what I'm concerned about is the real possibility of studying ourselves into a corner where alternative possibilities are prematurely dismissed in favor of reinforcing what we already know.

 

[Doltonius]

That's a sentiment I totally agree with, however, other statements in this thread indicate otherwise, in that all of those aspects are encoded in FR, which is a very different statement from we haven't been able to eliminate FR as a confounding factor.

To verify that there are other factors not yet measured at play in subjective perception, it is necessary to eliminate FR as a confounding factor. When we make sure that two iems have the same FR at the eardrum, but there is still a difference in subjective perception that survives double-blind tests, we can conclude beyond doubt that there are other un-quantified factors at play. Before then, Occam's Razor will dictate that we don't entertain their existence; and the statement we entertain is that "everything is contained is what is already measured".

In this instance I very much believe Occam's Razor leads to the wrong conclusion

Occam's Razor is never wrong; because it says "Entities must not be multiplied beyond necessity". Once it is deemed necessary to entertain new assumptions and new causal effects, i.e., through removal of confounding factors, then even under Occam's Razor, we should happily entertain the existence of these new things.

but the conclusions drawn from the data they are now generating, specifically in reference to the assumption that FR at DRP encodes all aspects of sound quality

But that is not related to the quantitative bias of B&K. That is related to Occam's Razor of people who interpret the results, likely people outside the academia and in the audiophile community. B&K never claimed that their measurements provide the full answer to sound quality.

As long as one is not randomly dismissing a difference in subjective perception just because it is not reflected yet in measurements, I don't there is a big problem. Whether they attribute it to "unmeasured differences in FR" or "unmeasured differences outside of FR" plays little role on how the industry will progress. They are not measurable at the moment in any way, and so one would have to rely on qualitative things, including subjective perception, heuristics about drivers and acoustic designs, etc.

 

[jwilliamhurst]

To verify that there are other factors not yet measured at play in subjective perception, it is necessary to eliminate FR as a confounding factor. When we make sure that two iems have the same FR at the eardrum, but there is still a difference in subjective perception that survives double-blind tests, we can conclude beyond doubt that there are other un-quantified factors at play. Before then, Occam's Razor will dictate that we don't entertain their existence; and the statement we entertain is that "everything is contained is what is already measured".

Occam's Razor is never wrong; because it says "Entities must not be multiplied beyond necessity". Once it is deemed necessary to entertain new assumptions and new causal effects, i.e., through removal of confounding factors, then even under Occam's Razor, we should happily entertain the existence of these new things.

But that is not related to the quantitative bias of B&K. That is related to Occam's Razor of people who interpret the results, likely people outside the academia and in the audiophile community. B&K never claimed that their measurements provide the full answer to sound quality.

As long as one is not randomly dismissing a difference in subjective perception just because it is not reflected yet in measurements, I don't there is a big problem. Whether they attribute it to "unmeasured differences in FR" or "unmeasured differences outside of FR" plays little role on how the industry will progress. They are not measurable at the moment in any way, and so one would have to rely on qualitative things, including subjective perception, heuristics about drivers and acoustic designs, etc.

Blah blah blah, can we move these discussion’s please? I am not in this thread for this!

 

[Doltonius]

Blah blah blah, can we move these discussion’s please? I am not in this thread for this!

Of course you will choose to target the side in this debate that you don't agree with to say this.

 

[warrenpchi]

Of course you will choose to target the side in this debate that you don't agree with to say this.

It's not about taking sides in a debate, it's about the debate in general being off-topic for the Storm thread.

 

[Resolve]

I think a quick test anyone can do is just EQ a low pass on STORM with a 5db reduction on everything after 10k as suggested by Resolve's 5128 measurement and you'll find that a lot of the resolving ability is retained, still surpassing most IEMs with more upper treble energy, sustained or otherwise. Curiously enough our internal definition of true resolution is ability to render detail sans frequency response. Meaning to say that if the detail can be EQed in or EQed away then that perceived "detail" was artificially induced to begin with or brought to attention via FR emphasis as Resolve described.

I sort of stopped thinking about it as 'detail' specifically, and started thinking about it in terms of like... what do I enjoy. For me, it's about fixing the issues with the response (for any IEM) that don't jive with my anatomy. So I would actually say that provided the HRTF features my brain expects are reasonably replicated at my ear drum, a wide Q treble boost is precisely what enhances or brings the upper edges of certain tones more to the forefront, and in a way that could be enjoyable.

But of course, that's just me... for others, it may be more to what Precogvision has said in his description of detail - that detail is tasteful coloration, and that various features of the FR can cause a given detail in the music to stand out, and that makes sense to me too.

 

[Chang]

I sort of stopped thinking about it as 'detail' specifically, and started thinking about it in terms of like... what do I enjoy. For me, it's about fixing the issues with the response (for any IEM) that don't jive with my anatomy. So I would actually say that provided the HRTF features my brain expects are reasonably replicated at my ear drum, a wide Q treble boost is precisely what enhances or brings the upper edges of certain tones more to the forefront, and in a way that could be enjoyable.

But of course, that's just me... for others, it may be more to what Precogvision has said in his description of detail - that detail is tasteful coloration, and that various features of the FR can cause a given detail in the music to stand out, and that makes sense to me too.

So once you hit perfect compliance with your preference curve to your HRTF given low enough distortion, that's the endgame? Seems pretty achievable right now with EQ

 

[Resolve]

So once you hit perfect compliance with your preference curve to your HRTF given low enough distortion, that's the endgame? Seems pretty achievable right now with EQ

We talk a lot about measurements, but at the end of the day it's the experience that counts, right. So I won't say this is necessarily how it should be for everyone, but for what I personally enjoy most these days, it tends to work out that way yeah.

And by no means will I say I have all the subjective qualities to the experience figured out. Like there are times when I hear something that sounds more "punchy", and the flip side, something sounds more "soft", and then I feel the need to try and go down the rabbit hole and figure things out, because I really don't know what's responsible. I like to keep an open mind about stuff, like maybe there is a separate acoustic property not being measured currently - I'd certainly like to find that if that's the case. But at the very least, there has been some progress on correlating various subjective parts of the experience that people enjoy to existing datapoints, and I expect there's more to be found there.

But yeah, for me personally, basically if I can solve for HRTF, HpTF, and preference, all with EQ, it typically ends up working out for me, provided things like harmonic distortion and excess GD aren't doing anything weird.

 

[Chang]

We talk a lot about measurements, but at the end of the day it's the experience that counts, right. So I won't say this is necessarily how it should be for everyone, but for what I personally enjoy most these days, it tends to work out that way yeah.

And by no means will I say I have all the subjective qualities to the experience figured out. Like there are times when I hear something that sounds more "punchy", and the flip side, something sounds more "soft", and then I feel the need to try and go down the rabbit hole and figure things out, because I really don't know what's responsible. I like to keep an open mind about stuff, like maybe there is a separate acoustic property not being measured currently - I'd certainly like to find that if that's the case. But at the very least, there has been some progress on correlating various subjective parts of the experience that people enjoy to existing datapoints, and I expect there's more to be found there.

But yeah, for me personally, basically if I can solve for HRTF, HpTF, and preference, all with EQ, it typically ends up working out for me, provided things like harmonic distortion and excess GD aren't doing anything weird.

Also would like you opinion on the preference bounds. It's clear that there are quite a lot of people who like signatures that lie quite far outside those bounds. I see the usefulness of complying to a variant of DF tilt for mass market products to make sounding "normal" sounding and widely acceptable. But in you opinion are tunings that lie outside those bounds simply not good, not relevant or should not be catered to?

 

[Resolve]

Also would like you opinion on the preference bounds. It's clear that there are quite a lot of people who like signatures that lie quite far outside those bounds. I see the usefulness of complying to a variant of DF tilt for mass market products to make sounding "normal" sounding and widely acceptable. But in you opinion are tunings that lie outside those bounds simply not good, not relevant or should not be catered to?

So we're not actually using a tilted DF target - maybe you're already aware but just wanted to clarify. We're calibrating the response to the flat DFHRTF for JM-1, which is an approximation of population average pinna effects and canal transfer function, and then showing that relative to the preference bounds. The 10dB tilt is the dotted line there which primarily just indicates bass to treble delta, and isn't actually the 'target' in any traditional sense.


But to answer your question, this is just the established preference groups from the research, and there were absolutely outliers. So I don't treat as like "if it goes outside the bounds that's always bad", but merely just that it's more likely to be perceived as a coloration of some kind, depending on the person (and also not guaranteed to be perceived as 'bad'). I also don't think anything within the bounds is guaranteed to be preferable, ideally there should be more specific tolerances around each cluster based on the dendrogram. But that's very difficult to show without introducing a lot of visual clutter. We're still working on improving that whole visualization but it's a start.