Both my education and my own experiences (with music as well as with humans) 100% align with what you're saying. I'm curious, then, why do so many people claim to hear things that shouldn't be possible to hear? And I'm sure most of them are not liers, for them their experience is true. Is confirmation bias this strong? And why doesn't it affect me the same way?
mostly I'd say deplorable testing conditions. after that, it's too late. it's the same as doing very motivated and serious work based on a false axiom. if the starting point was a mistake, it probably invalidates everything else. of course a listener will get impressions from his experience, and of course he will honestly believe he's hearing what he thinks he's hearing. but if those impressions come from experiments of dreadful quality, we can only reject them entirely or at the very least put a low confidence index in front of the results.
so to answer your question, the most inconsistent and uncontrolled the experience, the more likely we are to get vastly different feedback from listeners. and why you're not affected that way? IDK, maybe you are but your bias is different, as in you already know what to expect based on measurements, or something like that? maybe you have poor listening skills? maybe there is nothing to hear and you lack imagination^_^? really, IDK.
all that of course doesn't mean 2 DACs can never be audibly different. but if the testing method is crap, the only reasonable conclusion until we procure more conclusive testing conditions, is that we can't say. my expectation of small or no difference comes from objective measurements and a very small number of controlled tests, that obviously isn't enough to make a claim about all DACs or really any of the DACs I never got to test properly(so most of them).
and I guess on a few occasions some guy somewhere really has a talent when it comes to listening and he will pick up stuff most people wouldn't(we'd all like to think we're that guy, we usually are not
).
