Audiophile objections to blind testing - an attempt from a layman

[gregorio]

I've always been fine with seeking the possible limits.

Hmm, I’m not sure, maybe, as long as we’re very careful how we define “possible”. For example, it’s “possible” we can hear up to 190dBSPL, although not for long because it will likely cause death. Causing death surely rules it out from being “possible”? What if say 170dBSPL doesn’t cause death but only a coma for a few weeks, does that mean 170dBSPL is “possible”, even though it’s highly undesirable? Where do we draw the line? If death or coma rule it out as “possible”, what about if it causes pain and permanent hearing damage after only a couple of seconds? Personally, I’d rather go with “practical limits” than “possible limits” because causing coma or pain and permanent hearing damage might be considered “possible” but are certainly impractical.

Conclusive is about having some confidence in the validity of the result(because of the testing protocol, and interpretation model).

The definition of “conclusive” I use and as defined by English dictionaries is: “proving something in a way that is certain and allows no doubt” (Cambridge Dictionary) or “putting an end to debate or question especially by reason of irrefutability” (Merriam-Webster). A single or couple of test subjects getting 10/10 (or even 16/16) in an ABX provides evidence and quite high confidence but does it irrefutably prove it or allow no doubt? What about if we have conflicting results from someone else’s ABX or a lot of other ABX’s, what about if what’s being tested is theoretically just beyond the audibility thresholds, is it still irrefutable proof that allows no doubt?

And Amir as far as I remember, always said he was looking for the limit of hearing. ….But that was about his desire, not that he didn't understand the actual limits of the testing conditions. I think.

I definitely don’t want to rehash it again but my beef with him was that he indeed didn’t understand the limits of human hearing or of music performance or at least he appeared not to. Given his level of general audio knowledge, I think he did understand and was deliberately misrepresenting the facts but I can’t be sure of that. We agreed that ~0dBSPL was the minimum limit of human hearing, not so much that 120dBSPL is the maximum but where we disagreed is that he took that to mean human hearing has a dynamic range of 120dB and more so that it was also the DR of live music. Of course, the peak SPL figure can only also be the dynamic range figure if the noise floor is 0dbSPL but in those conditions where we *might* withstand 120dBSPL or encounter it in live music performances, the noise floor is never 0dBSPL or even close to it.

G

 

[FunkyBassMan]

Do you honestly think that considering/discussing the validity of a methodology influenced by vested interests/agendas is neither scientific nor part of the scientific community? Is a knowledgeable person like yourself honestly unaware of the history of scientific papers funded by the tobacco and fossil fuel industries and numerous other examples in the medical and other industries, including in the audio industry? Or, are you deliberately misrepresenting the scientific community and mischaracterising my responses as not scientific?

@GoldenSound is 100% right. Yes, examining the validity of a methodology is scientific. But what is not scientific is hiding behind vague insinuations of ‘vested interests/agendas’ without presenting a shred of evidence to substantiate those claims. Science is grounded in facts, not basseless accusations or rhetorical hand-waving. If you believe this methodology is flawed due to undue influence, then lay out your case clearly: provide evidence of bias, explain the specific methodological errors, and demonstrate how they undermine the conclusions. Anything less is pure speculation.

Bringing up historical misconduct in science—like studies funded by the tobaco industry—does nothing to support your argument here unless you can draw a direct and substantiated parallel to this case. Simply invoking the specter of ‘agendas’ is a pathetic tactic, an attempt to dismiss valid work without addressing its substance. If you have no specific evidence about the methodology itself, then you’re not engaging in science; you’re engaging vapid ad hominem rhetoric.

 

[Davesrose]

@GoldenSound is 100% right. Yes, examining the validity of a methodology is scientific. But what is not scientific is hiding behind vague insinuations of ‘vested interests/agendas’ without presenting a shred of evidence to substantiate those claims. Science is grounded in facts, not basseless accusations or rhetorical hand-waving. If you believe this methodology is flawed due to undue influence, then lay out your case clearly: provide evidence of bias, explain the specific methodological errors, and demonstrate how they undermine the conclusions. Anything less is pure speculation.

Bringing up historical misconduct in science—like studies funded by the tobaco industry—does nothing to support your argument here unless you can draw a direct and substantiated parallel to this case. Simply invoking the specter of ‘agendas’ is a pathetic tactic, an attempt to dismiss valid work without addressing its substance. If you have no specific evidence about the methodology itself, then you’re not engaging in science; you’re engaging vapid ad hominem rhetoric.

I dunno, I've been on here long enough to know that you like to be a rabble rouser. And often times just to tread on gregorio's posts. Now I'm not the best fan of gregorio: he can be emotionally immature-not understanding a post and providing his own random outlined responses with asinine side chatter. In one exchange with him, we spent a few pages as to him not understanding context of stream vs sub-stream when it came to Dolby Atmos audio tracks in video files (of which, I'm versed). But even though he won't concede to me knowing about consumer video/audio formats, I've accepted he's an audio engineer that knows the professional standards outside of consumer delivery.

What gregorio brought up for GoldenSound is valid as far as how it's a site about videos that are more about the subjective impressions controlled measurements or bind testing.

 

[FunkyBassMan]

I dunno, I've been on here long enough to know that you like to be a rabble rouser. And often times just to tread on gregorio's posts. Now I'm not the best fan of gregorio: he can be emotionally immature-not understanding a post and providing his own random outlined responses with asinine side chatter. In one exchange with him, we spent a few pages as to him not understanding context of stream vs sub-stream when it came to Dolby Atmos audio tracks in video files (of which, I'm versed). But even though he won't concede to me knowing about consumer video/audio formats, I've accepted he's an audio engineer that knows the professional standards outside of consumer delivery.

What gregorio brought up for GoldenSound is valid as far as how it's a site about videos that are more about the subjective impressions controlled measurements or bind testing.

gregorio knows a lot, yes. He also has made some outrageously wrong statements, insulting anyone who points out his wrongness.

And this is another case. Here, the issue is simple. The fact that someone works in an industry they're passionate about does not constitute evidence of dishonesty as gregorio and others here constantly allege.

 

[Davesrose]

gregorio knows a lot, yes. He also has made some outrageously wrong statements, insulting anyone who points out his wrongness.

And this is another case. Here, the issue is simple. The fact that someone works in an industry they're passionate about does not constitute evidence of dishonesty as gregorio and others here constantly allege.

Yes, I'll agree that gregorio likes to start with insulting anyone he comes across. At some points when I've been cordial, I've learned about how professional mixing happens. Would love if there was an exchange of interest about how cinematography works or video processing: but no, he once indicated he's only concerned with his niche in pro audio. It came to a head the first time I butt heads with him when he revealed he didn't know blu-ray standards. From what I remember, our first exchange was audio standards with blu-ray, and he didn't realize it was lossless audio standards (he thought blu-ray could only do DD+ until I informed him the common protocol was DTS HD). Most recently it's been Dolby Atmos with UHD. So he told me he's done one Atmos mix....and he passed his master file on to a video production company and that was it. Then his official statement for anyone asking about how Atmos works with home video is that "well it's a complex system which you can't understand". No actually, with encoded video files for consumer formats, it's rather simple. There's a surround channel stream that's 5.1/7.1 and then a JOC Atmos stream that's 16 dynamic channels.

While gregorio might be uninterested about AV, his statements here are valid as far as science and eliminating bias.

 

[gregorio]

But what is not scientific is hiding behind vague insinuations of ‘vested interests/agendas’ without presenting a shred of evidence to substantiate those claims.

Huh, didn’t you even read the very first paragraph of the message to which you were responding? Which was not a “vague insinuation”, it detailed exactly his vested interest!

If you believe this methodology is flawed due to undue influence, then lay out your case clearly: provide evidence of bias, explain the specific methodological errors, and demonstrate how they undermine the conclusions. Anything less is pure speculation.

I’ve very clearly laid out my case against his methodology, the errors and how they undermine conclusions, what on earth are you talking about?

Bringing up historical misconduct in science—like studies funded by the tobaco industry—does nothing to support your argument here unless you can draw a direct and substantiated parallel to this case.

How is my argument that science and the scientific community does consider and discuss vested interests, that full disclosure is now a requirement in peer reviewed scientific papers, not supported by the studies funded by the tobacco industry when it was largely those tobacco funded papers which inspired that discussion and requirement?

If you have no specific evidence about the methodology itself, then you’re not engaging in science; you’re engaging vapid ad hominem rhetoric.

Firstly, you must be joking, my post was packed with specific explanations and examples of why his methodology is so fundamentally flawed and Secondly, even if I hadn’t even mentioned “the methodology itself”, how is it not acceptable in a science discussion forum to do what the science community itself does and consider/discuss vested interests in regard to the methodology of scientific/objective tests?

gregorio knows a lot, yes. He also has made some outrageously wrong statements …

You just keep repeating that over and over and yet every single time you’ve accused me of making “outrageously wrong statements”, it was demonstrated that you were one who was outrageously wrong, on one occasion even by the video you yourself cited, that was funny!

It’s just the same old story, you don’t even read, let alone understand the posts you’re responding to, and then you accuse me of doing what you’re guilty of! Don’t you ever get bored of humiliating yourself?

G

 

[castleofargh]

Hmm, I’m not sure, maybe, as long as we’re very careful how we define “possible”. For example, it’s “possible” we can hear up to 190dBSPL, although not for long because it will likely cause death. Causing death surely rules it out from being “possible”? What if say 170dBSPL doesn’t cause death but only a coma for a few weeks, does that mean 170dBSPL is “possible”, even though it’s highly undesirable? Where do we draw the line? If death or coma rule it out as “possible”, what about if it causes pain and permanent hearing damage after only a couple of seconds? Personally, I’d rather go with “practical limits” than “possible limits” because causing coma or pain and permanent hearing damage might be considered “possible” but are certainly impractical.

I don't see the problem. The result will be tied to the peculiar conditions that achieve it, and won't change the limits found under other more reasonable conditions. If you find volunteers and a group of ethic validating the experiment, which might be the tricky part here, I say go ahead!

The definition of “conclusive” I use and as defined by English dictionaries is: “proving something in a way that is certain and allows no doubt” (Cambridge Dictionary) or “putting an end to debate or question especially by reason of irrefutability” (Merriam-Webster). A single or couple of test subjects getting 10/10 (or even 16/16) in an ABX provides evidence and quite high confidence but does it irrefutably prove it or allow no doubt? What about if we have conflicting results from someone else’s ABX or a lot of other ABX’s, what about if what’s being tested is theoretically just beyond the audibility thresholds, is it still irrefutable proof that allows no doubt?

Yes, no, maybe. Almost nothing is irrefutable, clearly science still makes progresses and learns by lowering the bar of what counts as conclusive.

ABX is great to have around and discover things about ourselves, I'd say it's great for this hobby(although I read something about AXB being more accurate than ABX). But as I associate conclusive with how reliable the test and data analysis is, someone telling me he passed an ABX on his own, doesn't have that much of a value to me. I used my graphs to support my views many times, I don't think I ever have used my ABX results the same way. That tells you what you want to know, I do not consider my abx results as conclusive for others. They are for me, though.
I certainly won't call AES or some medical organization if something we considered impossible to hear, is passed by somebody in a personal ABX test. If we're talking research and findings that go against accepted knowledge, the burden of proof must get upgraded, and so does the testing protocol. I won't fight you on that.

But at the same time, when @GoldenSound passes something that might require hearing 20kHz or a little over it, he's still young, I don't have a hard time accepting that his abx is conclusive about him hearing those differences at those frequencies with the right signal. It's well within the realm of possibilities, and I don't see the point of doubting him just for the sake of doubting.
Does that vindicate any random 40+ audiophile empty claiming he also hears stuff above 20kHz? Never! Goldy's results are not conclusive for the human race, but they might be for himself in his setup at his listening level, if they're consistent repeatable results? If so, they mean something. Narrow the conditions of the conclusion or the population it applies to until it is a valid result. Or ask for details if they're lacking in some way. I think we have enough options to find something meaningful from almost any controlled experiment.

I'm not big on the experimenter being the subject, or sample sizes of 1. If you want to reject results on that basis, I also won't fight you.(I did start saying yes, no, maybe! ^_^).


But to get back to this specific situation, you're giving a hard time to Goldy and in passing to Amir when they're the only ones in the hobby right now who have some media presence, and try to rely on actual listening tests.
You have real research that couldn't care less if the new DAC is whatever(rightfully so IMO), and right behind you have those 2, trying to popularize blind testing as a natural part of a review, while the competition mostly keeps making the dumbest excuses to not even volume match equipments before ABing them sighted.
Is it satisfying to consider that some guys 3000 years ago had more common sense? Not really. Does it make the hobby look exactly like what it is? Yup.
So maybe, go easy on the 2 that try to move the needle in the right direction and have some power to do it. Money incentive or not, marketing relations or not, they're doing something better than the competition.
And any time you find them overreaching in their conclusions(I often do) or mixing sighted subjective things and unverified beliefs to their blind test analysis, you can point it out. But before doing so, remember that if not them, then it's Alpha Audio, Hans Beeksomething, and random guys doing what I call review therapy, where they let go of their mistakes and all their guilt, by blaming everything on the gear.

Perfection isn't human. I've just contradicted myself so much that it's kind of impressive. Yes, no, maybe!!!!!

 

[gregorio]

The result will be tied to the peculiar conditions that achieve it, and won't change the limits found under other more reasonable conditions.

Right, so if you just say something is “possible” to discern in a DBT without mentioning that it would for example cause hearing damage or death and don’t even mention that there are other “limits found under other more reasonable conditions” and it’s only these limits that would be applicable to consumers, wouldn’t that be dishonest? In fact, I don’t see the point of even mentioning what is “possible” under “peculiar conditions” that will cause say hearing damage or death, it may be of interest to science or the military but not to consumers who would be doing something drastically wrong if they’re actually achieving those “peculiar conditions”.

Yes, no, maybe. Almost nothing is irrefutable, clearly science still makes progresses and learns by lowering the bar of what counts as conclusive.

I would say we have plenty that isn’t rationally refutable and that’s proven beyond doubt. 1+1=2, Maxwell’s Laws, Shannon’s Theorem, what sound is, etc.

I used my graphs to support my views many times, I don't think I ever have used my ABX results the same way. That tells you what you want to know, I do not consider my abx results as conclusive for others.

Then I don’t understand. If your quote is true then you would not consider the assertion that passing a DBT “conclusively says "There is no other answer except that this is audible"” to be correct and yet you’re disagreeing with me stating it is not correct?

But at the same time, when @GoldenSound passes something that might require hearing 20kHz or a little over it, he's still young, I don't have a hard time accepting that his abx is conclusive about him hearing those differences at those frequencies with the right signal. It's well within the realm of possibilities, and I don't see the point of doubting him just for the sake of doubting.

I do see the point of doubting him if that “right signal” is say a pure tone or other content at a level that would never exist and/or be damaging in a consumer environment, when reviewing consumer equipment for consumers! There is also reliable evidence that restricting a broadband signal (rather than a “right signal” a consumer will never encounter) with a typical CD linear phase filter doesn’t even register in the auditory cortex (in subjects of similar age and younger). So, I am not doubting him for the sake of it, I’m doubting him because there’s reliable evidence supporting that doubt and also because even with no vested interests, the most honourable of intentions and even if it were made abundantly clear that such a possibility of audibility could never be achieved by consumers when listening to their recordings, still a very significant number of audiophiles would misinterpret that and conclude there is actually scientific/objective evidence supporting their claims of audible differences.

But to get back to this specific situation, you're giving a hard time to Goldy and in passing to Amir when they're the only ones in the hobby right now who have some media presence, and try to rely on actual listening tests.

True and I did state in my first response to GoldenSound that him publishing objective measurements of audio equipment was doing the community a service. However, that service is massively diminished IMHO, if the “actual listening tests” indicate or imply the audibility of differences that are inaudible in practice, thereby supporting the bedrock of false audiophile marketing and snake oil!

G

 

[eq1849]

There are many foods which smell (or taste) confusing or different when tested blind because they have a lot of chemical compounds in common.

A blind test doesn't have to be literally 'blind' as far as I know. E.g. comparing two speaker cables you can still have the cables on display to the listener as long as they don't get any clues whatsoever as to which cable is actually plugged in behind the speaker or amp.


Well, that is the point of a blind test: to ascertain whether we can discern a difference in sound. In order to ascertain whether there actually is a difference in sound we need sound measurements.

Food colored Sprite, to look like Coke, threw me off in a taste test.

I think the act of taking a blind test itself may alter the senses and confuse.

I have tricked myself into forgetting I was testing for a minute or so by doing due diligence tasks. I was then able to tell 16 versus 24 bit with statistical significance. But if aware I’m taking the test it all goes to mush.

 

[eq1849]

The majority of the taste sensation is smell, just ask anyone who has lost their sense of smell. In fact, blind tests for things like wine tasting (or indeed audio) do not remove our senses from the test, they just remove identification of the product, e.g. different amps, 16/44 v 24/96 and so on.

Which in itself may be more harmful than helpful if the senses are dulled when confused.

I think the best is Long Term sighted listening.

In the long term, for me, 16 is flatter and less clear. 24 has more weight and clarity. Since buying 24 bit since 2014 or so this effect has never gone away and is extremely consistent to me.

16 bit with compression sounds better than dynamic 16 bit to me, possibly because it utilizes the more resolute bits.

 

[eq1849]

I agree that there certaintly isn't a 1:1 relationship there - far from it. But would you say that if we took two things that smelled different, but only slightly so, that it would be far fetched to think it would be problematic smelling there being a difference if quickly switched back and forth under our noses?

I mean, I might be totally wrong in my assumption. I know nothing about this stuff at all.

It’s called going “nose blind” by some.

Check out fragrance hobby forums. They talk about it quite often.

Have you ever noticed a pig farm ceases to stink after a couple of minutes. The brain apparently has a peculiar ability in scent, hearing, taste to adapt and numb up. Pain, unfortunately, doesn't work that way it seems.

 

[colonelkernel8]

Which in itself may be more harmful than helpful if the senses are dulled when confused.

I think the best is Long Term sighted listening.

In the long term, for me, 16 is flatter and less clear. 24 has more weight and clarity. Since buying 24 bit since 2014 or so this effect has never gone away and is extremely consistent to me.

16 bit with compression sounds better than dynamic 16 bit to me, possibly because it utilizes the more resolute bits.

lmao. You’re just wrong though. Like this has absolutely no basis in fact and is *literally* a delusion you’ve paid for.

 

[gregorio]

Food colored Sprite, to look like Coke, threw me off in a taste test.

Yes, as explained (and scientifically demonstrated ages ago) the perception of something is a combination of senses, plus knowledge/experience, which is generated by the brain. Therefore changing the sensory input to one sense will affect perception, in this case changing what is seen (the colour) affects our perception when tasting something. Likewise, it has been demonstrated that changing what we hear also affects our perception when tasting something.

I think the act of taking a blind test itself may alter the senses and confuse.

It does not alter the senses, it alters the perception created by the brain from that sensory information and, the whole point of DBT is that it does the exact opposite of confusing, it clarifies! What is more confusing, the brain trying to figure out a perception from a combination of hearing, sight and knowledge/experience or only from hearing? What you “think” is not what the reliable evidence demonstrates!

I have tricked myself into forgetting I was testing for a minute or so by doing due diligence tasks. I was then able to tell 16 versus 24 bit with statistical significance. But if aware I’m taking the test it all goes to mush.

You do indeed appear to have tricked yourself, although you have concluded the wrong cause! It is trivially easy to tell 16bit from 24bit with statistical significance, given the right conditions (which do not match the conditions of consumer listening). Alternatively, it is trivially easy to misinterpret statistical significance and lastly, it’s trivially easy to make some other testing error that would give the same erroneous result, for example not volume matching precisely or comparing 16 and 24bit files that are slightly different in some respect other than just bit depth. However, you have fallen into the age old trap of the “correlation-causation fallacy”: EG. “A” happens at the same time as “B” and therefore “A” was caused by “B”, in this case; “A” - The test “all goes to mush” and “B” - You were not as aware you were taking the test.

This fallacy is shockingly common in the audiophile community and it leads to so many of the audiophile myths. In this case, there are several/many trivially easy alternatives to “B” being the cause which are simply being ignored/dismissed without sufficient or even any reason, not to mention that “A - The test all goes to mush” (you cannot actually tell the difference) may in fact be the correct result!

Doing a scientific test incorrectly or doing it correctly and not getting the result you expect does NOT demonstrate the test doesn’t work or is inferior. It demonstrates a lack of knowledge or understanding of science and of DBTs or a lack of logic or critical thinking, or indeed of all the above!

I think the best is Long Term sighted listening.

Yet it has been repeatedly demonstrated for many decades (to the point of it being established fact for many years) that properly executed DBTs provides not only more sensitivity but also more accurate and more reliable results. So, DBT is clearly best (unless you somehow define “best” as “the wrong results”) and therefore, what you “think” is the opposite of the demonstrated/established facts!

In the long term, for me, 16 is flatter and less clear. 24 has more weight and clarity.

And yet 16bit or 24bit have no concept of “term” or “long term”, the same bits over any “term” are the same bits. Additionally, 16bit is no flatter or less clear than 24bit and does not any have less weight! What you describe is not the symptoms of the difference between 16 and 24bit, it’s the symptoms of a slight difference in volume when comparing, or some other testing error.

Since buying 24 bit since 2014 or so this effect has never gone away and is extremely consistent to me.

If you’re still convinced of the effect, why would it have ever “gone away”? In other words, if the conditions (your belief for example) are “extremely consistent” why would you think the result would not also be “extremely consistent”?

16 bit with compression sounds better than dynamic 16 bit to me, possibly because it utilizes the more resolute bits.

There’s no such thing as “resolute bits”, I presume you mean “least significant bits” (LSBs)? If so, you have this entirely backwards, compression utilises fewer bits (LSBs). There are various psychoacoustic reasons why a sound/mix with more compression maybe perceived as better than the same mix with less compression but utilising more bits is not one of them. Again, it’s just yet another correlation-causation fallacy: “A” - More bits are better, “B” - Compression sounds better, therefore “B” is causing “A” (more bits being utilised).

Don’t you see any problem arguing on the basis of fallacy in a science discussion forum (or anywhere else)?

G

 

[eq1849]

lmao. You’re just wrong though. Like this has absolutely no basis in fact and is *literally* a delusion you’ve paid for.

Well that was convincing…

 

[BS5711]

Well that was convincing…

Unfortunately, no less so than your original assertion.