Head-Fi.org › Forums › Equipment Forums › Sound Science › An interesting take on DBT
New Posts  All Forums:Forum Nav:

An interesting take on DBT

post #1 of 45
4/13/09 at 6:21pm
Thread Starter 
So I found a very interesting argument posed by a member of a New Zealand audiophile forum.

Quote:

Blind testing of something that is measured subjectively is inherently flawed, and perception of what good audio sounds like is usually subjective.

I work in a field where the double-blind randomised controlled trial (RCT) is the gold standard, due to large placebo effects in any unblinded trials. I love RCTs. The key issue here is that the measures are objective - they are hard, measureable, and repeatable. RCTs using subjective outcomes tend to be poorly regarded unless they use validated scales or mass amalgamation of data. Put simply, an RCT using subjective outcomes has so much bias that the results may well be meaningless.

Someone mentioned wine - it's a good example. I do a lot of blind wine tasting and I'm good at it - it's useful to find great new wines with no preconceptions. But I am aware that I can taste the same wine (unknown) in different circumstances and have very different opinions. Things like temperature, my mood, what my palate has been exposed to previously, and comparison wines, all affect my judgement. I'm quite capable of liking a wine one day and disliking it another. And neither time am I wrong as my opinion is subjective. Others may like or dislike the same wine, neither of which invalidate my opinion.

Some wines have clear flaws. Good tasters can identify these (and they can be measured in a lab) and in a group of experienced tasters these measures can be objective, but wine quality never will be. Even wine judges, when awarding medals etc, will have dissenting opinions within the group.

Audio is the same. Unless you want to do double blind testing with machines measuring fixed validated outcomes like volume, bass depth, frequency sweeps, it's pointless. Firstly, it's usually single-blind as most people do it knowing the equipment involved which immediately introduces bias, plus all the outcomes are subjective, which makes any results flawed from a scientific viewpoint. And it's commonly done in groups - another source of bias, as groups tend to alter individual opinions towards a mean. Any preconceptions about the result (ie cables do/don't work) also introduce bias.

The net result of all this is that if you perform a group single-blind test with any preconceptions about the results (difference/no difference) you introduce significant bias into already subjective measurements, which makes it difficult to extrapolise the results to any other setting i.e. it doesn't help me!

Results from DBT may be interesting, but they are unlikely to be more valid that simply saying 'My music sounds better like this'
That by itself is somewhat conventional. However, after some elaboration...

Quote:
Let's accept that you can hear a difference in a blind test, and that that difference is real. The problem is, what exactly have you heard a difference in?

It can't be how good it sounds because that is subjective and unmeasureable [1]. You've heard a difference in something else that is not defined nor quantified. But this something is important to you.

Now let's do a test sighted. I decide that I much prefer the big shiny expensive one. The images have biased my thoughts. I can hear a difference. However that difference is exactly the same unmeasurable 'thing' as you hear above above, and this 'thing' may even be improved by nice surroundings, the coffee I'm drinking, or a lovely day. All these variables improve my feeling of difference.

Now here's the killer bit. I can now hear bigger differences than you in exactly the same unmeasurable thing. You could argue they're not 'real' but they are real to me. Because good audio is a perception as well as physical waves. And I'm getting more pleasure from my audio than you are because of them. I'm better off! Lucky me.

[*1] groups of trained 'experts' are likely to be able to repeatedly define a sound that meets their subjective views of what good sound is. A 16 yr old ipod wearer might well disagree with them. I'll form my own views about what good sound is. I certainly prefer some reviewers tastes to others.
... it gets really interesting.

I want to refrain from posting the actual thread because it's full of garbage and insults - but this poster, I think, is on to something.

DBT does seem to be the gold standard when it comes to objective variables, i.e. a reduction in meningitis with the use of penicillin. These parameters can be measured. However, when it comes to specific subjective effects, I think the argument is actually compelling when one considers bias as creating the same subjective "thing" as an actual component change. I don't know if this argument alone is enough to persuade me, but I think it's a little bit different than most arguments, if not necessarily in content as much as the way it's handled.

People's thoughts?

EDIT: As a disclaimer, this thread isn't supposed to be about whether or not cables make a difference, but specifically about the validity of double blind testing with regards to subjective phenomena - for example, you may be ok with eschewing DBT for audio, but what about pain relievers, etc? It may end up a cable debate, but if it does I'm disclaiming upfront.
Reply
post #2 of 45
4/13/09 at 6:25pm
DBT is as good of a measurement as you can get. His argument is that nothing can be measured, so we shouldn't even try. In my opinion that's flat-out wrong. If that were the case, then no piece of equipment can be said to be at all better than any other piece of equipment, so we might as well all go with the cheapest or the prettiest.
Reply
post #3 of 45
4/13/09 at 6:28pm
No, his argument is right.

But that basically makes your claims valid only in your frame of reference - and to take it even further, only valid in the exact conditions you listened to your audio equipment in.

Therefore, never, ever share your impressions of cables, headphones, amps, or DACs. Heh.
Reply
post #4 of 45
4/13/09 at 6:48pm
Thread Starter 
Arainach: That's not his argument. His argument is that there are objective measurable things in the world, but that listening isn't one of them, because listening is a purely subjective experience.

However, both you and Cegras are correct in that he's arguing that there's no such thing as an objectively "better" piece of equipment. I think there is something to that argument though - after all, even if hypothetically, say, everyone hates the K701 (no, not everyone does, but I'm making this up off the top of my head), but one guy really does like it, that doesn't make that lone guy wrong. This isn't to make impressions useless, only that it recognizes that those impressions aren't set in stone, and I think they aren't set in stone - I disagree with a lot of impressions for a specific headphone, and I also agree with other impressions.

However, there's something to be said about probability. The probability of you liking a piece of equipment is much higher if a whole bunch of people like it, or if someone who consistently shares your tastes likes it - you may still dislike it, but it's more probable that you'd like that equipment than a poorly regarded one that most people hate, etc. That's not to say that the former piece of equipment is necessarily better, or that the few that like the poorly regarded equipment are wrong.

Also, plenty of people like equipment that are measurably "worse" - tube amps often have much higher level of distortion and measure terribly, yet people still like them because they sound euphonic - again, a purely subjective response that we wouldn't call "wrong."

Specifically, cegras: impressions are always just opinions, and they're definitely never valid in any context. That doesn't mean, however, that you shouldn't share your impressions, just that you can't expect everyone to agree with you. Conversely, for something that is valid, you should expect everyone to agree with you (I expect everyone to agree that a circle can't be a square because the statement "an object can not be simultaneously a square and a circle" is a valid truth-bearing statement, and not an opinion).
Reply
post #5 of 45
4/13/09 at 7:07pm
We have different situations here. Take the case of lossy compression, for example. We have a source file, and the goal of lossy compression is to sound identical while making the file as small as possible. Here, double blind test is ideal. Any consistent identification of differences shows that the compression is not sufficiently transparent. It does not matter what is different, or even which sounds subjectively better.

Now, how about something like Logitech speakers vs. Klipsch? Nobody will claim they should sound identical. We can take for granted that neither will sound perfect, and furthermore that we will not have access to anything "perfect" to compare them against. So it is very much a subjective comparison, and there is nothing to prove. However, there is still a bias that could be removed. All of the things you've heard other people say about speaker A and speaker B are sitting there in your head, even though there subjective tastes may be completely different from yours. The better looking speaker will also stand out, but that is an impression that would fade over time if you owned the speakers. The quoted post even admits that blind wine testing is useful to eliminate preconceptions. Well, it is useful in audio also to eliminate preconceptions. It is not a scientific pursuit in this case so much as an attempt to be honest with yourself. Unfortunately such tests are not very practical for the average Joe, and if the magazine reviews would do them it becomes a matter of differing tastes.
Reply
post #6 of 45
4/13/09 at 7:16pm
Read carefully the argumentation : this guy doesn't talk about ABX or ABC/HR, but about statistical correlations between subjective opinions such as "A sounds more musical", or "B sounds deeper".

This doesn't apply to blind testing as done for audio, where the methodology is always ABX, ABC/HR, or another objective method.
Reply
post #7 of 45
4/13/09 at 7:17pm
Quote:
Originally Posted by royalcrown View Post
Arainach: That's not his argument. His argument is that there are objective measurable things in the world, but that listening isn't one of them, because listening is a purely subjective experience.

However, both you and Cegras are correct in that he's arguing that there's no such thing as an objectively "better" piece of equipment. I think there is something to that argument though - after all, even if hypothetically, say, everyone hates the K701 (no, not everyone does, but I'm making this up off the top of my head), but one guy really does like it, that doesn't make that lone guy wrong. This isn't to make impressions useless, only that it recognizes that those impressions aren't set in stone, and I think they aren't set in stone - I disagree with a lot of impressions for a specific headphone, and I also agree with other impressions.

However, there's something to be said about probability. The probability of you liking a piece of equipment is much higher if a whole bunch of people like it, or if someone who consistently shares your tastes likes it - you may still dislike it, but it's more probable that you'd like that equipment than a poorly regarded one that most people hate, etc. That's not to say that the former piece of equipment is necessarily better, or that the few that like the poorly regarded equipment are wrong.

Also, plenty of people like equipment that are measurably "worse" - tube amps often have much higher level of distortion and measure terribly, yet people still like them because they sound euphonic - again, a purely subjective response that we wouldn't call "wrong."

Specifically, cegras: impressions are always just opinions, and they're definitely never valid in any context. That doesn't mean, however, that you shouldn't share your impressions, just that you can't expect everyone to agree with you. Conversely, for something that is valid, you should expect everyone to agree with you (I expect everyone to agree that a circle can't be a square because the statement "an object can not be simultaneously a square and a circle" is a valid truth-bearing statement, and not an opinion).
Having heard headphones that are vastly different, I can attest to and agree with what you say. I was just joking, mostly : )
Reply
post #8 of 45
4/13/09 at 7:26pm
In general the OP is right. DBT is not perfect. Nothing is. But in science (and in real life) YOU ONLY DO WHAT YOU CAN DO. And DBT is vastly better than no test at all. It is used in determining very subjective things such as the effectiveness of analgesics. And there is nothing more subjective than pain. It may not work perfectly, but it is all we can do. No-one can ask for more.
Reply
post #9 of 45
4/13/09 at 7:52pm
Quote:
Originally Posted by Pio2001 View Post
Read carefully the argumentation : this guy doesn't talk about ABX or ABC/HR, but about statistical correlations between subjective opinions such as "A sounds more musical", or "B sounds deeper".

This doesn't apply to blind testing as done for audio, where the methodology is always ABX, ABC/HR, or another objective method.

Pio2001 hit the nail on the head. The wine tasting analogy is nothing more than a strawman. When people refer to doing a blind test in audio, they don't mean that you're supposed to listen to unknown samples(be it cables, codecs, amps, etc...) and describe your subjective reaction to each one. Such a test is obviously not objective. However, the blind test to which people actually refer is an ABX test or something like it. In this test, you're simply supposed to listen to an unknown sample and pick the one that matches it. While different individuals may differ in their abilities, the ABX test provides an objective measurement of each person's ability to detect differences in audio.
Reply
post #10 of 45
4/13/09 at 8:39pm
Who says there's nothing objective about testing cables? There's plenty of objectivity when you measure them with a testing device.

And to take his argumen further, listening tests are inherently subjective and humans are utterly unreliable. Therefore, why have humans in the equation at all?

If someone contends that a difference exists, they must be able to demonstrate it with repeatable, controlled and peer reviewed tests. People continually poin to jitter as a phenomenon that was unknown before it was heard. Today, we can measure jitter and control for it. There's no reason the same can't be done for cables. Assuming, of course, that a difference exists.
Reply
post #11 of 45
4/13/09 at 8:41pm
DBT's cannot determine if something is "Better" (subjective).
DBT's can determine if there is a "Difference" (objective).
You either hear it or you don't, can tell the difference or you can't.
Coming up with esoteric reasons why a simple test is inherently flawed because the results do not validate your assumptions smacks of desperation and grows tiresome.
Reply
post #12 of 45
4/14/09 at 12:33am
Thread Starter 
(Note: In this thread I'm just going to be playing devil's advocate because the original poster is to my knowledge not here at head-fi. Please don't take anything I say personally, as it's not necessarily my belief, I simply want to see how far this argument can go against counterarguments etc)

Quote:
Originally Posted by fjf View Post
In general the OP is right. DBT is not perfect. Nothing is. But in science (and in real life) YOU ONLY DO WHAT YOU CAN DO. And DBT is vastly better than no test at all. It is used in determining very subjective things such as the effectiveness of analgesics. And there is nothing more subjective than pain. It may not work perfectly, but it is all we can do. No-one can ask for more.
Sure pain is subjective in one sense, but it's different from audio evaluation in another sense - that is, humans nearly universally agree that it's better to have as little pain as possible - while the degree to which pain is reduced is subjective, it's far less value oriented. When it comes to the reproduction of audio, there's not even close to a consensus of what is "best." Some people love tube amps despite having high distortion and a nonlinear response, some like emphasized bass, etc. So with this, it really asks the question: is DBT actually better than no test at all? Is there a point to blinding test conditions for aesthetic qualities when the equipment in use is not in a blind environment? Granted it makes sense to blind for other factors that aren't encountered in day to day use: head colds, temperature, time of day, sleep deprivation, etc. But this gets at a deeper issue: is the Mona Lisa better when put in a nice frame versus just by itself? The question as I see it is far more philosophical, which is why I guess I find it most interesting: are the techniques of science really useful when it comes to aesthetic experiences?

Quote:
Originally Posted by deaconblues View Post
Pio2001 hit the nail on the head. The wine tasting analogy is nothing more than a strawman. When people refer to doing a blind test in audio, they don't mean that you're supposed to listen to unknown samples(be it cables, codecs, amps, etc...) and describe your subjective reaction to each one. Such a test is obviously not objective. However, the blind test to which people actually refer is an ABX test or something like it. In this test, you're simply supposed to listen to an unknown sample and pick the one that matches it. While different individuals may differ in their abilities, the ABX test provides an objective measurement of each person's ability to detect differences in audio.
Even with ABX, the problem still remains - what is an ABX actually trying to do? On the one hand, it's trying to isolate just being able to hear a difference using your ears alone. In this sense it's trying to get rid of "imagined" sound, right? But in the aesthetic experience, everything is "imagined." In terms of objective reality there's no such thing as color - no objective property of an object gives it its color, we simply perceive it as red or blue, etc. So it would go for any aspect of audio: We're creating music internally in our head, so in terms of qualitative differences, what makes a so-called "imagined" sound qualitatively different from a "real" sound?

Quote:
Originally Posted by Uncle Erik View Post
Who says there's nothing objective about testing cables? There's plenty of objectivity when you measure them with a testing device.

And to take his argumen further, listening tests are inherently subjective and humans are utterly unreliable. Therefore, why have humans in the equation at all?
Because humans are the ones experiencing the actual music. A testing machine only tells you if the external stimulus is identical, but if the aesthetic experience differs then the stimulus is meaningless - taking the above Mona Lisa example, would it matter if the Mona Lisa was portrayed in a dingy basement versus a clean quiet museum with nice music playing in the background? I think it would affect the aesthetic experience significantly, even if it was, indeed, the same exact Mona Lisa in both instances. Likewise, even if a given cable, for instance, didn't affect the sound one iota, it may as well have if it causes the aesthetic experience to differ in some way, shape, or form.

Quote:
Originally Posted by Real Man of Genius View Post
DBT's cannot determine if something is "Better" (subjective).
DBT's can determine if there is a "Difference" (objective).
You either hear it or you don't, can tell the difference or you can't.
Coming up with esoteric reasons why a simple test is inherently flawed because the results do not validate your assumptions smacks of desperation and grows tiresome.
But that "difference" is hardly objective because such a difference is experienced in wholly subjective terms. If, in fact, there's no qualitative difference between the "placebo" effect of a component and an "actual" audible difference, then it would then follow that the placebo effect changes the sound in a, so to speak, "real" way, and consequently isolating aspects that create an entire aesthetic experience is meaningless.
Reply
post #13 of 45
4/14/09 at 12:46am
One of the issues with subjective dogmatics is that they simply do not go as far as they could. If audio is truly subjective then why depend on physical objects to influence your subjective experience. Cut out the jewelry and work on the subjective machine between your ears.
Reply
post #14 of 45
4/14/09 at 1:42am
I think the quoted poster was spot on. He doesn't dismiss ABX DBT for all things audio, in fact he lists a few areas where they would be ideal, perhaps better done with measuring equipment but still, easily done by humans. Does speaker A go deeper than B? Objectively determinable. Does speaker A sound better than B? No way to objectively determine because even if a speaker was so horribly made and sounding, someone somewhere might prefer it to the best darn transducer ever created.

DBT's are excellent for determining differences. His notes about flaws in blind testing are good ones to record. It is one reason why I have advocated SBT or DBT's where a listener has the ability to listen to the gear for any length of time in the comfort of his/her own home. I've yet to read a study that takes these types of time periods into consideration.

That all said, like others, we do what we can. DBT's may be the best we have. They might not be, but then the testers must evaluate the methods and continually improve them. The key is to be OPEN to new methods, realizing that DBT's may well be flawed as they are currently implemented.
Reply
post #15 of 45
4/14/09 at 10:30am
I think a DBT is an okay system provided it is performed repeatedly. Otherwise it is an awful system.
The problem with DBT is that people will perceive differences that arent there with their mind. And then of course there is a luck factor in it.
DBT becomes a trustable source if more people pass it more frequently.
A large problem with a DBT that I am not involved with is that theres no idea if I can be assured the results are accurate.
Cheers.
Reply
New Posts  All Forums:Forum Nav:
  Return Home
  Back to Forum: Sound Science
Head-Fi.org › Forums › Equipment Forums › Sound Science › An interesting take on DBT