[landroni]

Originally Posted by Jason Stoddard /img/forum/go_quote.gif
The Right and Wrong of the Subjective and Objective Approaches (IMO) 
[...]
 
 
Some objectivist thought leaders—the kind that have written books—don’t portray their findings in a very (ahem) objectivist manner. 

 
This comes often in "Objectivist"/"Subjectivist" discussions whereas people assume unconditionally that the "objectivist" approach is necessarily "objective", but it must be mentioned that objectivist does not necessarily equate objective, and overlooking this tends to bias discussions in unhelpful ways. Actually, if we tilted the balance of bias slightly in the other direction, we may as well call the two groups the (measurement) Fanatics and  the (perceptual) Realists; the Fanatics with religious fervour bow in front of printed data dumps, whereas the Realists sagely and wisely trust their analogue senses...
 
But getting back to measurements... There is nothing, absolutely NOTHING inherently objective about measurements*. Measurements are only useful as far as their context is understood, and their limits accepted and acknowledged. Worse, they're just as open to manipulation and incompetence as statistics is** (cf meaningless specsmanship). So measurements are only as good as the human taking the measurements, and their goodwill and competence. And the human taking the measurements is just as prone to cognitive biases and associated plagues. After all, the "publishing bias" is an indictment not of the scientific method in itself, but of how humans use the scientific method and to what purposes (i.e. under which agenda). There is simply no way that measurements are absolutely objective or bulletproof in and of themselves. And what it is that we measure*, what measurements actually mean or what their scope is in the grand scheme of things is a whole different can of worms, which exposes further more the human element in the "objectivist" approach, which may or may not be objective...

 
* I too can measure temperature in my room with a barometer. How objective or useful would that be? And how does science go about measuring "rendered emotions"? Insofar as music is positively concerned with transmitting emotions...
** Those who pontificate about how "absolutely objective" measurements really are MUST also believe that statistics are similarly bulletproof: "absolutely objective". Of course cognitive dissonance must afflict at least a subsample of those of the "objectivist" persuasion

 

[Torq]

Where objectivists and subjectivists collide is in the double blind test. And invariably, it doesn't end well for the subjectivists. Just sayin'...

About ten years ago I used double blind testing to choose between the candidate speakers I was about to buy. I walked out of the dealership having NOT bought the B&W speakers I had intended to buy and instead I purchased the, much cheaper, Linn Espek speakers that the testing showed I preferred (despite them measuring "worse" than the B&Ws).

And you're right, this double blind test didn't work out well for me ...

I absolutely hated looking at those speakers and they marred my enjoyment of even being in my listening room, let alone actually sitting in there to listen.

True story, even if I am deliberately being a bit facetious with its application. I get that double blind testing is intended to determine if there is a reliably detectable DIFFERENCE not to direct what one does if there is/isn't. In this case, had there been no detectable difference I'd have bought the B&Ws and been much happier. My purist attitude to audio purchases at the time was doing me no favors in terms of enjoyment, Defensible testing and objectivity was not getting me a better result.

At the end of the day, with me, if you want to score points in a "can you tell the difference" pissing match, bring double blind testing ... if you want to influence what gear I use, bring cash.

 

[slefty]

  Where objectivists and subjectivists collide is in the double blind test. And invariably, it doesn't end well for the subjectivists. Just sayin'...

As possibly one of the few people here who has run numerous real, blind (single blind, double blind, random assortment blind, and several other variations) tests professionally, I can say that the testing says more about the person being tested than whether the differences exist or not. I used these various blind tests to evaluate people who wanted to work in my department years ago (a quality control department). It wasn't audio, but it was another industry where there was a strong objectivist/subjectivist thing going on, maybe even more so than in audio.
 
But here's the rub, I created three levels of samples, with diminishing perceptible differences. First test was set so that at least 50% to 75% of subjects could pass. This was the "customer level" test and showed the limits of what we expected our customers to be able to detect (and therefore complain about).
 
Second test I decreased the differences to where at best 25% of subjects could pass (75% failure rate), Passing this test meant you might be able to work in my department, but not in a final "evaluative" position. Maybe just in a technical role. However, with training, people in this group could sometimes greatly improve their abilities to detect differences and move to the next level.

The final test, I diminished the differences to where less than 10% of subjects would pass. Objectivists will tell you a 90% failure rate means the differences don't exist, but what it really showed was that we found people in the 10% that could detect the differences consistently. They were the people I relied on for real QC evaluations. What was surprising was that none of the supervisors in other departments could pass this test.

In other words, don't ever assume a double blind (or any level of blind) testing actually tells much about whether a difference exists. Most "double blind" tests in audio I've seen were seriously flawed in how they were set up or controlled.

And yes, my background is in the sciences (chemistry, physics, math, with post graduate work in Applied Mathematical Statistics) though not in engineering except for a bit of industrial engineering, including queuing theory and job shop design). 

 

[aamefford]

About ten years ago I used double blind testing to choose between the candidate speakers I was about to buy. I walked out of the dealership having NOT bought the B&W speakers I had intended to buy and instead I purchased the, much cheaper, Linn Espek speakers that the testing showed I preferred (despite them measuring "worse" than the B&Ws).

And you're right, this double blind test didn't work out well for me ...

I absolutely hated looking at those speakers and they marred my enjoyment of even being in my listening room, let alone actually sitting in there to listen.

True story, even if I am deliberately being a bit facetious with its application. I get that double blind testing is intended to determine if there is a reliably detectable DIFFERENCE not to direct what one does if there is/isn't. In this case, had there been no detectable difference I'd have bought the B&Ws and been much happier. My purist attitude to audio purchases at the time was doing me no favors in terms of enjoyment, Defensible testing and objectivity was not getting me a better result.

At the end of the day, with me, if you want to score points in a "can you tell the difference" pissing match, bring double blind testing ... if you want to influence what gear I use, bring cash.

A nice anecdote and I agree with the sentiment completely, though I am sorry you ended up with speakers that you just plain didn't like.  I've been down similar paths at different times.  Finally, my headphone gear is stuff I wanted, lusted after and finally bought.  Is it the best?  Maybe (at least for me).  Is there less costly gear that sounds as good? Maybe.  Am I stoked with my current rig, and really glad I have it, even though it is possible I overpaid for the sound quality?  Heck Yeah.  Am I comfortable ponying up extra for a bit of luxury cachet that has arguably nothing to do with sound quality?  Again yes.
 
I have a kind of fun story with my speaker purchase as well (a shade the opposite of @Torq's, but I think I've gone far enough off topic as is....

 

[Ableza]

Like I said, listening to music in your home is a holistic experience.  ALL your senses, and everything else in the room, matters.

 

[AudioBear]

Thanks Jason for another great chapter.  
 
I have to agree with @pdferguson that the collision occurs at the DBT (which I think will get my post deleted for the mere mention of the concept).
 
What I have never understood is why subjectivists aren't more keen on running DBTs than objectivists.  Objectivists think they can measure and compare differences, although many of them would concede to Jason that at the end of the day you have to trust your ears because what we are able to measure just doesn't tell the whole story.  Two devices that measure the same can sound different.  How do we know that? By listening.  And the DBT is ONE way to listen effectively (not the only one).
 
To digress, it surprises me that rational subjectivists (of which there are many) don't accept that expectation bias, brain burn-in, price, mood, percentage of alcohol in the bloodstream, difficulty in reproducing listening conditions and other factors can confound subjective judgment.  Subjectivists depend instead on repeated listening over a period of time to detect or discredit differences that weren't apparent on first listen.This results in buying gear with which they eventually become unsatisfied and creates an endless cycle of upgrade-itis which ends at buying the most expensive system they can afford.  Wouldn't it be simpler to do a blind comparison and if no difference is detected just move on?  Obviously one  doesn't need to do this with every comparison since the differences between two pieces of gear is often very clear (notice I relied on hearing not on measurements).  IMHO subjectivists should be more interested in blind testing than objectivists since it helps sort the wheat from the chaff.  OK, I'll concede that buying and selling gear is in fact a legitimate hobby in and of itself.  I don't want to ruin anybody's fun or rain on their parade.
 
On the other hand I can't understand why objectivists won't let subjectivists listen in peace when they prefer to buy gear that "sounds" good to them.  That is the whole point isn't it?  

 

[GearMe]

Like I said, listening to music in your home is a holistic experience.  ALL your senses, and everything else in the room, matters.

Or...outside the home. Some of my most memorable musical experiences have been listening to a boom box with friends, footballs/frisbees, and a few beers in the park. I can assure you the sound quality sucked...didn't matter!

 

[DJ1421]

M. Scott Peck was an American author best known for his first book, The Road Less Traveled, who also wrote a book named In Search of Stones. In Stones Peck spent some time presenting a psychological concept, overdetermination​, the notion that everything important has multiple causes. Perhaps he meant everything important – even our pleasure or displeasure listening to a particular piece of audio gear.
 
Torq described in his post how a careful double-blind test left him most unsatisfied with a speaker purchase, probably because the test engaged only the sound, and not the other variables in the pleasure behind a purchase. Peck said "overdetermination demands the integration of mutilple dimensions in order to see [hear?] the whole picture."
 
I chose Schiit without ever listening at all, but please understand why... 
Reputation is there – I listened to the community.
Value – Jason describes value in audio terms – great sound for the price, and at a variety of price points.
Made in USA – for what its worth...
Beautiful – show me crystal clear photos of your work, inside and out, and I'll be better able to see your intentions. And Red – red circuit boards just sound better. don't they?
Whimsy – see Beautiful. Yes, there is value to me in the willingness to play, take one's self less than seriously, and even share mistakes. 
Story – people make things. Bob Taylor makes guitars. Jason and Mike Moffat make Schiit. I like to hear the human story behind the product. I buy from people I like or want to be like.
Confidence – because there are no magic bullet claims my expectations are set at a realistic level. I am satisfied before I ever plug anything in.

​I think there are more, but a complete list is an objectivist goal, and who could prove the list is in fact complete. The relationship between human and product is complex, overdetermined. I like it that way, and have learned to be content in the ambiguity of it.

 

[Jason Stoddard]

Different industry, but you get the idea:
 

 

[Ableza]

One more comment on this subject and then I'll drop it.  It is relatively easy from an engineering point of view to design and build gear that will adequately reproduce and amplify music, and to design and build loudspeaker systems that will reproduce the full range of human hearing with acceptably low distortion, so that the objective side of the equation is fully satisfied.  It is a completely different thing to create systems that people enjoy, that they want to use, that they wish to purchase and own and brag about on forums, that sound "good."  This is why audio is both a science and an art, and why only those who are good at both, who can create the magic of proper reproduction and enjoyment at the same time, who become successful. 
 
Congrats to Schiit for creating some of that magic with your products.

 

[taxiq]

  Where objectivists and subjectivists collide is in the double blind test. And invariably, it doesn't end well for the subjectivists. Just sayin'...

Hmmmm, wrong.   Obeectivists like to think DBX'g is a test and but subjectivsts learn it is more trick than test.  Just sayin .  .  . 
 
edited

 

[Gimpinchair]

M. Scott Peck [COLOR=252525]was an American[/COLOR][COLOR=252525] author best known for his first book, The Road Less Traveled, who also wrote a book named In Search of Stones. In Stones Peck spent some time presenting a psychological concept, overdetermination​, the notion that everything important has multiple causes. Perhaps he meant everything important – even our pleasure or displeasure listening to a particular piece of audio gear.[/COLOR]

[COLOR=252525]Torq described in his post how a careful double-blind test left him most unsatisfied with a speaker purchase, probably because the test engaged only the sound, and not the other variables in the pleasure behind a purchase. Peck said "overdetermination demands the integration of mutilple dimensions in order to see [hear?] the whole picture."[/COLOR]

[COLOR=252525]I chose Schiit without ever listening at all, but please understand why... [/COLOR]
[COLOR=252525]Reputation is there – I listened to the community.[/COLOR]
[COLOR=252525]Value – Jason describes value in audio terms – great sound for the price, and at a variety of price points.[/COLOR]
[COLOR=252525]Made in USA – for what its worth...[/COLOR]
[COLOR=252525]Beautiful – show me crystal clear photos of your work, inside and out, and I'll be better able to see your intentions. And Red – red circuit boards just sound better. don't they?[/COLOR]
[COLOR=252525]Whimsy – see Beautiful. Yes, there is value to me in the willingness to play, take one's self less than seriously, and even share mistakes. [/COLOR]
[COLOR=252525]Story – people make things. Bob Taylor makes guitars. Jason and Mike Moffat make Schiit. I like to hear the human story behind the product. I buy from people I like or want to be like.[/COLOR]
[COLOR=252525]Confidence – because there are no magic bullet claims my expectations are set at a realistic level. I am satisfied before I ever plug anything in.[/COLOR]

[COLOR=252525]​I think there are more, but a complete list is an objectivist goal, and who could prove the list is in fact complete. The relationship between [/COLOR][COLOR=252525]human [/COLOR][COLOR=252525]and [/COLOR][COLOR=252525]product is complex, overdetermined. I like it that way, and have learned to be content in the ambiguity of it.[/COLOR]

I couldn't agree more! I went about my choice for my very first ever DAC/Amp & headphone purchase exactly as you described above. I loved the snarkiness of the FAQs on the Schiit website. They remind me of my favorite smartass; me. I couldn't be happier with my Bifrost & Lyr 2 stack. I "subjectively" feel they are a great match to my HE-400I's.

 

[RCBinTN]

 
I chose Schiit without ever listening at all, but please understand why... 
Reputation is there – I listened to the community.
Value – Jason describes value in audio terms – great sound for the price, and at a variety of price points.
Made in USA – for what its worth...
Beautiful – show me crystal clear photos of your work, inside and out, and I'll be better able to see your intentions. And Red – red circuit boards just sound better. don't they?
Whimsy – see Beautiful. Yes, there is value to me in the willingness to play, take one's self less than seriously, and even share mistakes. 
Story – people make things. Bob Taylor makes guitars. Jason and Mike Moffat make Schiit. I like to hear the human story behind the product. I buy from people I like or want to be like.
Confidence – because there are no magic bullet claims my expectations are set at a realistic level. I am satisfied before I ever plug anything in.
 

I too chose Schiit equipment without ever hearing it, based on the factors you list above + reading comments on this site.  The original UberBi / Asgard2.  Also bought my LCD-X that way .  I was never disappointed with either of those decisions.

 

[gwitzel]

   
This comes often in "Objectivist"/"Subjectivist" discussions whereas people assume unconditionally that the "objectivist" approach is necessarily "objective", but it must be mentioned that objectivist does not necessarily equate objective, and overlooking this tends to bias discussions in unhelpful ways. Actually, if we tilted the balance of bias slightly in the other direction, we may as well call the two groups the (measurement) Fanatics and  the (perceptual) Realists; the Fanatics with religious fervour bow in front of printed data dumps, whereas the Realists sagely and wisely trust their analogue senses...
 
But getting back to measurements... There is nothing, absolutely NOTHING inherently objective about measurements*. Measurements are only useful as far as their context is understood, and their limits accepted and acknowledged. Worse, they're just as open to manipulation and incompetence as statistics is** (cf meaningless specsmanship). So measurements are only as good as the human taking the measurements, and their goodwill and competence. And the human taking the measurements is just as prone to cognitive biases and associated plagues. After all, the "publishing bias" is an indictment not of the scientific method in itself, but of how humans use the scientific method and to what purposes (i.e. under which agenda). There is simply no way that measurements are absolutely objective or bulletproof in and of themselves. And what it is that we measure*, what measurements actually mean or what their scope is in the grand scheme of things is a whole different can of worms, which exposes further more the human element in the "objectivist" approach, which may or may not be objective...

 
* I too can measure temperature in my room with a barometer. How objective or useful would that be? And how does science go about measuring "rendered emotions"? Insofar as music is positively concerned with transmitting emotions...
** Those who pontificate about how "absolutely objective" measurements really are MUST also believe that statistics are similarly bulletproof: "absolutely objective". Of course cognitive dissonance must afflict at least a subsample of those of the "objectivist" persuasion

One of the best statements I have read so far! I sometimes wonder how people think 'science' works. I mean science in the sense of fundamental physics (where I work) or similar fields. There are many more requirements than only quoting numbers (or showing figures). One of the most important is to give the exact context of how you got the numbers. Only if someone else (better: many other scientists) is able to reproduce the numbers following your description of this context (e.g., a description of the exact setup of your experiment), your numbers become 'objective'. To interpret and integrate your findings in a wider theory is often a matter of decades of debate. The answer to the question how a measured number relates to the fundamental questions that are at the beginning of your experiment is very subjective and often shaped by strong characters and authorities in the field.

 

[landroni]

In other words, don't ever assume a double blind (or any level of blind) testing actually tells much about whether a difference exists. Most "double blind" tests in audio I've seen were seriously flawed in how they were set up or controlled.
 

This can't be repeated often enough.... In Objectivist/Subjectivist hagglings some will usually pop up at one point and proclaim: "you should just DBT it!".
 
The problem is that DBT is NOT a magical bullet, a panacea of some sort (like a talking mirror) that will instantly tell someone right from wrong. Just because it's DBT, it doesn't make the results absolute, incontrovertible facts. Methodology matters, and controlling for potentially confounding factors matters, too. After all, before we got DBT we had simpler blind testing, until someone realized that the methodology could be flawed by the scientist driving the testing results. While certainly DBT is very useful in many contexts, it is entirely plausible that when applied to audio gear, DBT may be failing to control for some systematic factor influencing results. For instance, I've never heard of a DBT in audio land that controls for long-term listening effects, whereas the art of listening to these gizmos is arguably a long-term (>5min back and forth), holistic experience. Another potential confounder is the state of mind induced by the lab-like, DBT setting: in potentially stressful exam conditions under the watchful eyes of the proctor, does the subject retrieve the serenity and calm demeanour and bliss that they conceivably achieve when listening to gear at home?
 
The other point re DBT that people tend to overlook is that NOT every single piece of commercial product can be subjected to DBT, for various reasons. While DBT is handy when dealing with small, ingestible pills which people may not be able to visually discern their contents, things are murkier in other domains. In audio land, DBT is often about comparing two pieces of production equipment (e.g. two DACs) done by experts in the field (e.g. hardcore audiophiles).
So, how do you go about DBT'ing two tennis rackets? Even if you strip labels and use identical color schemes, the tennis player will necessarily need to look at the racket when playing and by simple evaluation of geometry, materials or weight they may form an opinion as to the racket's suitability or performance even before playing a single ball.
What about DBT'ing two cars, say is there a "perceptible difference" between a production Renault Clio and a Porsche 911? Do what you will, but the driver will need to at least use their eyes when driving, and at least experience the interior of the cars. Even if you strip labels and try to otherwise remove identifying elements, a keen observer will immediately spot the interior of a Clio from that of a 911, rendering moot the whole DBT exercise. (And how do you blind the examiner in this case?)
And what about DBT'ing a Boeing from an Airbus, or to make it really interesting DBT'ing two rockets? Good luck with that. Simply operating each individual piece of equipment requires very specific and intricate knowledge of the systems of the given model, so any DBT attempt would end up still-born. Same thing for any attempt to DBT two racing tracks, to see if professional drivers detect any "perceptible differences in driving experience": any attempt at setting up a DBT between Spa and Monza would necessarily prove futile.
 
So while clearly useful in some settings, DBT isn't necessarily universal. That we can apply it to DACs is a matter of fortunate convenience (which might not have been the case, say, if DACs were room-sized like computers in the '50s). That DBT always necessarily controls for all potential confounders is false, and I'm yet to see an exhaustive discussion on which factors should be controlled for when attempting DBT on audio gear...