[skamp]

^ I don't think that's the kind of thing he was referring to… More like the kind of measurements that's in my sig. Like, the iPod measures much better than the Hifiman HM801, but he'd probably say the latter sounds better (hence making measurements useless in his mind).

 

[xnor]

There's a lot of placebo, I've noticed, in measurements. Because someone, whether they be a manufacturer or some random person on the internet writes or posts something that seems to be "science" doesn't make it either useful or a valid claim as to a component's performance. 

Oh but the brilliant thing is that measurements can be verified. For example, you don't need an expensive audio analyzer to see that the mini3 has big troubles with certain loads or that some "random" DIY amp by some "random" person performs better.

Anyway, I'd neither take measurements nor subjective reviews for granted, but I'd take measurements over (placebo affected) subjective reviews any day. Whether I like the component or not is my (ears) decision to make, not somebody else's. Measurements can help to separate the wheat from the chaff before even taking a listen / wasting time trying to interpret some random guy's (random) impressions.

 

[Currawong]

My point indeed was the relevance of measurements. They are useful to explain what we are hearing, but can be equally deceptive by themselves. Why wouldn't good-looking measurements be any less of a potential placebo than a shiny face plate?  An example might be product specs where the distortion measurement was only taken at one frequency under the most ideal load. Another might be the perfect square wave output of a NOS DAC, when square waves don't occur in music. Importantly though, if measurements can't be directly related to our experiences, they aren't much use. In that case, the measurements are arguably a placebo for people who value those things when buying equipment.
 
There was an interesting write-up in Stereophile of the very expensive Zanden 5000 MK.IV DAC where the measurements were awful. It's worth reading the subsequent write-up where they  investigated to attempt to understand how what they had heard related to the measurements.  
 
If the idea of reading anything from Stereophile puts you off, I think Tyll sets the best example at Innerfidelity where he does his best to correlate measurements with listening experience.
 
I don't think the issues are a lack of being able to measure the capabilities of components, but the lack of ability to translate those measurements into something meaningful for the average person.

 

[Radio_head]

I had a speaker guy at my house a few weeks back.  He was amused by the headphone rig and asked to try it.  After a few minutes he got really into it and started critiquing the sound - it needed more accentuation in the high mids.  So I showed him the brightness filter (he was listening on the Headroom Max.)  I flipped the switch and flipped it back out of curiosity.  (ie he thought it was on when in reality it was not.)  He told me with the filter switched it was now had a perfect balance to his ears.  
 
Placebo does exist.  I had another thought though - people want to have golden ears.  They want to be dissatisfied with the sound of a given setup, make a tweak, and be convinced they just made things better.  
 
Everyone has a bias.  Most have a bias convincing them they are immune from bias.  Some admit their bias but go on living with it.  A select few recognize it and go about finding a way to prevent it from affecting their judgement.

 

[mikeaj]

I think Tyll's writings are good examples of misinterpreting measurement data sometimes.  Some of the things he says about those square wave graphs need a second, more rigorous examination, to say the least.
 
Any time you have outside influences like reading measurement data or somebody's review of a product, that's going to introduce some expectation bias, sure.  But it's not like the actual performance is going to be different than what is measured, except if you're talking about acoustic measurements and the (probably small? but maybe not always) inaccuracies of the dummy head and ear canal with respect to modeling your own anatomy.  Data is data, though, and I think you (Currawong) are having a different conversation than the rest of us.  The data isn't misleading.  Most people just don't know what it means or don't interpret it correctly, to varying degrees.  Some people don't understand the limitations and the test conditions, and other people think that there are a lot more limitations and lack of relevance than there actually is.

 

[anetode]

I'd say that cognitive biases in general, not just the placebo effect (which, if memory serves, cannot actually be argued to apply here), are rampant in CE in general.

Yes! If there's going to be a default catchall term for subjective perceptual phenomena, then it should be cognitive bias rather than placebo.

Why wouldn't good-looking measurements be any less of a potential placebo than a shiny face plate?

Good measurements may serve to allay usual predispositions/hypercritical tendencies, but knowledge of one set of measurements would not counteract the deleterious effects of another measurably flawed parameter. Luckily the usual measurements are easy to satisfy. People might not recognize a signal with 20% thd, depending on the value of the fundamental and the distribution of harmonics, which begs the question whether that measurement is the most useful guide.

 

[xnor]

My point indeed was the relevance of measurements. They are useful to explain what we are hearing, but can be equally deceptive by themselves. Why wouldn't good-looking measurements be any less of a potential placebo than a shiny face plate?  An example might be product specs where the distortion measurement was only taken at one frequency under the most ideal load. Another might be the perfect square wave output of a NOS DAC, when square waves don't occur in music. Importantly though, if measurements can't be directly related to our experiences, they aren't much use. In that case, the measurements are arguably a placebo for people who value those things when buying equipment.

Yes, imo measurements are magnitudes more relevant than somebody else's impressions of a component. A shiny face plate says absolutely nothing about performance. Product specs without any details of the test/measurement conditions are best treated as marketing gibberish. Regarding square wave output of NOS DACs, well, that's why we there are different types of measurements and square waves are certainly not a "standard" measurement for DACs.
Measurements cannot be directly related to our experiences because a) people fail to understand the measurements, b) the experiences are results of bias/placebo (shiny face plates, price tag) etc. In case a) they're still better off comparing components by measurements instead of by random impressions/opinions/"reviews".

There was an interesting write-up in Stereophile of the very expensive Zanden 5000 MK.IV DAC where the measurements were awful. It's worth reading the subsequent write-up where they  investigated to attempt to understand how what they had heard related to the measurements.

Imo that's a prime example of placebo and without the measurements the review would be a big pile of BS. Also see zanden-5000-review-damages-s-philes-credibility-my-book-1 for some opinions from people who actually (I don't) read (past tense?) stereophile.

 

[skamp]

Another might be the perfect square wave output of a NOS DAC, when square waves don't occur in music.

They do in Metallica's Death Magnetic! I want to listen to those square waves in their full glory!

 

[Mauricio]

True to form--and unsurprisingly--many here are more interested in discussing the placebo effects of measurements rather than cognitive biases themselves.  Time and time again, a tenacious burying of the head in the sand about the most common and widespread flaws of the vast majority of so-called "impressions", "reviews" on this and other forums (headfonia, for example).
 
"It is error only, and not truth, that shrinks from inquiry."
- Thomas Paine

 

[Currawong]

Radio_head: That's pretty funny. I've caught myself out sometimes thinking I was listening to one component when I'd switched in another. It was a good lesson, that's for sure. On the other hand, I've occasionally felt that the sound in my system was wrong and, after going over all the settings, found odd things such as having inverted the (digital) polarity.
 
Quote:

I think Tyll's writings are good examples of misinterpreting measurement data sometimes.  Some of the things he says about those square wave graphs need a second, more rigorous examination, to say the least.
 
Any time you have outside influences like reading measurement data or somebody's review of a product, that's going to introduce some expectation bias, sure.  But it's not like the actual performance is going to be different than what is measured, except if you're talking about acoustic measurements and the (probably small? but maybe not always) inaccuracies of the dummy head and ear canal with respect to modeling your own anatomy.  Data is data, though, and I think you (Currawong) are having a different conversation than the rest of us.  The data isn't misleading.  Most people just don't know what it means or don't interpret it correctly, to varying degrees.  Some people don't understand the limitations and the test conditions, and other people think that there are a lot more limitations and lack of relevance than there actually is.

 
I probably should have said that the data is potentially misleading. You're correct that it's in the interpretation where things go wrong. Either that, or people attempt to use it to make generalised conclusions which can not be derived from the data presented.
 
The good thing about Tyll is, he is always looking to improve and accepts suggestions about how he can do so.  That's related to my main idea and why it seems I'm having a "different conversation" as you put it -- most of the discussion around here has been a continual religious pissing contest about who is right or wrong on a topic (which is not what science is about), when it should be about learning and improving our understanding to help us make better choices when buying. However, the problem is that a large amount of science is involved with audio, ranging from psychology through to physics, and our common tendency to want to over-simplify the understanding of all of it results in people writing over simplistic explanations which can be misleading. These end up being, in effect, a different kind of placebo, as people end up believing they know all they need to know.

 

[tmars78]

Quote:

Yes, imo measurements are magnitudes more relevant than somebody else's impressions of a component. A shiny face plate says absolutely nothing about performance. Product specs without any details of the test/measurement conditions are best treated as marketing gibberish. Regarding square wave output of NOS DACs, well, that's why we there are different types of measurements and square waves are certainly not a "standard" measurement for DACs.
Measurements cannot be directly related to our experiences because a) people fail to understand the measurements, b) the experiences are results of bias/placebo (shiny face plates, price tag) etc. In case a) they're still better off comparing components by measurements instead of by random impressions/opinions/"reviews".
Imo that's a prime example of placebo and without the measurements the review would be a big pile of BS. Also see zanden-5000-review-damages-s-philes-credibility-my-book-1 for some opinions from people who actually (I don't) read (past tense?) stereophile.

 
I agree.  You know why they heard something that they thought was great even though it was a POS? Because they knew it cost $45,000, plain and simple. Too often "audiophiles" want to dismiss measurements because of price. That is EXACTLY what Stereophile did. "It cost $45,000, there is no way the it sounds as bad as the measurements say it does." Just goes to show how full of s**t some people are, and honestly that review makes me laugh and puke at the same time. They desperately needed it to sound good, so it did, no matter if it did or not. And people fell for it and say, "Look, this proves measurements don't mean anything." I'm not saying they are an absolute, but you can get something for $30 from Walmart that will sound better than that. I guarantee that you can. 

 

[Magick Man]

It's hard to ride the line between scientific testing and personal taste, but I think Tyll does a fantastic job. In fact, he and Guttenberg are my two favorite headphone reviewers, not because they painstakingly go over every tiny bit of data, but because they aren't afraid to lay it on the line and say, "this is great" or "this is terrible". Tyll's not a scientist, but I sure trust his opinion on whether a set of cans are worth trying. If he sometimes makes a mistake, it's only because he isn't a robot. I'll take his views over pure raw data every time.

 

[Mauricio]

Of course.  They refuse objective testing and measurement because they are financially or personally invested, and they stand to lose something--credibility, face, status, bragging rights, profits, advertisement revenue, dumbed-down mass readership, website hits, etc--if they have to face quantitative, objective, replicable scrutiny.  Why they refuse to face the issue straight on is thoroughly transparent and predictable...

 

[tmars78]

Quote:

Of course.  They refuse objective testing and measurement because they are financially or personally invested, and they stand to lose something--credibility, face, status, bragging rights, profits, advertisement revenue, dumbed-down mass readership, website hits, etc--if they have to face the objective, replicable scrutiny.  Why they refuse to face the issue straight on if perfectly comprehensible and predictable...

 
 
Exactly. I wish audio equipment was tested like cars are. Car and Driver, Motor Trend, etc. will post tests, and tell you that a car is junk, if it is junk, no matter the price tag. Michael Fremer is the biggest buffoon in the audio world. I don't know if he honestly believes what he writes, or he is just really Ashton Kutcher and he is punking everyone who considers themselves an audiophile. 

 

[Mauricio]

Quote:

 In fact, he and Guttenberg are my two favorite headphone reviewers...

 
Is this the same Guttenberg who on a so-called "review" on this forum said that he eschews lossless codecs because they--i forget the exact phrase--end up making the soundstage flat?