Chapter 5 - Getting It Wrong

[castleofargh]

people lie all the time, people are wrong all the time, and we all make judgments based on our very own but always limited experiences. if we were all right all the time we wouldn't be unique individuals.(one day machines will rule the world!!!! muhahahahaha!!!!!)
the problem isn't for the review to be faulty or not, it will be somewhere and it's unavoidable. the problem is when we forget that reality. that in turn makes us say "it's like this" instead of "I believe it's like this", even if we don't have more than gut feelings or weird but seemingly valid correlations like "I saw a black cat, I fell, therefore black cats are bad luck".
people shouldn't be scared to express their opinions because they might be wrong. they will be wrong and so will be other people. we share when we believe we can bring a perspective, it doesn't really matter if it's right or wrong as long as it's not expressed in a way where we pretend to know more than we do. if we're not sure of something there is no shame in saying it.
but I feel that I must insist for the billions Brooko's fans reading this ^_^, post your impressions, even if they help only one guy, then it's already a meaningful post. most of what we know, we learned from someone else.
be skeptical toward yourself when you write, be skeptical toward others when you read and all will be fine. look I forgive you in advance for your next 10 mistakes. call me and I'll put a stamp on your card.
I relate to what Brooko wrote completely, in fact I've reached a point where I'm not scared of being wrong, but I am of being wrong and have nobody say it. that's the real worst case scenario for me. I stay in my mistake, and a few people will conclude that what I said was right because it stayed unchallenged.
 
 
about graphs, they shouldn't be used independently, but as you did with the dunu and Co, to show variations from one measure to another. so with the references you gave, you weren't really wrong IMO. the notion of neutral was skewed, but the IEM was indeed weak in the sub compared to those specific other IEMs. and as the idea of enough sub is one of the most subjective thing in audio, I think you could have gotten away with it ^_^.
 
about raw data and compensations, the assumption with RAW data is that it comes from an already calibrated dummy head or microphone or whatever. so usually one raw data is likely to be close to another raw data(nothing is perfect but it's as good as it gets). now with what brooko and I use, we do not have a proper calibration and the device is far from being one of those expensive dummy heads. so our RAW data doesn't mean more than "we didn't try to compensate for diffuse field or whatever, yet". but you can't relate to it to compare another source of raw data. in fact I personally have made a compensation just so that my measurement would kind of look like other raw data ^_^. that's how much I need to cheat to look a little like the pros. and then from there I sometimes apply another compensation to simulate diffuse field or others(still working on some ok-ish harman target). that's why despite all my efforts, my measurements should be compared to nothing but my measurements. and same thing for Brooko's or any other source of data for headphones/IEMs.
 
 
edit: did I say it's a great post and it needs to be said from time time? no, oh well , next time I'll try to remember.

 

[MissChristie]

It's awesome that you are able to look back on your past critically and break yourself down. I hope to bang out some more reviews while on Head-fi, and your entry is very helpful for being mindful about certain areas where I could screw up. I will doubtless make my own mistakes, and I hope to have the capacity you demonstrate to be critical of myself. I always know I'm in for an awesome learning experience when I see a Brooko review on the first page.

 

[husafreak]

Cool blog. I'm probably a little out of my element here but I think anyone who takes on the mantle of reviewer in an official capacity has to understand that "getting it wrong" can be damaging financially to companies selling products and also to folks that buy products based on reviews only. The only option for many! So it is not a game and it is cool to read your blog and see how you take it very seriously. Lots of experience and good test equipment will only make you better. Thank you!
That said don't beat yourself up over those Savants, I was smitten with them but ultimately passed due to my perception of light weight bass. I told the Noble gang that after a 1/2 hour listening session (never having read a review) and they said hey no problem and handed me a set of Dulce Bass IEM's to try. Suspect graphs aside it is probably a good thing in hindsight that bass heads didn't run out and buy Savants after reading your review. You said the right thing even if your bass bias was skewed by recent auditions. Um, thanks for that too!

 

[FullCircle]

  Cool blog. I'm probably a little out of my element here but I think anyone who takes on the mantle of reviewer in an official capacity has to understand that "getting it wrong" can be damaging financially to companies selling products and also to folks that buy products based on reviews only. The only option for many! So it is not a game and it is cool to read your blog and see how you take it very seriously. Lots of experience and good test equipment will only make you better. Thank you!
That said don't beat yourself up over those Savants, I was smitten with them but ultimately passed due to my perception of light weight bass. I told the Noble gang that after a 1/2 hour listening session (never having read a review) and they said hey no problem and handed me a set of Dulce Bass IEM's to try. Suspect graphs aside it is probably a good thing in hindsight that bass heads didn't run out and buy Savants after reading your review. You said the right thing even if your bass bias was skewed by recent auditions. Um, thanks for that too!

 
 
I doubt bass heads would have purchased the Savant to begin with, as it has never been described by Noble or anyone associated with Noble as a product designed for bass heads.   That being said, those that appreciate a balanced sound, may have been "put off" by the massive role off that was represented by the original graph.
 
 
Regardless, I'm over weight and the bills are paid...   obviously Noble survived.   I understand that the graph was not done in order to malign Noble, but this does show that graphs can be potentially harmful if not performed in an accurate manner.   

 

[MMansell]

Great stuff. I liked the part about acknowledging your misjudgments. I think reputation can create stubbornness, unfortunately.

I have a fundamental question about one's perception:

Would you say that a review that is pure aural experimentation, is imperfect or say, unprofessional? I say this while looking at the fact that there are some people whose budgets are let's say, not very high, but think about training their ears to eventually analyze sound rather accurately…

I know different people may have different ears but generally speaking!

 

[Brooko]

I'm afraid I can't really understand what you're asking me - if you can explain what you mean by aural experimentation, perhaps I can answer it.
 
If you mean I'm suggesting that beginners shouldn't review - I'm not saying that at all,  I think getting as many data points as possible is important to reach a consensus.  But I would say that having more experience lends a better sense of perspective over what you are reviewing - and where it sits against other things that are out there.
 
I think if you look at the products I review - cost doesn't come into it.  I review anything that has good sound - from $5 earbuds right through to $1K monitors.  Good sound is good sound regardless of budget.

 

[MMansell]

  I'm afraid I can't really understand what you're asking me - if you can explain what you mean by aural experimentation, perhaps I can answer it.
 

What I meant by cost wasn't about the reviewed product but about the testing gear for headphone response. And by "aural experimentation" I meant critical listening 

. Sorry about the vagueness BTW!
I mean... there are lot's of people who believe: " yes, measurements are cool and all... but the verdict should always be what you hear", and as a result, some of them don't even try buying or building the test gear.
They're like when a football administrator says we don't need video check in football cause we're satisfied with referees. The referee being their ears and the video check being data sheets showing frequency response, step response, impulse response, etc.
 
So I guess my real question is... how much right or wrong do you think these folk are? 

 

[Brooko]

Ah - thanks - that makes sense.
 
I know enough about myself to trust my ears to a certain point - despite the tinnitus and my age, I can discern things like channel balance pretty well, and I'm starting to get a good ear for frequency variations.
 
But I also know enough not to 100% trust me own perceptions - or at least to check them to make sure they're on track.  Often anyone who simply argues "I only trust my ears" - I completely disregard, or take their comments with a large amount of scepticism.  What works for me is listen first, then measure, then listen again and start to ask "why?" when I have both sets of data.
 
Video's like this heavily influenced me along the way - Ethan Winer's Audio Myths Workshop

 

[Cinder]

How would you reccommend those of use just starting out go about designing our own measurement rigs? I want one, but don't want to drop enormous amounts of money to get this data.

 

[castleofargh]

 

  I'm afraid I can't really understand what you're asking me - if you can explain what you mean by aural experimentation, perhaps I can answer it.
 

What I meant by cost wasn't about the reviewed product but about the testing gear for headphone response. And by "aural experimentation" I meant critical listening 

. Sorry about the vagueness BTW!
I mean... there are lot's of people who believe: " yes, measurements are cool and all... but the verdict should always be what you hear", and as a result, some of them don't even try buying or building the test gear.
They're like when a football administrator says we don't need video check in football cause we're satisfied with referees. The referee being their ears and the video check being data sheets showing frequency response, step response, impulse response, etc.
 
So I guess my real question is... how much right or wrong do you think these folk are? 

the way I see it:
nobody will only measure. that's an alternative that doesn't actually exist. so I would shift the options from listening vs measuring, to the actual choice that is listening vs listening+measuring.
and from there it's pretty easy to know what will give the most information.
but the choice is free, and certainly not everybody should need a lab before he dares make a review ^_^. many professional reviewers go full subjective, they wouldn't even bother with volume matching and stuff that would very obviously help a fair judgment. it's their own choice and people seem to enjoy reading them. I tend to run away, but that's my very own choice as a reader.
 
 
 

  How would you reccommend those of use just starting out go about designing our own measurement rigs? I want one, but don't want to drop enormous amounts of money to get this data.

my own very limited  opinion: ^_^
if you're only into IEMs, then the vibro veritas is an ok solution IMO. it's really not perfect and often you struggle finding a tip that will stay in place, but overall I'm glad I got one. it's a little adventure for still relatively lazy people ^_^. you could also get a little microphone and DIY a little tube to get the IEM sealed. 
else for headphones you'll have to make your own dummy head(ok I mean a box, but dummy head looks cooler) to hold the headphone in place, and then go mad with the material you'll use around the microphone. I believe people doing this have gone from spontex sponge, to super complicated stuff like trying to mold their own ears.
 
for microphones, I would go for one that goes straight to USB(the right price is the price you can afford), that way the ADC part in included and even if the input of your soundcard sucks, you bypass it and get your results.
 
 
in any case, the system you will create will most likely not have a proper calibration and will most likely create a resonance at a frequency different from what a real ear would do. so the frequency response will be informative only when comparing 2 of your own measurements. it sucks, but that's the reality of things. the graph itself doesn't mean anything, only the variations from headphone to headphone do.
for other measurements, like distortions, impulse response... what you will measure is always the headphone+your little system including the mic and the ADC, so it may be hard to get significant results about the headphone alone(also you would need to be in a very very silent room to get very low values not masked by noises).
sadly there is a reason why real dummy heads are so expensive and are used in pro environments most of the time.
 
 
if you want to do measurements of other devices(DAC/amp...), this little guy is probably the most serious contender at a still reasonable price https://www.quantasylum.com/content/Products/QA400.aspx
but from reading about it, I'm a little afraid that someone with zero experience in measurements may end up destroying it, and that would be a shame. practicing with a soundcard input or some cheap ADC if the soundcard only has a very crappy mic input instead of a real line in, might be a good idea before jumping on a QA400(I still haven't made the move on this guy, but ClieOS has a few others, they can talk about it more).
 
edit: I just notice they have updated the model and maybe this one won't blow in your face when you mess up? ^_^ I need to go read a lot.

 

[Brooko]

  How would you reccommend those of use just starting out go about designing our own measurement rigs? I want one, but don't want to drop enormous amounts of money to get this data.

 
Thread 2 in my blog shows what I use - http://www.head-fi.org/t/796996/chapter-2-anatomy-of-a-review-the-equipment
 
Might help .....

 

[husafreak]

Off topic alert, hope no one minds, but this thread has got me thinking. I need to come clean! I went to CanJam SoCal on a mission to purchase a set of IEM's. It came down to the Noble Savants and the 64 Audio U6's. In the end I chose the U6's. Wait! Please read read on! How on earth do reviewers get peeps to read on? Anywhooo, remember how I told the Noble guys that I wanted more bass and they gave me the Deuce Bass to try? Well, I wasn't having any of that, I'm an audiowhatever! I would never buy earphones designed to enhance the bass range. No sir. So I ordered the U6's and by the time I got them and had a good listen I realized they are quite bassy, (descriptive term used by illiterate reviewers) and rather dark, but also that 64 Audio sells different "modules" that support their ADEL technology and their earphone's sound... Well, I just finished a lengthy comparison of the module I bought at CanJam, and the one I am going to use instead of it. The one I like best has a less full, but flatter, more revealing, sound signature. It is probably a lot more like the Savants than the one I bought! Yep, I was hooked by a show demo with seductive bass. While being put off by a set of earphones advertising a seductive bass. What a putz. And go ahead and look at the graphs for the different U6 modules, they will not prepare you for a big difference in actual perceived sound. I have two prominent reviewers backing me up on this observation too. (The reviews came out after my purchase). So beware all you non experts with little experience. Having studied reviews by our esteemed colleagues may enable you to make better decisions out there in hi-fi land! And not be such a putz.

 

[Brooko]

I loved the Savants - really my kind of Sig.  I also love the U6 and B1 module I have now. Both to me are at the same level of excellence - despite one being 6 BA and one being 2 BA.  Good sound is good sound.

 

[MMansell]

 
the way I see it:
nobody will only measure. that's an alternative that doesn't actually exist. so I would shift the options from listening vs measuring, to the actual choice that is listening vs listening+measuring.
and from there it's pretty easy to know what will give the most information.
but the choice is free, and certainly not everybody should need a lab before he dares make a review ^_^. many professional reviewers go full subjective, they wouldn't even bother with volume matching and stuff that would very obviously help a fair judgment. it's their own choice and people seem to enjoy reading them. I tend to run away, but that's my very own choice as a reader.

 

  Ah - thanks - that makes sense.
 
I know enough about myself to trust my ears to a certain point - despite the tinnitus and my age, I can discern things like channel balance pretty well, and I'm starting to get a good ear for frequency variations.
 
But I also know enough not to 100% trust me own perceptions - or at least to check them to make sure they're on track.  Often anyone who simply argues "I only trust my ears" - I completely disregard, or take their comments with a large amount of scepticism.  What works for me is listen first, then measure, then listen again and start to ask "why?" when I have both sets of data.
 
Video's like this heavily influenced me along the way - Ethan Winer's Audio Myths Workshop

Thank you both Brooko and castleofargh. Very informative. I'll absolutely watch that video later.
I actually tried the volume matching device mentioned in your thread 2, but I in a very newbie way ...
I installed an app on my smartphone for sound pressure level measurement that sensed the sound from headsets mic. The headphones were broken but the mic worked just fine. So I superglue'd a foam tip around the tiny mic hole on the broken headset. I know it was inaccurate because of the calibrations and whatnot ... but there's a certain joy in crafting something out of the junk you already own... well, the foam tip only had one size so I think using superglue might have been a bad choice

.
I probably should have mentioned it under your second tread... Oh well, it's not like it's a discovery or something

 

[RebeccaSugar]

You've inspired me