Chapter 5 - Getting It Wrong

[Brooko]

So we all have reviewers we follow – people we trust not to steer us wrong – people who have similar tastes, and even like the same types of music.
 
But reviewers are human, they are fallible, and they can fall victim to bad judgement, over-enthusiasm, personal bias, and placebo just like anyone else. I've been meaning to write this one for a while, and recent events with a pair of earphones have prompted me to get off my butt and finally finish it. It'll cover things like ego, like sighted bias, like plain old lack of experience – but it'll also be about how to improve.
 
And yes – the subject here is me.
 
Stepping back in time - an early lesson
So lets skip back to my early reviewing on head-Fi – before people would send me stuff, before I ever had a front page review, when I was virtually unknown. I honestly can't remember what I was reviewing, or even who the person was who pulled me up – it was a long time ago. But I happened to make the comment about female vocals being most prominent in the 2-3 kHz area. I was so sure of myself, and stated it big and bold. And then I got pulled up on it, and set straight. I was also told bluntly that I didn't have a clue what I was talking about. But how can that be? Changing EQ in that area drastically changes the tonality. Yes dummy (me) – that's presence, harmonics if you like – its not fundamentals. After being sufficiently put in my place, I proceeded to spend the next few months using the interactive frequency chart and other sites trying to understand where I'd gone so wrong. I'd spend hours at my PC with my HD600 playing tracks I really knew well, and altering frequency response through EQ – first single bands, then multiple ones – finding out what changes, where instruments “play' in terms of frequency response, and how different changes can alter your perception. It was one of the most brutal lessons I learnt – but its also one of the most valuable – especially now. And it was about this time that I started learning how to read a frequency chart, and really learning what my preferences were.
 
Not  the most rapid learner though!
Fast forward a couple of years, and I'm starting to be noticed. I have companies approaching me to review their products. I'm being involved with tours. People are following what I write.
 
Let me repeat that – people are following what I write.
 
That is a big responsibility that I recognise now – but going back a couple of years ago, I was probably too consumed about making a name for myself, and by simple ego. Seems the lessons of the past aren't always learnt the first time. I do think I'd improved as a reviewer by then. I understood my own bias a lot better, and I was trying to be more objective about what I was doing, and I was relying on measurements rather than by ear alone. This is all good. So where is it leading?
 
Well as Chris (HawaiiBadBoy's) video reviews so eloquently state – it was about this time that Brooko – knew he'd [expletive] up. Only I didn't know it – not when I wrote it. I do now.
 
Noble was kind enough to tour a Savant, and I got the chance to spend just under 2 weeks with them. I actually wrote a pretty honest review – and I was dead set sure I'd covered all the angles. I loved the IEMs – but I wrote something which then got parroted quite often. I said they had a problem with the sub-bass.
 
So I need to paint a picture before I continue so that you know now what I didn't realise then.
 

• I had been listening to a lot of triple hybrids for a while and while I thought I knew what quality bass sounds like – I know now I was heavily skewed toward the IEMs I'd had experience with. And quite a few of them had enhanced sub-bass.
• I measured the Savant (I spent hours doing it) – but my measurement rig at the time consisted of an SPL meter, some tubing, test tones, and a spreadsheet. I measured different frequency responses, and basically used those to build a graph using C weighting and then a conversion table provided by Head-Fiers twj321 and DJScope to show what I'd found.
• And now the biggie – I created the graph before I did the critical listening

 
So what happened? This beautiful sounding IEM was critiqued by me because of something I saw on a graph before actually noticing aurally – placebo anyone? After graphing it, and there is no way my graphs were 100% accurate (although I did not know it at the time), I then compared it with other IEMs with enhanced sub-bass, and incorrectly drew the conclusion that the Savants were sub-bass light.
 
Here is my initial graph.
 

 
Here is my comparison graph.

 
It wasn't until many months later (with my current measurement rig) that I got to measure them properly. And this time here is the correct measurement.

Big difference huh?
 
Note that even this one isn't correct above 4-5 kHz - as my coupler is not calibrated 100% - but I'm pretty confident with everything below that.  Also know that this is raw uncalibrated data.
 
I made a very bad call, I did it publicly, and I was wrong – very wrong. I since updated the review with the new graph, and if I ever get the chance again with the Savant – I'll rewrite the review completely – but this time with far wiser eyes/ears and a more open mind.
 
I take this opportunity now to apologise unreservedly to Noble, to the people who may have been misled by the review, and particularly to Dr John Moulton.
 
And a special note of thanks to Jude who (when we were having a chat about measurements) was gracious enough to nicely give me some advice about what I did wrong.
 
Reviewing is a learning game – none of us is perfect – and especially not me.
 
So what about now?
So have I improved? I'd like to say yes – but I still make mistakes along the way. I'm a lot more aware of them nowadays though – and more importantly I'm very open to being corrected, and also going back to correct mistakes. Take my Brainwavz S3 review – have a look for the corrections in red (I think I updated those 3-4 months ago). None of us is bulletproof – we all have faults, bias and ego. The measure of the reviewer (in my eyes anyway) is how much we are open to correcting those errors and learning from them.
 
Fast forward to the present time, and if anyone has seen the QT5 thread, they'll notice some real discrepancies. I only got involved because someone mentioned the new Fidue Sirius as being overpriced (funnily enough they hadn't heard them yet still tendered that opinion) – when there are other 5 driver hybrids around at a fraction of the price. The QT5 was mentioned so I investigated. I found someone in NZ who had a pair, arranged to swap them for my 64Audio Adel U6 for a week, and proceeded to review them. They are among the worst IEMs I've reviewed in my entire time as a reviewer (not the worst – but getting there). The review is here – if anyone is interested.
 
The point is that they were touted (by more than one source and on other review sites than Head-Fi) as being basically 5 star earphones. Since I reviewed them I've had a lot of PM messages thanking me for posting the review and expressing the wish that I'd had the chance to review them earlier (before they had bought them). The stories have all been the same. They were expecting something amazing and got something disappointing. I'm only one data point - but the general consensus seems to be emerging that either ZhiYin's QC and consistency is all over the place, or that reviewing standards need to be lifted. I'm not pointing fingers - it could genuinely just be the QC (I do have my doubts though given the picture that seems to be emerging).
 
What can we do?
So for those prospective or current reviewers out there – my advice is just to be aware that the advice we give in our reviews leads to people spending real money. We owe it to them, and to ourselves to question (continually) everything we write. We need to be more objective (and that means all of us – and especially me). And if we make mistakes – we need to own them, and we need to correct them.
 
We can all lift the standard – but to do that we also need to recognise that we all have room to improve.
 
Thanks for reading.
 
Paul

 

[Brooko]

One more point to prospective reviewers which I forgot to mention earlier.
 
We don't write a review for the manufacturer
 
Or for the store who may have supplied the product
 
We don't even write the review for the readers / prospective buyers
 
We shouldn't write it for anyone.  We should merely review the product - give an honest opinion of what we think of it. As soon as we start writing for anyone but ourselves - we introduce bias - and if we ever want to minimise that, the place to start is stripping away any influencing factors
 
Food for thought anyway.

 

[Hawaiibadboy]

You are self aware.
 
Stubborn to step in the shoes of another and see yourself from their view.. but ...able to do it if needed.
 
That and the competitive drive made you the best reviewer on this site bar none. I get great pleasure in watching the cream rise to the top.
Time is the great equalizer. It catches up to everyone and that is a good thing or bad thing depending on how folks live their life.
 
Turning out good for you,this site and folks who trust in it
Bad for less than honest,self serving folks.
 
I tell my students,
Next time life is feeling just too awesome to believe...stop and check a mirror cuz that is the person who put you there.
Next time life is feeling like too much stuff to handle...stop and check a mirror cuz that is the person who put you there.
 
Kids get it right away and adults much slower if at all.
 
Folks who cannot see the problems are likely part of the problem. Time will catch them. They won't look in the mirror so it's up to time, and enough folks with no title beyond the strongest...consumer..no emotion beyond anger and no review skill beyond "this sucks" , to come together and cause an obvious question to come to mind.
 
 What?
 
And two basic options
 
"A"
Did a bunch of folks with no relation besides owning the same item suddenly lose their sense of hearing?
 
or
 
"B"
 
Did one reviewer with more than no relation get carried away and separate folks from their money based on pure BS ?
If the simplest answer is the most likely one, which it often is, then "B" it is.
 
 
 
 
You have become a pillar of the site and of great benefit beyond well structured and honest reviews easily consumed by the paying public.  Your standard setting work is and will ruffle feathers among your peers. They can glad hand you all they like but in a competitive game it is what happens. Enjoy their sweet tasting tears. They can't even broach the disclaimer section of the review without stuffing up and looking bad. They are so far behind you. Enjoy being on top.

 

[1TrickPony]

Thanks Paul for the insightful post(s). I'm really glad to see a dedicated and outspoken contributor like yourself set great examples like this! Cheers, more good times to come!

 

[twister6]

Paul, I can actually relate to this experience, and still remember how a few year back in some reply I was arguing about "micro-details" I hear while listening to W40, while people were telling me "how are you getting micro-details from a lush warm tuned iem?"...  But I also think a lot of the early mistakes are not intentional but rather due to a lack of experience.  There was a number of earphones from the past I thought were the greatest thing ever because at that moment in time they were the best IEMs I have tested.  With more review experience and a chance to hear premium products, your perspective changes and adjusts accordingly.  That's a reason why I'm so against star review ratings because within a short time period it becomes irrelevant and outdated, and you are stuck with a choice of either leaving it as is or going back and re-evaluating everything.  To this date I get PMs from people asking which product to choose, referencing something I've reviewed in the past and recently with a similar rating.  I almost feel like saying, "it was a mistake to rate the older product higher..." but then I come to realization that it was rated high relative to that time period and other products I heard in comparison.  My point, we are not perfect and make mistakes, but sometime mistakes are not intentional.
 
But your main point is well taken.  It was a lot easier recommending budget IEMs, but now with a chance to review TOTL products, it becomes a lot harder and I'm starting to feel responsibility pressure.
 
@Hawaiibadboy : Chris, wonder about the difference in reaction you get from saying this to kids in New Jersey vs to your students in Japan.  I can only guess, you might hear back a few of the words that got censored under the asterisk in your reply

 

[Brooko]

Alex - that is a wonderful and very true point.  Experience is a great leveller. Things you thought were great when you started are brought down to earth when compared again with the benefit of experience - especially of other gear.  But why should a review stop at a moment in time?  I've gone back and edited reviews later (months or even years later).  I now make any changes in red - so that people can recognise the edits, and I try to date them as well.
 
Unfortunately the biggest issue is getting enough time to do it - but the rewards are there when you do.  You not only make your body of work more accurate - but you also learn a lot about yourself along the way.  Sometimes those lessons are good.  Sometimes they are very hard.  As long as you are completely honest when doing them - they can be extremely rewarding personally.

 

[Brooko]

Chris - thanks for the kind words - and my discussions with you in the early days are part of what set me on this path.  I hope you don't mind - but I've slightly edited you post - it would only get flagged.  Rules are rules.  I get the sentiment though
 
I might also add at this point (and this isn't just modesty at play) - there are still many reviewers on here I look up to - I am simply one data point. But when the sea rises, all boats are lifted.  If we all make a concerted effort to be better at what we do, the results are better for all.

 

[Cinder]

As someone who is just now getting into reviewing headphones, I thank you for sharing your wisdom. I think that even today I let my lack of experience get the best of me with my most recent review.
 
It's a little frustrating though. I know I need more information to make a completely solid recommendation, but lack the time/means to be able to gather the information. One day though, I hope I can be helpful in a way that the giants of Head-Fi are now.

 

[Loquah]

Great article, Paul. I feel similarly to you and Chris about the way our preferences change overtime as we experience new products and learn more about the industry and science of audio.

To offer a point of opinion (i.e. a thought, not a truth), I decided a while back to err away from measurements and objectivity and to focus on the subjective experience of listening for two reasons. Firstly I didn't own or have the desire to invest in testing gear like you and other reviewers. Secondly, I felt that I could never make objective conclusions with certainty (i.e. that a certain frequency response was good / bad) because I still feel there is so much we don't completely understand about auditory perception with earphones / headphones. I don't say this to denigrate the use of objectivity in reviews. In fact, I seek out reviews like yours for this very reason. My point is that, like you and many others, my reviewing style has evolved and matured into my own style and hopefully it will continue to improve. I sometimes look back on old reviews of mine with a hint of embarrassment, but then I remember that we all have to start somewhere.

Thanks for the great article to remind us all that we're all human, all fallible, but also always improving if we have the humility to acknowledge our mistakes.

 

[Brooko]

Thanks Lachlan
 
The beauty of getting multiple reviews is that slowly it build into a consensus - and that I think is what most people are looking for.  Its good to have a variation of styles and content too.  I'm often told mine are too wordy, too dry, and pretty much lacking much emotion. I actually don't mind the critique - it just means that they are looking for another view point or style - and that is where other reviewers help complete the whole picture.

 

[Loquah]

Thanks Lachlan

The beauty of getting multiple reviews is that slowly it build into a consensus - and that I think is what most people are looking for.  Its good to have a variation of styles and content too.  I'm often told mine are too wordy, too dry, and pretty much lacking much emotion. I actually don't mind the critique - it just means that they are looking for another view point or style - and that is where other reviewers help complete the whole picture.

Exactly. There are reviewers who I consistently rely on because their body of work tells me that I either agree or disagree with their preferences. Both are valuable.

 

[Brooko]

I was able to get a proper reading on the Savant, and thought his might be of interest.  I can't mention the source - but I know it was performed with a  IEC60318-4 G.R.A.S.ear simulator - so it is a recognised standard.
 
Here's a true reading on industry standard, certified equipment.
 

 
And here in comparison again is the recording I took with Veritas
 

 
You can see why I'm overall reasonably happy with the readings I take - and especially when they are simply used to contrast with other gear on the same equipment.  My ultimate would be able to afford a set-up that is industry standard - but realistically I think it is a pipe dream.  The slight mid-bass difference could be the tips used, and could be the seal / insertion depth etc.  Anyway its not a million miles away.  The 5-12 kHz does measure low on Veritas - but as long as people are aware of this, they can allow for it - I always state it.
 
And when I get the chance I aim to send half a dozen or so IEMs to someone who has a similar industry standard, and hopefully use their data to calibrate mine.

 

[Loquah]

  I was able to get a proper reading on the Savant, and thought his might be of interest.  I can't mention the source - but I know it was performed with a  IEC60318-4 G.R.A.S.ear simulator - so it is a recognised standard.
 
Here's a true reading on industry standard, certified equipment.
 

 
And here in comparison again is the recording I took with Veritas
 

 
You can see why I'm overall reasonably happy with the readings I take - and especially when they are simply used to contrast with other gear on the same equipment.  My ultimate would be able to afford a set-up that is industry standard - but realistically I think it is a pipe dream.  The slight mid-bass difference could be the tips used, and could be the seal / insertion depth etc.  Anyway its not a million miles away.  The 5-12 kHz does measure low on Veritas - but as long as people are aware of this, they can allow for it - I always state it.
 
And when I get the chance I aim to send half a dozen or so IEMs to someone who has a similar industry standard, and hopefully use their data to calibrate mine.

 
That certainly confirms what I heard with the Savants and why I didn't disagree with your initial thoughts regarding the bass / sub-bass. I admired the sound of the Savants from a technical perspective, but couldn't warm to them based on my personal tastes. One thing I've never fully understood is when / how the raw data from some measurements is then adjusted / compensated for. This can lead to significant confusion if different people are compensating the raw data differently. What's your approach to this, Paul?

 

[Brooko]

 
That certainly confirms what I heard with the Savants and why I didn't disagree with your initial thoughts regarding the bass / sub-bass. I admired the sound of the Savants from a technical perspective, but couldn't warm to them based on my personal tastes. One thing I've never fully understood is when / how the raw data from some measurements is then adjusted / compensated for. This can lead to significant confusion if different people are compensating the raw data differently. What's your approach to this, Paul?

 
Jude and I had a good discussion on this last year during our long telephone call.  I just use raw data - but I would like more accurate raw data.  I know different people put different adjustments depending on equal loudness contours etc.  Originally I wanted to do that too - but there are so many different measurements being used, my preference is simply for consistency.  The only thing I'd like to get done is to eventually adjust my rig enough so that the raw measurements match an international standard (like the GRAS uses).
 
At the end of the day though - mine are "close enough", and more importantly they are good for providing comparative data.  If someone says one IEM is bassier than another I can check both and tell for sure.
 
As long as people state what they are using when they post the graphs - then I guess interpretation is up to the reader.

 

[Hawaiibadboy]