[Brooko]

Ian / Claritas - thanks for the input. I think one thing that helps - especially in the summary/conclusion section of a review - is simply to state your reason for the score you've given. Everyone will rank differently, but if you can explain why you've scored the way you have, then it brings real meaning to the review.

 

[62ohm]

I would really love to hear your thoughts on my turntable setup, Paul 

 

[glassmonkey]

 
I've noticed that some of the time, posted reviews are little more than an advertisement for a manufacturer - or at least that is how I see them - and this trend is helping no-one.  The pictures are pretty, the specs are quoted, but there is very little thought gone into the actual review.  Where are the actual comparisons with other gear?  Where are the real world measurements? I take it pretty seriously - because what I write might influence people who are spending real money.  Honestly, I've seen some reviews lately and I have to wonder what people are thinking - especially when I get to hear the exact same gear, and their "reviews" are miles away from my own experience.
 
That was part of the reason for the blog - to see if together we could all raise the quality of out reviewing standards (and I include myself in needing to improve).

I'm with you there. I've definitely read some reviewers who churn out too many reviews that are nothing more than puff pieces. I don't want to be one of them.
 
I think that we need to be conscious that a person's review being 'miles' away may just be a matter of different experience and different ears. I have a friend who if the treble on a headphone doesn't roll off sharply after 16k they aren't likely to like the headphone, while I love treble extension and wouldn't find that harsh. Both of us can describe what we hear, though, thus allowing others to make informed decisions off our impressions. Not everyone has the audio vocabulary to describe experiences well, either. I find that I struggle with this and an audio lexicon with concrete descriptions would be really helpful. I also need to study all the different instruments that I might listen to so I can make better descriptive comments without resorting to onomatopoeia. This difference in experience isn't just among ecstatic noobs either.
 
I personally rate stuff by whether I like it, so I may need to add a section to my reviews for an about me. I don't tend to be a bass-head and have tended towards neutral phones, but find myself enjoying quite a bit of v and u shape signatures now. If I'm listening to a bass-head phone I'm likely to rate it lower than a bass-head.

 

[Brooko]

  I think that we need to be conscious that a person's review being 'miles' away may just be a matter of different experience and different ears. I have a friend who if the treble on a headphone doesn't roll off sharply after 16k they aren't likely to like the headphone, while I love treble extension and wouldn't find that harsh. Both of us can describe what we hear, though, thus allowing others to make informed decisions off our impressions. Not everyone has the audio vocabulary to describe experiences well, either. I find that I struggle with this and an audio lexicon with concrete descriptions would be really helpful. I also need to study all the different instruments that I might listen to so I can make better descriptive comments without resorting to onomatopoeia. This difference in experience isn't just among ecstatic noobs either.
 
I personally rate stuff by whether I like it, so I may need to add a section to my reviews for an about me. I don't tend to be a bass-head and have tended towards neutral phones, but find myself enjoying quite a bit of v and u shape signatures now. If I'm listening to a bass-head phone I'm likely to rate it lower than a bass-head.

 
Great post there - thanks. I agree that we all hear differently (or I should say have different preferences), and I have no problem with that.  As long as you state what your preferences are, then the reader can interpret what you are saying, and hopefully relate that to their own preferences.  I'm probably a bit like you GM - I don't generally prefer the darker warmer side either, and I value clarity and at least some upper end definition - but I clearly state that in every review I write.  More people are doing that now which is a great thing.  When I first started, there were only one or two.
 
Your comment on the roll-off after 16K is an interesting one too. I wonder how many reviewers have taken the time to find out what they can actually hear, and what frequency changes will do?  I've taken my favourite pieces, and with a 31 band graphic equaliser I've experimented with cutting some frequencies all together, introducing slow and fast roll-offs, basically doing as much as I can to find out what affects me.  I could cut from 16 Khz up (rather than a roll-off), and for me personally (with music playing) it makes zero difference (YMMV). I wonder how many on head-Fi would be similar?
 
I know that at normal listening levels I can hear tones up to about 14 kHz, but 15 kHz for me is a complete blank (I think that is where my tinnitus kicks in).  I get a very faint signal at 15.5 kHz, and then nothing again beyond that.  With music actually playing practically everything above 12kHz is going to be effectively masked for me anyway.  At the other end (surprisingly) I can hear right down to about 22 Hz, and 30 Hz up is very clear. Yet we worry about the highest fidelity possible, and debate for hours about extremes that I'd think most of us would likely not be able to hear anyway.  The nature of our hobby.
 
Anyway - getting back to point - every serious reviewer should ultimately know themselves and their own limitations before they even attempt to tell someone else their own experiences with a product.  Doing otherwise is pointless. How can you explain subjectively or objectively how something sounds if you don't even know you can hear it, or more importantly what effect it has on your listening experience?

 

[castleofargh]

http://rationalwiki.org/wiki/Dunning-Kruger_effect you can see this in reviews or anywhere else. a few times I went head to head with a review that was IMO a total nonsense(not on headfi I would get banned for saying the stuff I said

). the most obvious thing every time is that the guy has no idea he's doing something wrong. he just got his new toy and like with each toy, he's overjoyed for 2 or 3 weeks and makes a review of how happy he is. nothing to do with the device tested, most of the time he could have bought a new watch and tell us how much it improved the soundstage for all that matters.
we can usually catch those guys with how overly positive they are if we also tested the device, but if we didn't, IMO a good cue is to look at how fast his brilliant gear will be sold and replaced by another so called brilliant gear.
 
and the other way around, I often discuss with people that have a pretty good hearing and understanding of where a device stands compared to the competition, but they will never do a review because they believe they're not qualified to do one(which really is our loss IMO).
 
 
I'd like to +100000 the EQ part. how can a guy explain the sound of a device if he can't identify a 5khz spike or where the signal rolls off? even the EQ haters, and we know it's still the majority of people here, should practice with a nice EQ on the device being tested as a way to identify the strength and flaws of the signature. just going "on the song XXXXX the second guitar is a little less vivid than when I use that other device". that will help 3 people who have at least the other device and will try the song. but if we talk about frequency response, then the guy can go move that frequency on his own system and understand what we're talking about pretty much all the time. an EQ is a great tool for that job IMO. when you talk about how the trebles are too strong with someone and you realize his idea of trebles was 3khz, Houston we have a problem.
 
about the stars I have no idea what I'm doing! and most of the time I guess I'm just rating if I found it was worth it in a very very insecure way ^_^. also I wouldn't review something I would rate 1 or 2 stars, so in my case the stars have close to no meaning.

 

[seanwee]

But what if the stars were used by readers to rate one's review?
 
anyways , IMO most ppl only read 4 star and above reviews and skip the lower rated ones entirely.
And if they were interested in any product and was researching it , a low star count may cause them to skip the review entirely, thinking that its not worth it.(which is probably true most of the time)
 
if i were to rate something , i would rate it based on how well it competes with other products in the price range.

 

[castleofargh]

  But what if the stars were used by readers to rate one's review?
 
anyways , IMO most ppl only read 4 star and above reviews and skip the lower rated ones entirely.
And if they were interested in any product and was researching it , a low star count may cause them to skip the review entirely, thinking that its not worth it.(which is probably true most of the time)
 
if i were to rate something , i would rate it based on how well it competes with other products in the price range.

to rate the review the thumbs up already do it. they decide the sorting order in the list of reviews about a device. if you have no "like" on a hd600 review, you end up page 5 and no number of stars will take the dust out of your review.
it's a jungle out there! ^_^

 

[ryanjsoo]

Really interesting read, Brooko, always found your reviews accurate and reliable, part of the reason why I started writing reviews and a small blog (I`ll have to look into a disclaimer...). Anyway, thanks again for sharing, really interesting stuff!

 

[Brooko]

No problems - been impressed with yours as well. The one thing you're missing is the means to mix the objective with the subjective.  If you get a nice mix of both - then your reviews become not only informative, but also reliable.  The equipment does need to be expensive either
You've got better skills with the camera than I have too 

 

 

[Jackpot77]

Hi guys - resurrecting a quiet thread here, but have been thinking a lot recently about how I review things and remembered Brooko's series on how he goes about things so dived in to the back pages for some sage advice.

I'm struggling to be objective with star ratings, as I always try to rate an item based on overall quality as well as my own personal preferences (darker and more musical over neutral and sharp, which isn't everyone's chip of tea). I have some Campfire gear for review at the moment, and although one model is technically (and audibly) better than the other, the tuning just didn't click with me in the same way. In that scenario, it feels wrong to rate the "worse" gear with a better star rating, but if I had the choice between the two, I'd listen to the so-called "lesser" IEM each time.

I always end up explaining my ratings now at the end of any review, giving my "personal" star rating based on my own preferences in the text and my "objective" rating in the main slot. Anyone suggest a better or clearer way of doing things? I should add that for the Campfire gear it's more splitting hairs than night and day as both products are very good, but I'm wary that anyone who goes to the effort of reading something I wrote should get the clearest possible indicator of where I think the gear sits, especially in this sort of price range. Any advice very gratefully received!

 

[Brooko]

I can only speak for myself - but here is how I generally do things (in this example, IEMs).
 
Take into account:
 

• Overall build
• Fit
• Comfort
• Accessories
• Sonics - and here I try to look more objectively, and pas my usual preferences.  I'm now evaluating sub-bass, mid-bass, low-mids, upper-mids, lower-treble - bit all relative to each other. I'll mark lower if one part of the band is grossly over-emphasised or under-emphasised relative to other frequencies. EG - I'm Ok with a U or V shaped signature - but not if the bass is pumped up so far that it bleeds into the mid-range.  I'm OK with a brighter signature - but not if it becomes too piercing.  I'm OK with a coloured upper-mid-range, but not if it becomes too unnatural sounding, or diffuse and grainy.
• For signature which I really don't like - if they are still cohesive, and I can EQ, then I tend to be more lenient - but I mention where they disagree with my own personal preferences
• I comment on staging and imaging, but generally don't mark down too much unless they are deficient.  EG I'd rather a smaller but circular stage than a wider by one dimensional.
• Most importantly I looks at a series of IEMs to try and subjectively gauge what is good (middle of the road) for the price point, and then look at the review IEM's performance against that benchmark.  I've found a few IEMs that have very few flaws, but are priced way out of their bracket.  That will tend to get an average score from me (3 to 3.5).  I have no problems giving a far cheaper IEM an almost perfect score if they punch way above their price point though.

 
Not sure if this will help - but it might give you insight into how I do it.  As long as you bring a measure of relativity and consistency to a review - then that is all you can hope for.
 
Classic example for me is my 64Audio U6 and U10 review.  U6 was a 4.5 and the more expensive U10 was a 3.5.  The m,ain difference was the Adel module vs Apex, and the price.  U10 is a great IEM - but the U6 is exceptional for its price point. 

 

[Brooko]

I might add that I've noticed a disturbing trend amongst some of the newer crop of reviewers - to hype and give overly glowing reviews of some products.  IMO its not what our role is.  We're not here to sell the product (and if we start doing that - its time to give it up).
 
We're here (or at least I'm here anyway) because I enjoy writing, and I enjoy giving an honest viewpoint (objectively subjective) about how I view a product.  I don't review for the gear - I've always been open about returning it if the manufacturer wants it back.  I review for myself - and because it helps this community get a read on possible performance.

 

[Jackpot77]

Thanks, Paul - really useful advice, and insightful as always.

I try and do most of the things you mention above, but will see what else I can add to my methods. One of the joys of being a "hobbyist" reviewer is that until very recently I only tended to review things I had physical purchased, so naturally tended to buy/trade stuff that I thought would fit my personal preferences, which makes it easier. Being involved in a few tours recently has broadened What I am listening to, so just wanted to make sure I am giving a fair representation of the merits (or deficiencies) in anything I end up writing about.

Also worried I have given a few glowing reviews recently, so trying to self-monitor to make sure it doesn't become my default style (I'm a naturally grumpy man!) - the IEMs I have rated have been truly outstanding for my personal tastes in their price bracket though (IT03 and Vibro Aria).

 

[CraftyClown]

Always a pleasure reading your blogs Paul. They're incredibly helpful to the starting out reviewer.

I did have one question for you though regarding choice of test tracks. Up until now I have just been using a selection of my own favourite music, as well as some binaural stuff including some Chesky test tracks to pinpoint imaging and soundstage.

Do you have a methodology for music selection when demoing gear? ie do you have certain tracks that you test lower mids with and another that you test sub bass with etc?

 

[materix01]

Very well put. You address a lot of what I worry and have thought about as a new reviewer. Especially agree with the trend of "hyping and give overly glowing reviews of some products". I certainly have done that unintentionally in my early reviews not because I wanted to shill for a product but because I thought that was what people wanted to read about. I'm still wondering what I would do when I review I product I really hate. I did receive one such loaner product of something I hated a couple of months ago that I'm returning next week, but I've been sitting on the written review for months now as I just don't know how to stay professional whilst being honest. How do I stay fair? I can give a brand the benefit of a doubt with something like a QC issue I encounter, but what if I think it sounds rubbish for the price. It worries me even more if it's a smaller brand with fewer products compared to say a larger brand.

Going to binge through all these blog posts tonight. Appreciate the writeup Brooko.