[avitron142]

 
There is no right or wrong way of doing it, unless if you receive a review samples and have a full review ready the same or the next day  

 
Oops. 

 Now we get into a problem when that review is sometimes more in-depth and accurate than others that take more time to publish... do we read it, or ignore it out of spite? But I've moved away from that, anyway.
 
Being serious though, I think it's important to mention how crucial the target audience is. Writing and rating a review on a budget headphone has very different mechanics than one of a more expensive product. That and there's an ongoing battle of how much should build quality/aesthetics/isolation/UI make up of the rating, in comparison to the sound. How much do design objectives count, when consumers are expecting a bit more? These are all questions that should garner some serious thought when writing a review.
 

  Quick message to any readers of the thread - please don't bother about clicking the report button on the post above me. Part of operating a blog is that I'm an actual moderator - so every time the report button gets hit, I get the notification.
 
Mr ThickT is entitled to his opinion, just as anyone else is.
 
I'm choosing to ignore it with all the hidden disdain it deserves.  Suggest you all do likewise.

 
Usually mods have that tag on their profile, I think that's where the confusion lies.

 

[Brooko]

@HiFiChris - thanks for the in-depth comments mate.  There does seem to be a common trend among some of us with regard to the thoroughness of preparation, the realisation that the longer we listen the more we form reliable impressions of the attributes, and the desire to be as accurate as possible for prospective buyers.  I see this as a huge positive.
 
Unfortunately the downside (as always) continues to be time constraints.
 
Really appreciate the feedback Chris - it helps a lot.

 

[Brooko]

  Thanks Brooko! All I really need is hands on experience--though I find that sometimes sound gets jumbled in my head and I can't clearly point out different points in audio. Any suggestions concerning this?

 
Great question - hopefully others will chime in with their thoughts as well - but here is what I've realised (personally) over the last few years - some of it may be helpful.
 

• The biggest challenge was associating what I thought I was hearing with meaningful data that people could relate to.  One of the best things I did was using this chart:
  http://www.independentrecording.net/irn/resources/freqchart/main_display.htm
  I then sat down with music featuring some of those intruments, a graphic equaliser, and started playing with the levels to see what frequencies affected my listening
   
• I looked for as much info on line as I could about audio workshops (Ethan Winer's ones are particularly enlightening) regarding what is audible, what is not, and what is important when it comes to listening and perception of sound.  Some of the videos were huge eye openers.  I would then take that information further - and test myself.
   
• Build your own selection of test tracks - tracks you know really well - that highlight different things you're going to be testing.  I have tracks for detail, staging, imaging, male and female vocals, bass quantity and quality.  If you have a particular track (say on imaging), and you know its a live track, see if there is a youtube link for the master track you're using (has to be the same recording).  I use one for one of Loreena McKennitt's concerts (I have the lossless live recording) but used the youtube link so I know the exact positioning of the various instruments.  That helped me so that I know when I'm listening to my recording - where the positional cues should be.
   
• Know your own limits and make sure you state them.  For example - I can't pick up hiss from IEMs most of the time - so I get my wife or daughter to help.  You can still be thorough even if your own abilities are compromised.  The secret is knowing we aren't all supermen with superhuman hearing 
  
  .  Personally - I try to keep the whole ego thing at bay.  I'm "merely human" and proud of it.
   
• Volume match - don't need to say a lot more about this.  It is one of the most important things in audio.
   
• In my experience, night and day differences generally don't exist or are overstated.

 
Hope some of that helps - I'll try and think of some more, and hopefully some of the other guys chime in as well.

 

[Brooko]

  Oops. 

 Now we get into a problem when that review is sometimes more in-depth and accurate than others that take more time to publish... do we read it, or ignore it out of spite? But I've moved away from that, anyway.

 
Generally (and definitely not being specific here) I find that the best reviews of product are the ones where the reviewer has taken adequate time with a product.  I've seen some reviews on this site which (unfortunately) amount to some pretty photos, some specs, some loose audiophile terms, and very little usable data.  In short - they are little more than marketing for the company.  Most of the reviews that do not take adequate time are sub-par in my experience.  The ones I've rushed, I've often regretted later (or had to amend).
 
Quote:

  Being serious though, I think it's important to mention how crucial the target audience is. Writing and rating a review on a budget headphone has very different mechanics than one of a more expensive product. That and there's an ongoing battle of how much should build quality/aesthetics/isolation/UI make up of the rating, in comparison to the sound. How much do design objectives count, when consumers are expecting a bit more? These are all questions that should garner some serious thought when writing a review.

 
Actually I can't agree with this.  The target audience is the listener.  They all pretty much want the same thing - good build, good fit, good sound, good features/design and good value.  The only thing that changes is the money they are prepared to pay, and that does affect the features, build, sound etc  The important thing for a reviewer is to state clearly how they are rating a product. And then to always be consistent.  I always take value into account.  You can get a 5 star product which may not sound as good as TOTL, nor have the same features etc - but has exceptional value for what it does deliver.  The danger is when you aren't consistent - especially if critiquing a manufacturer for features that one item of audio equipment. but not applying the same standard to another one.  At the end of the day - we all differ in our method, but we should be consistent in stating what we judge on, and in delivery of that verdict. And it doesn't matter if I'm reviewing a $25 IEM for Brainwav or a $600 IEM for 1964Ears - they both get the same time taken, and the same methodology applied.
 

  Usually mods have that tag on their profile, I think that's where the confusion lies.

 
Doesn't apply to the blog section. I'm a moderator only for my blog - not the site.  Guys like Jason and Mike have the same ability in their own sections. Thats why I stated it  - so that people realised it was easier to ignore ThickT's post rather than report it. At the end of the day a drive-by comment like that isn't worth replying to anyway.  Says much more about his character than mine.

 

[avitron142]

 
Actually I can't agree with this.  The target audience is the listener.  They all pretty much want the same thing - good build, good fit, good sound, good features/design and good value.  The only thing that changes is the money they are prepared to pay, and that does affect the features, build, sound etc  The important thing for a reviewer is to state clearly how they are rating a product. And then to always be consistent.  I always take value into account.  You can get a 5 star product which may not sound as good as TOTL, nor have the same features etc - but has exceptional value for what it does deliver.  The danger is when you aren't consistent - especially if critiquing a manufacturer for features that one item of audio equipment. but not applying the same standard to another one.  At the end of the day - we all differ in our method, but we should be consistent in stating what we judge on, and in delivery of that verdict. And it doesn't matter if I'm reviewing a $25 IEM for Brainwav or a $600 IEM for 1964Ears - they both get the same time taken, and the same methodology applied.

 
I dunno... didn't you say yourself that people are willing to spend lots of money on incremental upgrades, that may not be revolutionary? That wouldn't seem to apply to a budget item - I think we can agree that the higher end items have a lower rate of return on the money you're spending than budget ones. I.e. a $600 IEM is not 24x better than the $25 Brainwavz, etc.
 
The same methodology applies, for sure. The same amount of time as well. But don't we have to take into consideration that the $600 product is for people who don't mind spending more money on more subtle incremental upgrades? I mean, that applies to the whole summit-fi category; a few hundred dollars difference is inconsequential to what customers want - top audio quality, even if it comes at a higher rate of cost.
 
I think there's definitely more than one type of target audience when it comes to Head-fi - some would much rather prefer more features and easier UI, even if it comes at a cost of some SQ, and others (the "hardcore audiophile" type) couldn't care less if the UI is pretty, or the design looks good - sound is the number one priority, and if it's functional, then that's all they need. The S6 and M2 have very different values depending on who's viewing it. People who are just coming into this whole community may feel like a portable amp is a waste of money - since it takes up so much space, and doesn't have a massive jump in quality. Others will have this entire portable rig that's heavy and huge, but has a blazing good SQ.
 
Similarly, you have some people (college students for example, like me) that are personally looking for the absolute best price/performance ratio - and wouldn't even think about buying TOTL headphones; we'd consider many of them overpriced. Others are on the quest for audio Nirvana, and spending lots of money for small improvements is what the game's all about. The audience is different depending on the item, the price, and who's going to use it. Maybe I'm wrong about this... I don't see where though. I think it's important to say who you're "rating" the product for, and who might have a different take on it.

 

[twister6]

@avitron142 : Avi, I do agree that different products will be targeted to a different audience, but as a reviewer did you do anything different in your write up about $800 Savant vs $125 Titan?  You just write your honest opinion about what you see and what you hear and how it compares to other headphones, and if it's a good value or not.  Then it's up to a reader to decide if those are fitting their budget.  You know what I found from numerous PMs I get weekly from people asking me to recommend either headphones or DAPs or my opinion "what I really think about it" - so many people come to a conclusion that instead of buying 10x $25 IEMs they should have invested into one decent $300 model, and not necessary because it will be 10+ times better, but because it will give them a level of refinement to last for a long time and will hold a value when they can afford to upgrade it (by selling) to something else.  That just tells me that audience who is on a budget still curious to read about mid-fi and summit-fi headphones to add it to their wish list and they do appreciate the effort we, as reviewers, put into our work.  Unless I misunderstood the argument about the target audience in this thread, I personally find it irrelevant because I'm not trying to sell the product, but rather doing my best to describe the product, make my write up a little less intimidating (technical) and easier to read for all the audience, and let the audience decide if it fits their budget.

 

[avitron142]

  @avitron142 : Avi, I do agree that different products will be targeted to a different audience, but as a reviewer did you do anything different in your write up about $800 Savant vs $125 Titan?  You just write your honest opinion about what you see and what you hear and how it compares to other headphones, and if it's a good value or not.  Then it's up to a reader to decide if those are fitting their budget.  You know what I found from numerous PMs I get weekly from people asking me to recommend either headphones or DAPs or my opinion "what I really think about it" - so many people come to a conclusion that instead of buying 10x $25 IEMs they should have invested into one decent $300 model, and not necessary because it will be 10+ times better, but because it will give them a level of refinement to last for a long time and will hold a value when they can afford to upgrade it (by selling) to something else.  That just tells me that audience who is on a budget still curious to read about mid-fi and summit-fi headphones to add it to their wish list and they do appreciate the effort we, as reviewers, put into our work.  Unless I misunderstood the argument about the target audience in this thread, I personally find it irrelevant because I'm not trying to sell the product, but rather doing my best to describe the product, make my write up a little less intimidating (technical) and easier to read for all the audience, and let the audience decide if it fits their budget.

 
I'd agree, but unfortunately we have to give a rating of the product, and a conclusion as well. On top of that, in each category, we aren't only giving over information, but judging it (build quality, UI, comfort, etc.) as well - that should matter depending on which perspective we're coming from.
 
Although the bridge is being gapped, I'd say that the ones asking the questions have enough trust in you to think your mindset corresponds with theirs, which unfortunately can't be true for everyone from a statistical perspective. The "budget people" are probably not so budget if they're asking about higher quality equipment; either that or they have some free money to spend on. We've all drawn this magical line of "that's enough value for me", but the lines differ for each person - I only put it into two groups because it's easier to conceptualize that way. I'm sure the lines are blurred more than I made it to be, even if the range is still there.
 
"That just tells me that audience who is on a budget still curious to read about mid-fi and summit-fi headphones to add it to their wish list and they do appreciate the effort we, as reviewers, put into our work"
 
Dreams, my friend, dreams. We all wish we can win the lottery and buy a Stax. At least for myself, it's a great thing to daydream about, even if there's no way on earth I'd be able to have enough money to forgo the "value" factor.
 
"As a reviewer did you do anything different in your write up about $800 Savant vs $125 Titan?"
 
Sure. The rating. I rated the Savant as a $800 item as I think a $800 item should perform as an expensive piece of gear, not 6x the Titan. I tried coming in with the mindset that the improvements in performance wouldn't be staggering. The same went for the m9XX I reviewed; I try coming from a financially richer perspective, and go from there - instead of giving it a 1.5 star rating automatically "there's no way in hell that this item should be so expensive, compared to the Titan it's not twice as good to justify the price jump!"
 
I wish we could just describe what we see. Value judgement is something I think all reviewers do though, and we have to adjust accordingly. That's why comparisons are so useful 

 

 

[avitron142]

@Brooko The issue I have with brain burn-in is that we will get used to essentially any headphone, no matter the flaws, over enough time. Does it mean the headphone is good? Maybe not, but our brains got used to the sound signature all the same. So then it seems that a review is unnecessary altogether - since at the end of the day a customer will be fine with it in the long run. I'd say that the newest gear is not the one we know the least, but the most - since our brains haven't compensated yet for the flaws it may have.
 
But I agree completely with the Dunu comment. Although I kept the review the same for build quality, comfort, isolation (that varied a bit for me, actually), etc., when it came to sound they could have been from two different companies, for all I know. Th worst type of review (and the one that actually applied to a DUNU review), is reviewing a headphone and not covering a sound section. Like... that's the most important part of the headphone...

 

[Tom22]

  @Brooko The issue I have with brain burn-in is that we will get used to essentially any headphone, no matter the flaws, over enough time. Does it mean the headphone is good? Maybe not, but our brains got used to the sound signature all the same. So then it seems that a review is unnecessary altogether - since at the end of the day a customer will be fine with it in the long run. I'd say that the newest gear is not the one we know the least, but the most - since our brains haven't compensated yet for the flaws it may have.
 
But I agree completely with the Dunu comment. Although I kept the review the same for build quality, comfort, isolation (that varied a bit for me, actually), etc., when it came to sound they could have been from two different companies, for all I know. Th worst type of review (and the one that actually applied to a DUNU review), is reviewing a headphone and not covering a sound section. Like... that's the most important part of the headphone...

I'm not brooko but I thought I would chime in.
 
In regards to the brain burn in, I think, the "getting" use to part of it maybe misleading.
 
from my perspective I find the "getting use to part" is more so giving it a chance so to speak, because for example and bear with me here
 
if I was say using my hifiman re400 (which could be considered as very balanced with a hint of warmth), which caters to a different kind of sound and audience compare to lets say I review the Nuforce NE800M (which is much bassier and much warmer in comparison).
 
then I jump to say the Dunu Titan 5 (which is a bit more on the bassier side> re400, but brighter sounding then the ne800M)
 
in all three scenarios you have three earphones that have different sound signatures and if you were to enjoy the hifiman re400 (which I really do),
 
then jump toward the nuforce ne800M, you could immediately tell there is a such a thick lower mids area that would impact its speed and clarity for bass notes compared to the leaner re400.
 
and then jumping immediately to the dunu titan 5, you would find the titan 5 overly bright compared to the very smooth, chocolately ne800M.
 
 
if you were to base your reviews on those initial impressions alone then I feel it might not be as indicative of the overall presentation of its sound.
 
why is that?
 
because its unlikely that someone were to come from the same direction in terms of "earphone-jumping" as you.
 
I think those interested in a more bassier sound, the ne800M would appeal to them more so, and they may never even consider the hifiman re400 for example. and if they were to see that I said that the ne800M sounds slower RELATIVE to the hifiman 400 (that would be accurate). but evaluating the ne800M based on what it is,
 
a bassy, smooth sound, that would require a bit more adjustment for the brain (at least for me) and then I can properly evaluate it based on its merits for its "target sound" and "target audience".
 
similarly, the dunu titan 5 is a on the brighter end and some users may not even look at the nuforce ne800M (because its a lot smoother overall), maybe too smooth for some, maybe to the point of being muffled to some.
 
I feel like its a period that helps "reset your brain" and ears so to speak.
 
and I don't think if something say sounds terrible at first listen, and after many weeks. (it still sounds terrible, and you may get use to it in a sense (because you spend $X amount of money on it, but that doesn't change the fact that your not satisfied.

 

[Brooko]

What I'm saying is that if you are consistent with your measure of quality (build, fit. comfort, sound etc), and then apply value to that, you get a consistent guide to everyone no matter what their budget.  The Titan 1 is phenomenal value considering price, sound, build, practically everything about it.  It gets a 90-100% rating from me because it does that amazingly well considering the value it represents.
 
Now take the Thinksound Rain 2 which wasn't far off the same price. It's an OK earphone.  But doesn't reach the same heights and has what I consider flaws.  It's roughly in the same price bracket, and I'm consistent in my ranking based on the criteria I set.
 
But go up in price or down in price, and apply the same consistent method of ranking.  Good qualities, flaws, but then apply my perception of value.  The important thing is that I do this for every review.  It may be a little different for me because I know exactly what to expect from diminishing returns, I have bought +$1000 audio items (not review samples), and I know my own expectations.
 
Everyone will have their own different criteria, but my point is that they should state as much as possible what those criteria are when ranking, and be consistent.
 
You can't try and apply logic to who you think is the probable audience.  That's you making assumptions, and you not being accurate.  IMO you need to always be consistent with your reasoning, and application of how it appears to you.  You are the reviewer - no-one else.  You try to bring in as much objectivity as you can - but at the end of the day it is a subjective opinion.  The measure of the popularity of a reviewer is how consistent they can be, and how well they can convey that to the reader - so that then they can interpret it applying their own personal bias.

 

[avitron142]

It's not about the money, I legitimately think that after time with any headphone the issues pretty much dissolve, it's something I'm not spiting but agreeing to.
 
As for the "jump", as you call it, that is a serious thing that can trip up reviews. To get around that, I usually rotate the headphones I use between the ones I have, so I'm not used to any one headphone. 

 

[Brooko]

  @Brooko The issue I have with brain burn-in is that we will get used to essentially any headphone, no matter the flaws, over enough time. Does it mean the headphone is good? Maybe not, but our brains got used to the sound signature all the same. So then it seems that a review is unnecessary altogether - since at the end of the day a customer will be fine with it in the long run. I'd say that the newest gear is not the one we know the least, but the most - since our brains haven't compensated yet for the flaws it may have.

 
Again - I know where you are coming from, but you missed the point.  How can you actually know a headphone if you're knowledge of it is so clouded by what you have been previously listening to. If you are used to a warm headphone, you'll hear a neutral one as bright.  How accurate is that?  Get intimate with the sound signature, get to know it thoroughly.  The when it's time to write it up, start comparing.  Going in cold does no-one any good.  If you're reviewing something within 24-48 hours of hearing it (and this is just my own personal opinion), you're doing both your readers and the manufacturer a huge dis-service.

 

[avitron142]

  What I'm saying is that if you are consistent with your measure of quality (build, fit. comfort, sound etc), and then apply value to that, you get a consistent guide to everyone no matter what their budget.  The Titan 1 is phenomenal value considering price, sound, build, practically everything about it.  It gets a 90-100% rating from me because it does that amazingly well considering the value it represents.
 
Now take the Thinksound Rain 2 which wasn't far off the same price. It's an OK earphone.  But doesn't reach the same heights and has what I consider flaws.  It's roughly in the same price bracket, and I'm consistent in my ranking based on the criteria I set.
 
But go up in price or down in price, and apply the same consistent method of ranking.  God qualities, flaws, but then apply my perception of value.  The important thing is that I do this for every review.  It may be a little different for me because I know exactly what to expect from diminishing returns, I have bought +$1000 audio items (not review samples), and I know my own expectations.
 
Everyone will have their own different criteria, but my point is that they should state as much as possible what those criteria are when ranking, and be consistent.
 
You can't try and apply logic to who you think is the probable audience.  That's you making assumptions, and you not being accurate.  IMO you need to always be consistent with your reasoning, and application of how it appears to you.  You are the reviewer - no-one else.  You try to bring in as much objectivity as you can - but at the end of the day it is a subjective opinion.  The measure of the popularity of a reviewer is how consistent they can be, and how well they can convey that to the reader - so that then they can interpret it applying their own personal bias.

 
With comparisons, I can be consistent - like you said, in the same price bracket, your judgement of value has to stay the same - what are the flaws, pros, etc. and see how they all add up.
 
What I'm trying to get at though it that you can't have the same sense of value for a more expensive product. Like you said, "It may be a little different for me because I know exactly what to expect from diminishing returns."
 
Why is that? Why do you expect (or are tolerant of) this diminishing rate of returns? Shouldn't we drop the rating by a few stars, because of the diminishing rate of returns? After all, you aren't getting the same amount of value for the money.
 
The answer is no. You have to go in expecting the rate of return for that product's price range. Nobody's arguing about being consistent - but if we were truly consistent, then a diminishing rate of returns should automatically disqualify a product from a 5 star rating.
 
"You can't try and apply logic to who you think is the probable audience. That's you making assumptions, and you not being accurate."
 
I state these assumptions in both the intro and the conclusion. I say who I think it would be a good product for, and who it wouldn't be. But there's (regrettably) only one rating you can give, and that I think is for who the type of item is intended for. You can't go in there and say the UERM is a one star product because budget folks would not consider the value to be good - you have to rate it on your own terms, and from a perspective of one who's actually interested in the item. Does it perform well on those terms, or does it fall flat?
 
 

   
Again - I know where you are coming from, but you missed the point.  How can you actually know a headphone if you're knowledge of it is so clouded by what you have been previously listening to. If you are used to a warm headphone, you'll hear a neutral one as bright.  How accurate is that?  Get intimate with the sound signature, get to know it thoroughly.  The when it's time to write it up, start comparing.  Going in cold does no-one any good.  If you're reviewing something within 24-48 hours of hearing it (and this is just my own personal opinion), you're doing both your readers and the manufacturer a huge dis-service.

 
Like I just told Tom, I rotate - exactly because of the problem you're describing now. There's bright headphones, bassy headphones, cold headphones, warm headphones, and different sources to even it all out, so you don't get used to one. I think Alex does the same, though I'm not sure enough I would bet on it.
 
Honestly, I'm not sure why, but I haven't had this phenomenon happen to me - whatever I heard on day one, and took notes of, I heard on day 50, and felt the same way, long before I became a reviewer - unless, of course, I used only that headphone for the entire time, in which my brain clouds the judgement.
 
Just to note: This isn't to "prove" one way or another. I'm just saying how I feel about certain things and why, but that in no way invalidates someone else's opinion here. Just a different take on the same subject.

 

[eldss]

   
Great question - hopefully others will chime in with their thoughts as well - but here is what I've realised (personally) over the last few years - some of it may be helpful.
 

• The biggest challenge was associating what I thought I was hearing with meaningful data that people could relate to.  One of the best things I did was using this chart:
  http://www.independentrecording.net/irn/resources/freqchart/main_display.htm
  I then sat down with music featuring some of those intruments, a graphic equaliser, and started playing with the levels to see what frequencies affected my listening
   
• I looked for as much info on line as I could about audio workshops (Ethan Winer's ones are particularly enlightening) regarding what is audible, what is not, and what is important when it comes to listening and perception of sound.  Some of the videos were huge eye openers.  I would then take that information further - and test myself.
   
• Build your own selection of test tracks - tracks you know really well - that highlight different things you're going to be testing.  I have tracks for detail, staging, imaging, male and female vocals, bass quantity and quality.  If you have a particular track (say on imaging), and you know its a live track, see if there is a youtube link for the master track you're using (has to be the same recording).  I use one for one of Loreena McKennitt's concerts (I have the lossless live recording) but used the youtube link so I know the exact positioning of the various instruments.  That helped me so that I know when I'm listening to my recording - where the positional cues should be.
   
• Know your own limits and make sure you state them.  For example - I can't pick up hiss from IEMs most of the time - so I get my wife or daughter to help.  You can still be thorough even if your own abilities are compromised.  The secret is knowing we aren't all supermen with superhuman hearing 
  
  .  Personally - I try to keep the whole ego thing at bay.  I'm "merely human" and proud of it.
   
• Volume match - don't need to say a lot more about this.  It is one of the most important things in audio.
   
• In my experience, night and day differences generally don't exist or are overstated.

 
Hope some of that helps - I'll try and think of some more, and hopefully some of the other guys chime in as well.

Excellent blog Brooko! have learned a lot from you guys and really enjoyed your approach. Another tool I've found very helpful when learning to equalize and recognize frequencies/eq is an app called "Quiztones." It's available on apple and Android devices and pretty much, it will quiz you by generating tones and you must identify the frequency. It also takes songs from your library, boosts frequencies and you can learn to identify which frequency is being boosted, etc. 

 

[ExpatinJapan]

another great write up. lots of your points ring true for me also.