Chapter 3 - Anatomy of a Review – The Timeline
Feb 20, 2016 at 4:13 AM Thread Starter Post #1 of 35
Joined
Jan 30, 2011
Posts
13,315
Likes
24,353
Sorry this one has been a bit later in posting.  It’s in our busy season at work at the moment, so I might have been a bit ambitious about expecting to post once a week.  So I might be a bit slow over the next couple of months – but I promise we’ll get there.
 
My original aim for this section was going to be the actual content of the review, why I write in the format I do, and the sort of preparation I do. It would be too much to cover in one sitting, so we’ll look at preparation and timing now. The next instalment we’ll have a look at the individual sections and why I include them.
 
Some of this might be controversial, it might not be what you expect, and it may even challenge other reviewers.  I make no apologies for this – it is purely my opinion, but one based on me being both a consumer and a reviewer.  I’ll explain my reasons – if you don’t agree, all I ask is we have a friendly discussion, but we both clearly state the reasons for any disagreements in style.  That way we can see the pros and cons of both sides – and maybe there is some merit in changing what I currently do as well (I am always still learning).
 
On Arrival
I’ve received a courier pack with a new set of IEMs / headphones or other audio gear.  So what is the first order of business?
 
If I have the time, I’ll immediately photograph the unboxing while they are in a pristine condition.  This isn’t essential but, it can help down the track – especially because no matter how I try, I can never pack things like cables as well as they do out of the factory.  If I don’t get the time to photograph – I’ll usually just have a really quick listen to get an early impression.
 
After my quick listen, I’ll usually email the manufacturer confirming arrival, and pop into the appropriate thread on Head-Fi, and leave an initial impression post.  At that stage – depending on my workload – they usually go back in their box until I’ve finished my current review (whatever I’m working on at the time).
 
One thing I try not to do is run concurrent evaluation listening of new gear at the same time (if it is the same type).  I know a lot do, and it is quite compelling to try it – especially when the reviews start piling up, and it looks as though there is no clear sky on the horizon under the workload you have in front of you.  One of my terrible habits with reviewing is not saying no.  And it isn’t greed or anything else (remember the gear is NOT mine), but more that I like to hear new stuff and when I’m not overworked it is a genuinely pleasurable release (the writing).  The big issue is that when the reviews pile up – there is a constant background pressure to catch up.  That is not fun.
 
Getting Started
Anyway – I’ll often listen to a few items in my free time – but never critically, and never when I get into review mode.  When I get to the stage that I’m ready to start “the process”, I’ll then switch to the item I’m reviewing, and use it, and solely it for the next 2-3 days minimum.  The reason I do this is simple – I want to know the signature of the item I’m reviewing, I want to know it in and out, and I want to know it as if I was an owner, and it was my sole product.  The problem I see with some reviews, if a review is rushed, or if they are constantly changing and comparing – is that they’ve never had the chance to adapt fully to the new signature.
 
This is hugely important in the overall review.  I’ll give an answer I often used (funnily enough when people talk about burn-in).  At one stage I had a Grado 325i along with my HD600.  Both are excellent headphones.  But spend too little time with each, and your opinion will be formed based on what you are used to.  And that is sometimes not really that objective.  If you start with the Grado and get used to it – it is bright, lively, punchy, energetic, and gorgeous with vocals and strings.  Now switch to the HD600 – and first impressions are that they are slow, veiled, bloated – but with a really big soundstage.  Get used to them and they become very natural sounding with brilliant tonality and timbre, the speed becomes normal as does the detail.  The bass sounds natural, and the soundstage returns to a more normal size.  Now switch to the Grado, and suddenly they are screechingly bright, narrow, artificial sounding, bass light, and overly fast.  Has either headphone changed?  No – but our brain has adapted to them and learnt how to cope with them.  The more practise it has, the easier they slip into familiarity, and the easier we adapt.
 
So it’s natural to want to make immediate comparisons when we get new gear, but it’s also the absolutely worst thing we can do.  And why – simply because the newest gear we have is also the gear we know the least, and yet the impressions we give are for prospective buyers.  As a reviewer, cutting corners is something I simply am not allowed to do.
 
Warning – controversial bit here
wink.gif
.  A real example of this was the DUNU Titan T3 & T5 review samples. A lot of reviewers got them together and compared them side-by-side.  I got both together, but I reviewed them separately.  And the reviews I write – while they had extremely similar comments on packaging and build (they are almost identical after all), for the sound and everything else, I treated them as completely different review samples.  The reason I did this is that very few people will buy both.  Also DUNU sent me both expecting review for each one. Simply spending time with both, writing one review, and changing the parts around is (IMO) not doing DUNU any service, not doing my readers any service, and encourages me to spend more time with the one I like rather than look at each one solely on its own merits.  Yes – I compared all of the Titan range in my review – but I actually reviewed them separately – weeks apart in fact.  That allowed each to shine on its own.
 
So the first 2-3 days of active reviewing (and depending on the gear and how tight my schedule is, I’ll extend it out to 3-5 days), is actually just general use.  No notes.  No measuring. No comparisons.  Just me getting thoroughly used to the product.
 
Getting To the Serious Stage
Once I think I have enough of a feel for the product, I’ll next get into critical listening.  That is usually another 1-2 days but can stretch longer.  During this period I’m usually taking copious notes, but more importantly using well known test tracks, and using the item with well-known gear, and looking at specific areas of strength and weakness,
 
Once I’ve completed this, I next move to comparisons, and this is pretty much when the test gear comes out, and I’m now looking at comparing it with gear that to me is my reference.  This then allows me to critically look at the sonic aspects of the review sample.  At this stage I’ll also choose appropriate gear which is similar (price range and features) to contrast against.  I’ll volume match, compare and take more notes.
 
Around this time I usually do my measuring, and then compare it to what I’ve written, and make sure the measurements gel with what I’ve been hearing.  I need to stress at this stage that the measurements are there to support the review – not the review there to support the measurements.  I’ve made mistakes in the past with this (a topic for a future blog post), so now I tend to measure after I’ve already made up my mind on what I’m hearing.
 
Now – can I change my mind after seeing the measurements?  Well actually yes – and usually when the measurements don’t gel with what I think I’ve heard, its good ground to retest, and also a good chance to include some commentary in the review about the changes.  The one thing we have to be is honest – to our readers, and to ourselves. But basing a review solely on measurements or solely on what we hear is IMO a recipe for a possibly pretty inaccurate review.  I’ll usually go back to the beginning when this happens, and look for the reasons why I haven’t picked something up.  The most important thing though is being honest (about my own deficiencies) and being as accurate as I can be.
 
Pictures Can Really Tell The Story
At this point I’m on the home stretch and it’s time for photography.  When I’m taking the shots, you’ll note that I have a few glamour shots, but a lot more of what I would call “clinical” or ”informational”. If I’m a prospective buyer – I don’t want to just see the gorgeous shots against a backdrop.  I want to see the product from all angles.  I want to know what I can expect from the point of view of fit and build.  I want to know which tips definitely work.  I want to see genuine shots of a GUI in action.  The other thing I don’t do is get the review to match the photos.  The photos are instead there to support the review.
 
And then the final steps are the editing, spell checking, uploading, and re-editing before I’m finally happy to push “submit”.
 
Some Common Questions I'm Asked
So how long does a typical review take from start to finish? For me it is usually 7-10 days.  It can be longer depending on my work load or other commitments.  This isn’t usually a bad thing as it gives me even more time to listen to the thing I’m reviewing.
 
OK – well how many reviews can you have in progress at any one time?  That gets tricky.  I like to keep it to a minimum – but I’d be lying if I didn’t say that I’ve had 4 in various stages of being written at any one time.  The important thing though is that I never have two IEMs, or two sources, or two amps etc. on the go at once.  That is when it gets confusing.  Usually when I have multiple reviews in progress it’s because I’ve been sent something with a limited time to get a review up – and I’ve had to stop one to start another one.
 
The other thing worth mentioning is that I'd never try to write a review of a new IEM and combine it with a new source and/or new amp. The key is consistency.  For me that means testing headphones and IEMs mainly with iDSD, or X3ii / E17K combo - although with the time I've now spent with X7, that is likely to join the "regular test gear" stable now.
 
I think at this point I’ll stop for now and look at the various sections in the next post.
 
Your Turn - Lets Discuss
So – if you’re reviewing – how do you do it? If you’re simply reading this blog post, is there anything that concerns you in what I’ve written? Anything you agree with?  Anything you disagree with? As always, I’d love to hear your feedback!
 
Feb 23, 2016 at 6:04 PM Post #2 of 35
I can't believe it, Paul, you posted this 3 days ago and I completely missed it!
 
There are definitely a number of steps I can relate to and some that I do differently.  There is no right or wrong way of doing it, unless if you receive a review samples and have a full review ready the same or the next day :)  My review process used to be a little more chaotic, but I got it recently under control with one review a week.  I know that taking actual pictures when you doing the unboxing guarantees everything in a perfect arrangement as intended by manufacturer, but sometime I want to learn more about the product, about its details to capture it in my picture shots.  Thus, instead of taking one set of purely unboxing pictures and later design detail pictures, I try to take it later at once under the same lighting condition.  Unfortunately, I don't use lightbox or have a "lab" corner, my reviews are "cooked" in the kitchen and only lately I got a light source to eliminate a glare of our recessed sealing lights.  Plus, sometime I do a "photo session" by photographing 3-4 products at a time, then go through pictures to select whichever I want to keep, resize them, rename them, upload to photobucket, and copy into a separate file all the URL links.  When I'm writing the review in Word, I copy the URL links, and when everything is copied on-line to publish the review, I cut'n'paste every link to insert as a picture.
 
Once I have a file with links to all the pictures, I use that file to take my notes about the product, keeping it all organized in one spot, including description of the design, fitment, experience using accessories, sound analysis, and comparison to other headphones in the same category.  I do spend a few days listening on'n'off to headphones exclusively, especially once I'm formatting, renaming, uploading, and getting picture links, or at work during the lunch break.  But when I'm ready to analyze the sound, I always have to compare it to multiple pairs of other headphones I'm intimately familiar with.  I trust my brain more when doing multiple a/b comparisons.  BUT, the key here is to choose a group of headphones with a similar sound signature.  I absolutely agree, it makes no sense to compare warm/smooth vs analytical - this contrast is good to note, but can seriously skew your sound analysis.  I'm also starting to make it a habit to capture FR with Luke's Vibro Veritas after I'm done with my sound evaluation to compare "notes".
 
When all the notes are captured, sometime I start working on the review right away, and sometime I set it aside to capture notes for another pair of headphones.  But when I'm ready to start working on the write up, I always give it more listening time and go back and review my sound analysis notes.  Writing for me takes awhile, not because I don't know what to write, but because I go back'n'forth checking and correcting grammar, etc (after all, even so I spent half of my life in US, English is not my first language).  Once review is published, it's a sigh of relief... for 5min until I update my Review Index on Head-fi with a link and then go to update the list of upcoming reviews to realize that I have more work to do lol!!!
 
Feb 23, 2016 at 6:23 PM Post #3 of 35
Great post Alex - and thanks for the insight into your process.  Would love to see your new photography set-up at some stage too.
 
Sounds like you and I follow a similar pattern.
 
Feb 23, 2016 at 10:06 PM Post #4 of 35
I'm just curious why a review takes so many days... surely it shouldn't take a week to discover the strengths and flaws? Perhaps this would make more sense to me if you were trying a lot of gear combinations, but this doesn't really seem to be the case?
 
Feb 23, 2016 at 11:13 PM Post #5 of 35
Simply because I know (through experience) that many times my first impressions are quite different to something a little more long term,  Call it brain burn-in, or the honeymoon period, or whatever you'd like to, but generally speaking it takes me 3-5 days of casual use to really discover the ins and outs of audio gear, and sometimes even longer.  Maybe I'm just slow, but if people are going to be basing buying decisions off something I;m writing, then I want to be as thorough, well researched, and sure of things as I can be.
 
Virtually every time I've rushed a review, I've had to change something later.
 
YMMV - but personally I'd ignore any review from someone writing it up within 0-3 days of getting it.  And I have seen the odd person doing this within 24 hours of getting an item.  Just the way I look at things, others may be different.
 
Even when I'm setting up critical listening (and comparisons), it can take several hours to set-up - including constant volume matching and rechecking.  Measuring for something simple can sometimes take 2-3 hours.
 
And I do all of this in my spare time - so that's where the days can mount up.  Not so bad I guess if you are a student, or don't have a lot of other community commitments as well.
 
Feb 23, 2016 at 11:32 PM Post #6 of 35
Interesting, though I can see the logic. Brain burn-in makes a lot of sense, I know I've experienced it. So many comparisons I've had the opportunity to do were done in a shorter time frame, where there wasn't time for anything to burn-in. At the time of brain burn-in, I think you can compare things better, but the main can of the review will start to sound too neutral IME. It seems you have protections against that in your plan though... thanks for the reply. That certainly gives me a new way to approach gear. I usually think 2 days is the minimum, though much past that and I start to fatigue.
 
Feb 23, 2016 at 11:33 PM Post #7 of 35
 
Your Turn - Lets Discuss
So – if you’re reviewing – how do you do it? If you’re simply reading this blog post, is there anything that concerns you in what I’ve written? Anything you agree with?  Anything you disagree with? As always, I’d love to hear your feedback!

 
Fantastic Paul!
 
its great you brought this up for discussion!
the following is going to be like a long winded passage so be warned!
 
I myself believe when i reviewing as you said "consistency is key". When i get something new to listen to , whether its a review unit or not. I approach it the same way. Like yourself i try to take pictures of it if possible because my handiwork when coiling them up and such is never going to be a nice or presentable as if it was when it came stock!
 
Because i like to do unboxing videos too, i find it fun to unboxing it and let the viewer join in on the ride... so to speak...
 
but when listening i do like to hook em up to my ipod touch or my lg optimus g pro (which i always do my reviews with) and play some of my favorite and familiar songs. to get a good baseline of what i'm hearing.
 
 
once i listen through (no specific time frame here, on my initial listening), i put em away and pull out some of my good ole reliable gears that i think remind me of what i just listen to.
 
listen to my old and familiar ones (usually 1-3 at max),  then write some notes (either electronically or the old fashion, pen and paper)
 
and then set em aside on the edge of my table. and come back to em later  
 
then of course i have other responsibilities and stuff to take care of and get back to the "newly arrived item" some time after. Refreshing my mind or ears so to speak.
 
and give em at least a week or so of listening.. to hear whats lurking underneath, cause as we all know initial listening isn't always accurate
 
then sportatically throughout the week i would pop in those "good ole reliables" to see if they are similar to the "new thing". and narrow them down to 1 to 2 max
 
afterwards i give them another week alongside, " my good ole reliables" and base my findings from there, and write whenever i have some free time to sit down to listen more extensively... because at this point i gave myself some good solid time ALONE with them as well as with similar sounding ones that i know well.
 
sadly this is why my reviews are often later then some of the other head-fiers. cause unfortunately the last thing i want is to pop out a review, that i look at  weeks/ months later... going like....hm.... what was i doing.... 
 
and also because i like to do video reviews as well, i tend to record closer to the posting date... there by giving myself some time to solidify my findings once more, and make changes according to the written form as well..
 
and then 1 month flies by.... and here we are! Posting date!
 
i also try to stay away from the early reviews if possible at least until i get something down on pen and paper, to see what my eyes, ears and hands and brain thinks before looking at other's opinions.
 
My method is far from scientific, but i like to stay as consistent as possible, cause i find  comparing two things your not very familiar with is the same as writing and posting the review moments after actually getting it from the mail.
 
probably not the most efficient method, as i tend to fall behind. but i like the challenge!
 
Feb 23, 2016 at 11:46 PM Post #8 of 35
  I'm just curious why a review takes so many days... surely it shouldn't take a week to discover the strengths and flaws? Perhaps this would make more sense to me if you were trying a lot of gear combinations, but this doesn't really seem to be the case?

i feel that the honeymoon period is quite a powerful thing... as i think thats what a lot of people get tied up in, hence the sky hype and fall that many products in some of the forums here suffer from...
 
i like to give myself some time to sit back and take it in so to speak....
 
letting the "novelty" aspect fade away and evaluate it based on its own merits. 
 
i like playing with new toys as much as the next person, and i know from past experience (not just headphone related) that i'm extremely impressed with something  initially and willing to overlook all flaws,  (like say a new shirt from h&m)... only to come back to later " say weeks or after a few washes.... and find out how disappointed at the quality of materials used, and how that impacts the overall aesthetics of the clothing, thus impacting my initial enthusiasm
  Simply because I know (through experience) that many times my first impressions are quite different to something a little more long term,  Call it brain burn-in, or the honeymoon period, or whatever you'd like to, but generally speaking it takes me 3-5 days of casual use to really discover the ins and outs of audio gear, and sometimes even longer.  Maybe I'm just slow, but if people are going to be basing buying decisions off something I;m writing, then I want to be as thorough, well researched, and sure of things as I can be.
 
Virtually every time I've rushed a review, I've had to change something later.
 
YMMV - but personally I'd ignore any review from someone writing it up within 0-3 days of getting it.  And I have seen the odd person doing this within 24 hours of getting an item.  Just the way I look at things, others may be different.
 
Even when I'm setting up critical listening (and comparisons), it can take several hours to set-up - including constant volume matching and rechecking.  Measuring for something simple can sometimes take 2-3 hours.
 
And I do all of this in my spare time - so that's where the days can mount up.  Not so bad I guess if you are a student, or don't have a lot of other community commitments as well.

ironically i feel the same way while doing my write up to your earlier post!
 
i'm not sure how they do it! i would think they get it much earlier then i do, considering i'm from the great north (Canada) and everything gets here ridiculously slow....sigh./... to be a canadian...
 
 
and NO!  even students can get busy! especially when their doing two part time jobs and going to school part time as well! =) hence my tendency to fall behind.....
 
the days can certainly add up! 
 
Feb 24, 2016 at 12:14 AM Post #9 of 35
Thanks Tom
 
Great insights :)
 
And sorry - I just looked back on the "student" ref and realised how bad that sounded.  It wasn't intended like that.  I guess I was just trying to convey that there was a difference between the amount of spare time I had around 30 years ago, and the time I have available now.  Probably doesn't help that we're at our busiest time period at work.  My last 7 days looked like this )and work during the day is usually around 7am to 6pm):
 
Monday night - Board meeting school
Tuesday night - free (worked for 3 hours though - catch up)
Wednesday night - presentation to local businesses (work)
Thursday night - Sacramental program (Community/Parish)
Friday night - free!  Did some listening and some writing
Saturday - mix of around home + some audio stuff
Sunday - 1/2 the day Church and Community, other half family and a bit of audio
 
And so far this week it's been meeting Monday, writing Tuesday and I now have another meeting tonight and tomorrow.
 
I'm looking forward to the time things slow down a bit :)
 
Feb 24, 2016 at 12:21 AM Post #10 of 35
Thanks Tom

Great insights :)

And sorry - I just looked back on the "student" ref and realised how bad that sounded.  It wasn't intended like that.  I guess I was just trying to convey that there was a difference between the amount of spare time I had around 30 years ago, and the time I have available now.  Probably doesn't help that we're at our busiest time period at work.  My last 7 days looked like this )and work during the day is usually around 7am to 6pm):

Monday night - Board meeting school
Tuesday night - free (worked for 3 hours though - catch up)
Wednesday night - presentation to local businesses (work)
Thursday night - Sacramental program (Community/Parish)
Friday night - free!  Did some listening and some writing
Saturday - mix of around home + some audio stuff
Sunday - 1/2 the day Church and Community, other half family and a bit of audio

And so far this week it's been meeting Monday, writing Tuesday and I now have another meeting tonight and tomorrow.

I'm looking forward to the time things slow down a bit :)



haha no worries only kidding!

you look a lot more organized then i am, outlining your week ahead.

i just know what i need to accomplish in the coming week or 2, and what days i have work and school, readings and such.


such is life i suppose. Age difference or not, time is definitely short =(.

24 hrs/day sometimes isnt enough
 
Feb 24, 2016 at 2:11 AM Post #11 of 35
Excellent meta thread, OP. I showed your modus operandi to the team of monkeys I have working shifts on my forum posts and they threw their excrement at me.

This is a positive thing, believe me.
 
Feb 24, 2016 at 2:55 AM Post #12 of 35
I remember the headfi days before brooko and twister 6. I remember the days of joker and glarlsagan and dsnuts reviewing things. I dont recall them starting pretentious threads about their own method of review. Get over yourself there, guy.

How much elitism does one guy need? You must work in the private sector....
 
Feb 24, 2016 at 4:10 AM Post #13 of 35
Quick message to any readers of the thread - please don't bother about clicking the report button on the post above me. Part of operating a blog is that I'm an actual moderator - so every time the report button gets hit, I get the notification.
 
Mr ThickT is entitled to his opinion, just as anyone else is.
 
I'm choosing to ignore it with all the hidden disdain it deserves.  Suggest you all do likewise.
 
Feb 24, 2016 at 7:36 AM Post #14 of 35
Good write-up as always, Paul. 
beerchug.gif

 
I see that there are many things where we have similarities, but also others where we slightly differ - especially in terms of time, as I usually take at least two weeks of listening to the gear (every day) and just then start to write the review on the computer, as imho that is required to really get used to the "new stuff" and be sure nothing is left out.
 
If you don't mind, I'd also like to share a brief summary of my process of reviewing (headphones, as I have a different way to evaluate source devices):
Upon arrival, I take the pictures and also take notes about the unboxing experience, build quality and things I like/dislike. All of the notes (also the ones about sound, but more about that later on) are usually written down on paper, I'm somewhat old-fashioned in that regard.
Then the casual listening starts, where I listen to the headphone every day to about anything, but without really comparing them yet nor doing the critical stuff - that is the "getting used to the headphones/brain-in" period. After the first few days, I write down some initial impressions, just to see whether they change over time (and they sometimes do - in certain cases, I was like "wow, they're good!" in the first few hours/days, but after a certain time I thought "hmm, there are things that are not that good though"; sometimes it is also the other way 'round when I initially think "the headphones are okay and good, but not outstanding", but it switches to "oh, they don't do anything wrong and are actually really good!"; sometimes my opinion on the headphone also stays identical from the first to the last day before I write the review).
When I feel familiar with the headphone (usually between the first and second week, but sometimes more), I start listening to sine sweeps (that's nerdy, isn't it?), take down my notes, measure the headphones (and see whether the result in the Veritas coupler matches my impression or not - if it does, I also sometimes include it in my reviews), use an EQ to find out by how much certain frequency areas are boosted (in comparison to "flat" stuff like the Etymotic ER-4S), take down the notes and so on.
Then I start cross-comparing with many other headphones and also decide if I want to include comparisons to some of them that I find matching and also decide which of them I include for comparisons. This usually takes a few more days, but the critical phase hasn't started yet.
When I have a rough overview with comparisons to other stuff, the critical listening phase starts where I use a wide variety of tracks I'm familiar with, reaching through many different genres, complexity levels, speeds, mastering qualities, density and spatiality. This includes many notes. So, the critical listening phase is over, isn't it? No, it isn't, as now I do the same thing in comparison with the other headphones I found appropriate for a comparison, including critical cross-comparisons between the to be reviewed and "competitor's" headphone with the same tracks.
The minimum of that all is two weeks, but it takes usually between three weeks and three months until a review is finished (not including proof-reading, editing and selecting the photos, checking whether I have written down everything I wanted and so on - that takes one or two additional days). Before I don't feel confident about knowing the headphone in and out, I don't even start critical listening.

The most important thing for a review is time - a whole lotta time (and love [#this-was-a-Led-Zeppelin-reference])! And everybody who has ever written a more in-depth review will agree that you need to take a lot of time to do that and that it also takes a lot of enthusiasm for what you do and you need to love what you do - because unless you do it professionally as an editor of a magazine as your main job, a review doesn't make real sense for the writer from an objective point of view (it won't create you income but just takes up your time that can be spent on doing other things), nonetheless if you love writing, love headphones, love music and have at least basic experience, you will do the reviews because you love what you do and want to share your impressions and thoughts with people that have the same hobby as you - at least that's why I'm reviewing audio products.
And I am quite lucky that I can do a good amount of the basic casual listening and taking notes alongside at the office, though most of the "review magic" happens in the evenings and on weekends.
 
Feb 24, 2016 at 9:37 AM Post #15 of 35
Thanks Brooko! All I really need is hands on experience--though I find that sometimes sound gets jumbled in my head and I can't clearly point out different points in audio. Any suggestions concerning this? :)
 

Users who are viewing this thread

Back
Top