- Joined
- Jan 30, 2011
- Posts
- 13,315
- Likes
- 24,362
Sorry this one has been a bit later in posting. It’s in our busy season at work at the moment, so I might have been a bit ambitious about expecting to post once a week. So I might be a bit slow over the next couple of months – but I promise we’ll get there.
My original aim for this section was going to be the actual content of the review, why I write in the format I do, and the sort of preparation I do. It would be too much to cover in one sitting, so we’ll look at preparation and timing now. The next instalment we’ll have a look at the individual sections and why I include them.
Some of this might be controversial, it might not be what you expect, and it may even challenge other reviewers. I make no apologies for this – it is purely my opinion, but one based on me being both a consumer and a reviewer. I’ll explain my reasons – if you don’t agree, all I ask is we have a friendly discussion, but we both clearly state the reasons for any disagreements in style. That way we can see the pros and cons of both sides – and maybe there is some merit in changing what I currently do as well (I am always still learning).
On Arrival
I’ve received a courier pack with a new set of IEMs / headphones or other audio gear. So what is the first order of business?
If I have the time, I’ll immediately photograph the unboxing while they are in a pristine condition. This isn’t essential but, it can help down the track – especially because no matter how I try, I can never pack things like cables as well as they do out of the factory. If I don’t get the time to photograph – I’ll usually just have a really quick listen to get an early impression.
After my quick listen, I’ll usually email the manufacturer confirming arrival, and pop into the appropriate thread on Head-Fi, and leave an initial impression post. At that stage – depending on my workload – they usually go back in their box until I’ve finished my current review (whatever I’m working on at the time).
One thing I try not to do is run concurrent evaluation listening of new gear at the same time (if it is the same type). I know a lot do, and it is quite compelling to try it – especially when the reviews start piling up, and it looks as though there is no clear sky on the horizon under the workload you have in front of you. One of my terrible habits with reviewing is not saying no. And it isn’t greed or anything else (remember the gear is NOT mine), but more that I like to hear new stuff and when I’m not overworked it is a genuinely pleasurable release (the writing). The big issue is that when the reviews pile up – there is a constant background pressure to catch up. That is not fun.
Getting Started
Anyway – I’ll often listen to a few items in my free time – but never critically, and never when I get into review mode. When I get to the stage that I’m ready to start “the process”, I’ll then switch to the item I’m reviewing, and use it, and solely it for the next 2-3 days minimum. The reason I do this is simple – I want to know the signature of the item I’m reviewing, I want to know it in and out, and I want to know it as if I was an owner, and it was my sole product. The problem I see with some reviews, if a review is rushed, or if they are constantly changing and comparing – is that they’ve never had the chance to adapt fully to the new signature.
This is hugely important in the overall review. I’ll give an answer I often used (funnily enough when people talk about burn-in). At one stage I had a Grado 325i along with my HD600. Both are excellent headphones. But spend too little time with each, and your opinion will be formed based on what you are used to. And that is sometimes not really that objective. If you start with the Grado and get used to it – it is bright, lively, punchy, energetic, and gorgeous with vocals and strings. Now switch to the HD600 – and first impressions are that they are slow, veiled, bloated – but with a really big soundstage. Get used to them and they become very natural sounding with brilliant tonality and timbre, the speed becomes normal as does the detail. The bass sounds natural, and the soundstage returns to a more normal size. Now switch to the Grado, and suddenly they are screechingly bright, narrow, artificial sounding, bass light, and overly fast. Has either headphone changed? No – but our brain has adapted to them and learnt how to cope with them. The more practise it has, the easier they slip into familiarity, and the easier we adapt.
So it’s natural to want to make immediate comparisons when we get new gear, but it’s also the absolutely worst thing we can do. And why – simply because the newest gear we have is also the gear we know the least, and yet the impressions we give are for prospective buyers. As a reviewer, cutting corners is something I simply am not allowed to do.
Warning – controversial bit here
. A real example of this was the DUNU Titan T3 & T5 review samples. A lot of reviewers got them together and compared them side-by-side. I got both together, but I reviewed them separately. And the reviews I write – while they had extremely similar comments on packaging and build (they are almost identical after all), for the sound and everything else, I treated them as completely different review samples. The reason I did this is that very few people will buy both. Also DUNU sent me both expecting review for each one. Simply spending time with both, writing one review, and changing the parts around is (IMO) not doing DUNU any service, not doing my readers any service, and encourages me to spend more time with the one I like rather than look at each one solely on its own merits. Yes – I compared all of the Titan range in my review – but I actually reviewed them separately – weeks apart in fact. That allowed each to shine on its own.
So the first 2-3 days of active reviewing (and depending on the gear and how tight my schedule is, I’ll extend it out to 3-5 days), is actually just general use. No notes. No measuring. No comparisons. Just me getting thoroughly used to the product.
Getting To the Serious Stage
Once I think I have enough of a feel for the product, I’ll next get into critical listening. That is usually another 1-2 days but can stretch longer. During this period I’m usually taking copious notes, but more importantly using well known test tracks, and using the item with well-known gear, and looking at specific areas of strength and weakness,
Once I’ve completed this, I next move to comparisons, and this is pretty much when the test gear comes out, and I’m now looking at comparing it with gear that to me is my reference. This then allows me to critically look at the sonic aspects of the review sample. At this stage I’ll also choose appropriate gear which is similar (price range and features) to contrast against. I’ll volume match, compare and take more notes.
Around this time I usually do my measuring, and then compare it to what I’ve written, and make sure the measurements gel with what I’ve been hearing. I need to stress at this stage that the measurements are there to support the review – not the review there to support the measurements. I’ve made mistakes in the past with this (a topic for a future blog post), so now I tend to measure after I’ve already made up my mind on what I’m hearing.
Now – can I change my mind after seeing the measurements? Well actually yes – and usually when the measurements don’t gel with what I think I’ve heard, its good ground to retest, and also a good chance to include some commentary in the review about the changes. The one thing we have to be is honest – to our readers, and to ourselves. But basing a review solely on measurements or solely on what we hear is IMO a recipe for a possibly pretty inaccurate review. I’ll usually go back to the beginning when this happens, and look for the reasons why I haven’t picked something up. The most important thing though is being honest (about my own deficiencies) and being as accurate as I can be.
Pictures Can Really Tell The Story
At this point I’m on the home stretch and it’s time for photography. When I’m taking the shots, you’ll note that I have a few glamour shots, but a lot more of what I would call “clinical” or ”informational”. If I’m a prospective buyer – I don’t want to just see the gorgeous shots against a backdrop. I want to see the product from all angles. I want to know what I can expect from the point of view of fit and build. I want to know which tips definitely work. I want to see genuine shots of a GUI in action. The other thing I don’t do is get the review to match the photos. The photos are instead there to support the review.
And then the final steps are the editing, spell checking, uploading, and re-editing before I’m finally happy to push “submit”.
Some Common Questions I'm Asked
So how long does a typical review take from start to finish? For me it is usually 7-10 days. It can be longer depending on my work load or other commitments. This isn’t usually a bad thing as it gives me even more time to listen to the thing I’m reviewing.
OK – well how many reviews can you have in progress at any one time? That gets tricky. I like to keep it to a minimum – but I’d be lying if I didn’t say that I’ve had 4 in various stages of being written at any one time. The important thing though is that I never have two IEMs, or two sources, or two amps etc. on the go at once. That is when it gets confusing. Usually when I have multiple reviews in progress it’s because I’ve been sent something with a limited time to get a review up – and I’ve had to stop one to start another one.
The other thing worth mentioning is that I'd never try to write a review of a new IEM and combine it with a new source and/or new amp. The key is consistency. For me that means testing headphones and IEMs mainly with iDSD, or X3ii / E17K combo - although with the time I've now spent with X7, that is likely to join the "regular test gear" stable now.
I think at this point I’ll stop for now and look at the various sections in the next post.
Your Turn - Lets Discuss
So – if you’re reviewing – how do you do it? If you’re simply reading this blog post, is there anything that concerns you in what I’ve written? Anything you agree with? Anything you disagree with? As always, I’d love to hear your feedback!
My original aim for this section was going to be the actual content of the review, why I write in the format I do, and the sort of preparation I do. It would be too much to cover in one sitting, so we’ll look at preparation and timing now. The next instalment we’ll have a look at the individual sections and why I include them.
Some of this might be controversial, it might not be what you expect, and it may even challenge other reviewers. I make no apologies for this – it is purely my opinion, but one based on me being both a consumer and a reviewer. I’ll explain my reasons – if you don’t agree, all I ask is we have a friendly discussion, but we both clearly state the reasons for any disagreements in style. That way we can see the pros and cons of both sides – and maybe there is some merit in changing what I currently do as well (I am always still learning).
On Arrival
I’ve received a courier pack with a new set of IEMs / headphones or other audio gear. So what is the first order of business?
If I have the time, I’ll immediately photograph the unboxing while they are in a pristine condition. This isn’t essential but, it can help down the track – especially because no matter how I try, I can never pack things like cables as well as they do out of the factory. If I don’t get the time to photograph – I’ll usually just have a really quick listen to get an early impression.
After my quick listen, I’ll usually email the manufacturer confirming arrival, and pop into the appropriate thread on Head-Fi, and leave an initial impression post. At that stage – depending on my workload – they usually go back in their box until I’ve finished my current review (whatever I’m working on at the time).
One thing I try not to do is run concurrent evaluation listening of new gear at the same time (if it is the same type). I know a lot do, and it is quite compelling to try it – especially when the reviews start piling up, and it looks as though there is no clear sky on the horizon under the workload you have in front of you. One of my terrible habits with reviewing is not saying no. And it isn’t greed or anything else (remember the gear is NOT mine), but more that I like to hear new stuff and when I’m not overworked it is a genuinely pleasurable release (the writing). The big issue is that when the reviews pile up – there is a constant background pressure to catch up. That is not fun.
Getting Started
Anyway – I’ll often listen to a few items in my free time – but never critically, and never when I get into review mode. When I get to the stage that I’m ready to start “the process”, I’ll then switch to the item I’m reviewing, and use it, and solely it for the next 2-3 days minimum. The reason I do this is simple – I want to know the signature of the item I’m reviewing, I want to know it in and out, and I want to know it as if I was an owner, and it was my sole product. The problem I see with some reviews, if a review is rushed, or if they are constantly changing and comparing – is that they’ve never had the chance to adapt fully to the new signature.
This is hugely important in the overall review. I’ll give an answer I often used (funnily enough when people talk about burn-in). At one stage I had a Grado 325i along with my HD600. Both are excellent headphones. But spend too little time with each, and your opinion will be formed based on what you are used to. And that is sometimes not really that objective. If you start with the Grado and get used to it – it is bright, lively, punchy, energetic, and gorgeous with vocals and strings. Now switch to the HD600 – and first impressions are that they are slow, veiled, bloated – but with a really big soundstage. Get used to them and they become very natural sounding with brilliant tonality and timbre, the speed becomes normal as does the detail. The bass sounds natural, and the soundstage returns to a more normal size. Now switch to the Grado, and suddenly they are screechingly bright, narrow, artificial sounding, bass light, and overly fast. Has either headphone changed? No – but our brain has adapted to them and learnt how to cope with them. The more practise it has, the easier they slip into familiarity, and the easier we adapt.
So it’s natural to want to make immediate comparisons when we get new gear, but it’s also the absolutely worst thing we can do. And why – simply because the newest gear we have is also the gear we know the least, and yet the impressions we give are for prospective buyers. As a reviewer, cutting corners is something I simply am not allowed to do.
Warning – controversial bit here

So the first 2-3 days of active reviewing (and depending on the gear and how tight my schedule is, I’ll extend it out to 3-5 days), is actually just general use. No notes. No measuring. No comparisons. Just me getting thoroughly used to the product.
Getting To the Serious Stage
Once I think I have enough of a feel for the product, I’ll next get into critical listening. That is usually another 1-2 days but can stretch longer. During this period I’m usually taking copious notes, but more importantly using well known test tracks, and using the item with well-known gear, and looking at specific areas of strength and weakness,
Once I’ve completed this, I next move to comparisons, and this is pretty much when the test gear comes out, and I’m now looking at comparing it with gear that to me is my reference. This then allows me to critically look at the sonic aspects of the review sample. At this stage I’ll also choose appropriate gear which is similar (price range and features) to contrast against. I’ll volume match, compare and take more notes.
Around this time I usually do my measuring, and then compare it to what I’ve written, and make sure the measurements gel with what I’ve been hearing. I need to stress at this stage that the measurements are there to support the review – not the review there to support the measurements. I’ve made mistakes in the past with this (a topic for a future blog post), so now I tend to measure after I’ve already made up my mind on what I’m hearing.
Now – can I change my mind after seeing the measurements? Well actually yes – and usually when the measurements don’t gel with what I think I’ve heard, its good ground to retest, and also a good chance to include some commentary in the review about the changes. The one thing we have to be is honest – to our readers, and to ourselves. But basing a review solely on measurements or solely on what we hear is IMO a recipe for a possibly pretty inaccurate review. I’ll usually go back to the beginning when this happens, and look for the reasons why I haven’t picked something up. The most important thing though is being honest (about my own deficiencies) and being as accurate as I can be.
Pictures Can Really Tell The Story
At this point I’m on the home stretch and it’s time for photography. When I’m taking the shots, you’ll note that I have a few glamour shots, but a lot more of what I would call “clinical” or ”informational”. If I’m a prospective buyer – I don’t want to just see the gorgeous shots against a backdrop. I want to see the product from all angles. I want to know what I can expect from the point of view of fit and build. I want to know which tips definitely work. I want to see genuine shots of a GUI in action. The other thing I don’t do is get the review to match the photos. The photos are instead there to support the review.
And then the final steps are the editing, spell checking, uploading, and re-editing before I’m finally happy to push “submit”.
Some Common Questions I'm Asked
So how long does a typical review take from start to finish? For me it is usually 7-10 days. It can be longer depending on my work load or other commitments. This isn’t usually a bad thing as it gives me even more time to listen to the thing I’m reviewing.
OK – well how many reviews can you have in progress at any one time? That gets tricky. I like to keep it to a minimum – but I’d be lying if I didn’t say that I’ve had 4 in various stages of being written at any one time. The important thing though is that I never have two IEMs, or two sources, or two amps etc. on the go at once. That is when it gets confusing. Usually when I have multiple reviews in progress it’s because I’ve been sent something with a limited time to get a review up – and I’ve had to stop one to start another one.
The other thing worth mentioning is that I'd never try to write a review of a new IEM and combine it with a new source and/or new amp. The key is consistency. For me that means testing headphones and IEMs mainly with iDSD, or X3ii / E17K combo - although with the time I've now spent with X7, that is likely to join the "regular test gear" stable now.
I think at this point I’ll stop for now and look at the various sections in the next post.
Your Turn - Lets Discuss
So – if you’re reviewing – how do you do it? If you’re simply reading this blog post, is there anything that concerns you in what I’ve written? Anything you agree with? Anything you disagree with? As always, I’d love to hear your feedback!