[Tom22]

  It's not about the money, I legitimately think that after time with any headphone the issues pretty much dissolve, it's something I'm not spiting but agreeing to.
 
As for the "jump", as you call it, that is a serious thing that can trip up reviews. To get around that, I usually rotate the headphones I use between the ones I have, so I'm not used to any one headphone. 

 
 
oh I didn't know you do that.
 
for me anyway if I like something (when I'm not reviewing something at the moment)
 
I would be drawn towards 1 of them (I'm not going to lie and say I give ALL my earphones the same amount of attention and time, there are times I just like to hear, what I wanna hear, musical enjoyment =)
 
which is why i feel my preferred sound signature is a mix of  v shape and balanced
 
we all have our preferences and because of that for example (i reach for my gr07be more often then my hifiman re400) taking other aspects aside.
 
right now my daily partners are the beats solo2 for portable headphones, Dunu Titan 3 (slightly modded) for moderately noisy areas , and my meelectronic a151 with comply t100s (for noiser areas)

 

[audionewbi]

Good read, here is how I normally do my reviews and the reason I use such methods:
 
1 Finding a set of test track:I normally select 9 test track of which 6 are I am familiar with and 3 I have not listened to before. The logic for this is to reduce the bias of knowing what I am about to hear. In another word the music that I am familiar with are used for testing how good is the IEM is in reproducing those details I am familiar with and the philosophy behind unfamiliar music is to see how much detail and information I am about to hear in this new music, they are just random music for the purpose to unfamiliarity basically because I believe once you are familiar with a track you are likely to hear those fine detail that likely you might not be able to do so.
 
2 Finding set of IEM/DAP: What is the point of a review if you are not going to compare it against its competition. What I do is try to compare it against gears of its price category. I also compare it against the best IEM i have in terms of soundstage, bass decay, treble detail and extension and note how the IEM compare against them. 

If I am testing a new DAP I test all the IEM I have in my disposal and note how that particular DAP pairs with that IEM.

 
2.3 Method: I will perform a direct A/B which has been widely use and supported in loud-speaker research. What this basically is I listen to product A and immediately switch to product B and note the changes I hear. If this is an IEM comparison I keep my source constant. If I am comparing DAP I select a set of IEM that I know intimately and compare how their performance change with different DAP using A/B method.

 
2.4 Terminology:To reduce any confusion the audio terminology I normally be referring to the terms used on stereophile website prepared by J. Gordon Holt. This is perhaps the hardest part as I find a huge inconsistency in how we all use particular terms, I myself included. 
 
I also find short indoor use is not a fair way to test the gear. I must take the item outdoor, use it in the most normal way possible. I dislike using the gear in its best possible way possible as realistically things aren't always optimal and normally I like to write a short impression on how it performs on daily normal usage. 

 

[Toom]

Alongside these forensically detailed slow'n'low reviews, I'd like to see some speedy, instant impression stuff to balance them out.  Preferably by people who are drunk and who swear a lot.

 

[audionewbi]

  Alongside these forensically detailed slow'n'low reviews, I'd like to see some speedy, instant impression stuff to balance them out.  Preferably by people who are drunk and who swear a lot.

I'll do that but mostly on private facebook audio pages, there are half a dozen IEM that I honestly like to strangle their maker for ripping us off. 

 

[earfonia]

Good one Paul! Thanks for starting this interesting discussion!
 
Read your first post, and I would say we mostly have similar approach. But I usually take much longer time period due to other commitments, family, job, church.
 
I agree mostly that our brain adapt, so rush comparison is not accurate. As you said, we must position ourself as buyer or user of the product, that will experience much longer time with the product, longer than what reviewers might spend their time for the product. That way we are not only knowing the product, but will have real day to day experience with it.
 
And one thing that really important for me in reviewing a product are these 2 questions:
1. Do I really enjoy it?
2. Would I spend my money for it if I need it?
 
Those 2 questions are subjective questions, but I think important to make honest review. And really important for me to finalize the overall rating. Anything below 4 stars means I don't really enjoy them, and most probably won't spend my money for it.