[Brooko]

So a review whether positive or negative is still publicity, therefore bad products probably shouldn't be covered at all unless the writer strongly averts buyers. And since most reviewers are very time constrained, and I assume this is the case for others too, we tend to gravitate towards models of interest or things that seem to possess some quality from initial impressions while blocking out the junk. Resultantly, only capable items get reviewed and scores look overly positive, but these products are objectively well-performing compared to products uncovered and unmentioned in the review.

I definitely don't have the time to look at all of the products I get sent (not always on request or to my knowledge), so I just feature the good stuff that I think people will value. Any thoughts on this? Of course, expression of thoughts on these products can easily become hype but I think it's a pretty common trend that should be addressed.

And a note on scoring, due to the aforementioned phenomenon, ratings are skewed positive, with the vast majority of products sitting on 4-stars or very negative around 2-stars. Is it fair to raise our standards and give products like the Kinera H3 and Simgot EN700 Bass 2-3 stars because it's just average in its price category, or does it get 4-5 stars because, as a product, it is quite fully-featured?

I can only tell you how I do it - others will have their own methodology.

1. If I receive something I think is seriously flawed, most of the time I will let the manufacturer know, and check to see if it is actually a fault. I've had a lot of samples with mis-matched drivers etc over the years. It happens. Most of the time I simply review the gear, and point out good and bad. I've had excellent gear and some not so good gear from both FiiO and Dunu. To their credit, both companies asked me to carry on the reviews with the gear I had issues with. They have been really good - preferring a review warts and all. Neither company tries to influence me - and both appreciate the feedback.
   
   
2. I review whatever I get sent. And it doesn't matter if its $10 ear-buds from VE or a $1300 DAP from L&P. They all get the same treatment - same care, same measurements.
   
   
3. Scoring can be done anyway as long as you explain how you do it. I take value into account. So when I'm reviewing a $100 IEM, I'm scoring it alongside other $100 IEMs, Some don't - but I find that if I review everything on the same scale, then only the expensive stuff would get the high scores, and any cheap stuff would be getting low scores. Thats not fair. people shop in different brackets. Value is just another feature / scoring point for me. The FiiO F9 scored high despite it's narrow treble peak because it was easy to EQ, had very good fit and build, and for a $100 IEM punched way above its price point. The Kinera H3 scored lower because it had significantly lower value to me. I didn't have to just apply a basic EQ - I had to EQ them extensively to correct their faults. And the treble peak wasn't narrow - it was wide and really unpleasant. That was the bit I struggled with - how other people couldn't notice.
   
   I haven't heard the Simgot EN700 Bass, but I have the Pro, and it is very good. Its coloured (mildly v shaped) but its done really well. Its build is excellent. And for $150 I can't really fault them. Good sound, good build, good value, good fit, good comfort.
   
   At the other end of the scale are IEMs like the RE800. its actually a very close to reference IEM tuning. And if you EQ the treble peak out, iits a really nice IEM to listen to. But I can't score it highly because of the price. IMO a $700 IEM shouldn't have those issues. Hopefully that makes some sense.
   
   Some people do scoring really well IMO - break it down and be very transparent with weightings etc. I just try to give reasons for the way I score.

 

[Dobrescu George]

So a review whether positive or negative is still publicity, therefore bad products probably shouldn't be covered at all unless the writer strongly averts buyers. And since most reviewers are very time constrained, and I assume this is the case for others too, we tend to gravitate towards models of interest or things that seem to possess some quality from initial impressions while blocking out the junk. Resultantly, only capable items get reviewed and scores look overly positive, but these products are objectively well-performing compared to products uncovered and unmentioned in the review.

I definitely don't have the time to look at all of the products I get sent (not always on request or to my knowledge), so I just feature the good stuff that I think people will value. Any thoughts on this? Of course, expression of thoughts on these products can easily become hype but I think it's a pretty common trend that should be addressed.

And a note on scoring, due to the aforementioned phenomenon, ratings are skewed positive, with the vast majority of products sitting on 4-stars or very negative around 2-stars. Is it fair to raise our standards and give products like the Kinera H3 and Simgot EN700 Bass 2-3 stars because it's just average in its price category, or does it get 4-5 stars because, as a product, it is quite fully-featured?

I agree about this somehow. It'd be best not to do a work that is negative, you're spending hours on it, why not try your best to say something nice? Time is not unlimited, neither are people's money. No one cares what to avoid, there are thousands of products, people care what they might like. A page doing negative writing would only be used for fun and giggles, if it is written in a funny way, at least I think... But our actual meaning would be to help the people, not to scare them off this hobby in general.

 

[ryanjsoo]

I can only tell you how I do it - others will have their own methodology.

1. If I receive something I think is seriously flawed, most of the time I will let the manufacturer know, and check to see if it is actually a fault. I've had a lot of samples with mis-matched drivers etc over the years. It happens. Most of the time I simply review the gear, and point out good and bad. I've had excellent gear and some not so good gear from both FiiO and Dunu. To their credit, both companies asked me to carry on the reviews with the gear I had issues with. They have been really good - preferring a review warts and all. Neither company tries to influence me - and both appreciate the feedback.
   
   
2. I review whatever I get sent. And it doesn't matter if its $10 ear-buds from VE or a $1300 DAP from L&P. They all get the same treatment - same care, same measurements.
   
   
3. Scoring can be done anyway as long as you explain how you do it. I take value into account. So when I'm reviewing a $100 IEM, I'm scoring it alongside other $100 IEMs, Some don't - but I find that if I review everything on the same scale, then only the expensive stuff would get the high scores, and any cheap stuff would be getting low scores. Thats not fair. people shop in different brackets. Value is just another feature / scoring point for me. The FiiO F9 scored high despite it's narrow treble peak because it was easy to EQ, had very good fit and build, and for a $100 IEM punched way above its price point. The Kinera H3 scored lower because it had significantly lower value to me. I didn't have to just apply a basic EQ - I had to EQ them extensively to correct their faults. And the treble peak wasn't narrow - it was wide and really unpleasant. That was the bit I struggled with - how other people couldn't notice.
   
   I haven't heard the Simgot EN700 Bass, but I have the Pro, and it is very good. Its coloured (mildly v shaped) but its done really well. Its build is excellent. And for $150 I can't really fault them. Good sound, good build, good value, good fit, good comfort.
   
   At the other end of the scale are IEMs like the RE800. its actually a very close to reference IEM tuning. And if you EQ the treble peak out, iits a really nice IEM to listen to. But I can't score it highly because of the price. IMO a $700 IEM shouldn't have those issues. Hopefully that makes some sense.
   
   Some people do scoring really well IMO - break it down and be very transparent with weightings etc. I just try to give reasons for the way I score.

I think most reviewers score like you, I do, and I'm not sure why you would disregard price in your rating, that's really not fair or constructive. However, what I'm trying to say is that when we start using numbers to get a point across, it's important to have a standard, that 5-star Kinera review could be according to very different priorities and references than the next person. Scores and measurements are important and have a place, no doubt there, but when everyone is scoring on different criteria, weighting and priorities, doesn't that devalue the score?

For example, I don't like to use eQ, so I didn't enjoy the F9 nearly as much as you did and didn't appreciate its potential hence our differing opinions. I guess therein lies the value of multiple reviews, some readers will inevitably align with both of us based on their uses. I also think a score diminishes the personal preferences of the consumer, many have contacted me for recommendation, stuck between a model they know will fit them better and another that I've scored higher due to better overall performance. The Simgot for instance, is a really nice sounding earphone that may very well warrant a 5-star to you, but if the buyer values passive noise isolation, that sound quality goes out the window when ambient noise is high when compared to something more isolating, say the Pinnacle P2. I've kind of struggled to communicate this at times, but it's so so so important.

 

[TheoS53]

I think a universal scoring system is needed to some extent. Yes, we all have our own preferences, but we also need to try and get some sort of standard going.
On the one hand, I really hate how we've become so dependent on specifications, graphs, numbers and other technical details, and I feel a lot of times a product's worth or real world performance takes a back seat to hardware specs. Like a product that has 0.003% THD+N vs one that has 0.0003%. Technically, the latter is 10 times better... But that doesn't actually mean that it sounds at all better to us, never mind 10 times better lol.
I think this is the worth of reviews though.... Honest, unbiased opinions about how a product actually performs in the real world. None of the marketing BS.
But as I said, numbers can be important to compare items. So maybe we should look at creating a review formula for how we score products.

 

[Dobrescu George]

I think most reviewers score like you, I do, and I'm not sure why you would disregard price in your rating, that's really not fair or constructive. However, what I'm trying to say is that when we start using numbers to get a point across, it's important to have a standard, that 5-star Kinera review could be according to very different priorities and references than the next person. Scores and measurements are important and have a place, no doubt there, but when everyone is scoring on different criteria, weighting and priorities, doesn't that devalue the score?

For example, I don't like to use eQ, so I didn't enjoy the F9 nearly as much as you did and didn't appreciate its potential hence our differing opinions. I guess therein lies the value of multiple reviews, some readers will inevitably align with both of us based on their uses. I also think a score diminishes the personal preferences of the consumer, many have contacted me for recommendation, stuck between a model they know will fit them better and another that I've scored higher due to better overall performance. The Simgot for instance, is a really nice sounding earphone that may very well warrant a 5-star to you, but if the buyer values passive noise isolation, that sound quality goes out the window when ambient noise is high when compared to something more isolating, say the Pinnacle P2. I've kind of struggled to communicate this at times, but it's so so so important.

I agree with this.

 

[Brooko]

I agree about this somehow. It'd be best not to do a work that is negative, you're spending hours on it, why not try your best to say something nice? Time is not unlimited, neither are people's money. No one cares what to avoid, there are thousands of products, people care what they might like. A page doing negative writing would only be used for fun and giggles, if it is written in a funny way, at least I think... But our actual meaning would be to help the people, not to scare them off this hobby in general.

Jeepers - I really don't know what to say about this. So what you're saying is rather than being honest about a products shortfalls - we should gloss over them so we can be more positive? You just kind of summed up the whole point of what i wrote in a couple of sentences. If this is where we are headed, then why am I bothering ......

And your 2nd post about fun factor?

I'm going to stop now before I say something I regret. You do realise that people are making buying decisions based on what reviewers write?

 

[Hawaiibadboy]

Jeepers - I really don't know what to say about this. So what you're saying is rather than being honest about a products shortfalls - we should gloss over them so we can be more positive? You just kind of summed up the whole point of what i wrote in a couple of sentences. If this is where we are headed, then why am I bothering ......

And your 2nd post about fun factor?

I'm going to stop now before I say something I regret. You do realise that people are making buying decisions based on what reviewers write?

I have a rant in the type box i will just delete. That freaked me out too.

You folks need to be honest and remove people from positions that do more harm than good. Read into that how you may.

I would be a bad MOD. Half the site would be banned in days.

Other folks should really not be doing reviews. my point is clear and related to what I quote.

 

[Dobrescu George]

Jeepers - I really don't know what to say about this. So what you're saying is rather than being honest about a products shortfalls - we should gloss over them so we can be more positive? You just kind of summed up the whole point of what i wrote in a couple of sentences. If this is where we are headed, then why am I bothering ......

And your 2nd post about fun factor?

I'm going to stop now before I say something I regret. You do realise that people are making buying decisions based on what reviewers write?

We should mention the shortcomings of something that has a few of them, always. My thinking was that if something is seriously flawed, it isn't worth writing about at all.

 

[crinacle]

I agree about this somehow. It'd be best not to do a work that is negative, you're spending hours on it, why not try your best to say something nice? Time is not unlimited, neither are people's money. No one cares what to avoid, there are thousands of products, people care what they might like. A page doing negative writing would only be used for fun and giggles, if it is written in a funny way, at least I think... But our actual meaning would be to help the people, not to scare them off this hobby in general.

And this here is why I have less faith in positive reviews in general, unless I trust the reviewer. A product that sounds bad is bad, no sugarcoating. What you're saying here is to fluff the negatives to make them seem less unattractive... and why? You're not doing it for profit clearly; you're the reviewer, not the seller. So why?

My hopefully-popular-but-seemingly-unpopular opinion: a reviewer's "job" is to inform, not to pamper. You're not supposed to be the one saying good things, you're the one who's supposed to be nitpicking the flaws. Trying to find good things to say is a telltale sign that the product doesn't have many to begin with (read: BAD). Positivity is for fanboys. Critique is for reviewers.

 

[ryanjsoo]

Jeepers - I really don't know what to say about this. So what you're saying is rather than being honest about a products shortfalls - we should gloss over them so we can be more positive? You just kind of summed up the whole point of what i wrote in a couple of sentences. If this is where we are headed, then why am I bothering ......

And your 2nd post about fun factor?

I'm going to stop now before I say something I regret. You do realise that people are making buying decisions based on what reviewers write?

I think George is referring more to personal enjoyment and how he tests with various people with various tastes to achieve a more accurate if not particularly objective measure. As for avoiding negativity... I guess don't disrupt those who are enjoying a previously purchased model but do inform those considering a purchase.

I have a rant in the type box i will just delete. That freaked me out too.

You folks need to be honest and remove people from positions that do more harm than good. Read into that how you may.

I would be a bad MOD. Half the site would be banned in days.

Other folks should really not be doing reviews. my point is clear and related to what I quote.

In that sense, there are no shortage user impressions hyping things up nowadays, I swear the forum wasn't like this when I joined. Just look at the F9 thread, and yes, it is a fine earphone for the price, but people come along and compare it favourably to something like the SE846 then later say, whoops actually compared them and the Shure is WAY better. Like what! If you're going to say something so strongly opinionated at least give it some thought! Reviews are one thing, but this problem is far more pervasive than just that, a lot of little voices can be just as influential as one big one.

 

[Dobrescu George]

And this here is why I have less faith in positive reviews in general, unless I trust the reviewer. A product that sounds bad is bad, no sugarcoating. What you're saying here is to fluff the negatives to make them seem less unattractive... and why? You're not doing it for profit clearly; you're the reviewer, not the seller. So why?

My hopefully-popular-but-seemingly-unpopular opinion: a reviewer's "job" is to inform, not to pamper. You're not supposed to be the one saying good things, you're the one who's supposed to be nitpicking the flaws. Trying to find good things to say is a telltale sign that the product doesn't have many to begin with (read: BAD). Positivity is for fanboys. Critique is for reviewers.

I'm nitpicking when I do write about something, I was more like, Hey, that thing is really flawed. Let's ignore it entirely instead of complain about it. Why? Basic respect for the ones sending it. They wanted your input, and it is strongly negative. At least tell them, send it back, and go to something that has flaws, but that also has positives.

If you read my works, I usually recommend EQ for what has issues, so I never gloss them over...

Also, agreed, we don't make money form this, so there is no reason to be nice.

I think George is referring more to personal enjoyment and how he tests with various people with various tastes to achieve a more objective measure. As for avoiding negativity... I guess don't disrupt those who are enjoying a previously purchased model but do inform those considering a purchase.

In that sense, there are no shortage user impressions hyping things up nowadays, I swear the forum wasn't like this when I joined. Just look at the F9 thread, and yes, it is a fine earphone for the price, but people come along and compare it favourably to something like the SE846 then later say, whoops actually compared them and the Shure is WAY better. Like what! If you're going to say something so strongly opinionated at least give it some thought! Reviews are one thing, but this problem is far more pervasive than just that, a lot of little voices can be just as influential as one big one.

I noticed a strong hype as well. It used to be more neutral back when I joined. Sometime the few moments of happiness I am getting from this hobby are confused with hype and it makes me even sadder...

 

[Brooko]

I think most reviewers score like you, I do, and I'm not sure why you would disregard price in your rating, that's really not fair or constructive. However, what I'm trying to say is that when we start using numbers to get a point across, it's important to have a standard, that 5-star Kinera review could be according to very different priorities and references than the next person. Scores and measurements are important and have a place, no doubt there, but when everyone is scoring on different criteria, weighting and priorities, doesn't that devalue the score?

For example, I don't like to use eQ, so I didn't enjoy the F9 nearly as much as you did and didn't appreciate its potential hence our differing opinions. I guess therein lies the value of multiple reviews, some readers will inevitably align with both of us based on their uses. I also think a score diminishes the personal preferences of the consumer, many have contacted me for recommendation, stuck between a model they know will fit them better and another that I've scored higher due to better overall performance. The Simgot for instance, is a really nice sounding earphone that may very well warrant a 5-star to you, but if the buyer values passive noise isolation, that sound quality goes out the window when ambient noise is high when compared to something more isolating, say the Pinnacle P2. I've kind of struggled to communicate this at times, but it's so so so important.

I think you misunderstood me Ryan. I always take value into account. Some people don't.

So genuine question - you rated the F9 at $100 a 4/5, and the Kinera H3 at $100 a 5/5 - but you don't use EQ. I graphed both - same rig - and you can clearly see the peak is far wider. Unless you got a very different batch (and I now have 3 other graphs which are essentially the same, and 3 other people saying the same thing about them), then something is wrong here. Any chance you get get your pair of H3 to Crinnacle for measuring. I'd be very interested to see if there are significant differences between the pair he measured and the ones you have.

And just so we are clear - I'm used to reasonably bright headphones. I have a pair of HD800S and a pair of MS Pros. I spent a heap of time with Grado 325is, K701, DT800 and T1. All are bright headphones. I have age related hearing loss as well as tinnitus. Brightness doesn't bother me unless its excessive. On the F9 it is a very narrow peak. On the H3 is is a wide band - and its so bad that its almost painful to listen to (music that is normally warm is sibilant). Thats what I want to know - why the difference .....

 

[Dobrescu George]

At the end of the day, I think that my English might be still a bit flawed compared to some of other people. I am noticing that my point is actually misunderstood a bit...

 

[Houba]

What I learned so far is that we can't trust all reviewers for various reasons, such as a misleading information on measurements, specifications and lack of knowledge on the product. Recently I saw a reviewer on Youtube (well known on headfi) who don't even know that Noble iems have a detachable cable, what a shame to trust a guy who is too lazy to get such basic information before uploading his review, what makes me laugh : he asked the audience to post the answer in the comments when he already has the product in hand and supposedly spent enough time with it for a review XD

Where are we going ?

 

[ryanjsoo]

I think you misunderstood me Ryan. I always take value into account. Some people don't.

So genuine question - you rated the F9 at $100 a 4/5, and the Kinera H3 at $100 a 5/5 - but you don't use EQ. I graphed both - same rig - and you can clearly see the peak is far wider. Unless you got a very different batch (and I now have 3 other graphs which are essentially the same, and 3 other people saying the same thing about them), then something is wrong here. Any chance you get get your pair of H3 to Crinnacle for measuring. I'd be very interested to see if there are significant differences between the pair he measured and the ones you have.

And just so we are clear - I'm used to reasonably bright headphones. I have a pair of HD800S and a pair of MS Pros. I spent a heap of time with Grado 325is, K701, DT800 and T1. All are bright headphones. I have age related hearing loss as well as tinnitus. Brightness doesn't bother me unless its excessive. On the F9 it is a very narrow peak. On the H3 is is a wide band - and its so bad that its almost painful to listen to (music that is normally warm is sibilant). Thats what I want to know - why the difference .....

Sorry, wasn't talking about you with the value comment, that was in reference to those that don't take value into account! And regarding the H3 review, my take was a hideous misappropriation of the truth. I've considered taking it down, I don't know if it's even worth re-writing at this point, I fully acknowledge that this was one of my worst reviews, I take it as a learning experience and I don't think I've made the same mistake since...