How Can I Trust Head-Fi Reviews?
Jun 25, 2015 at 9:11 AM Post #121 of 155
  Definitely not getting carried away with the hype, fella as you can tell from my source gear
biggrin.gif

 
Just edited my last post with some review stats on the X3ii, Only 5 of the 33 reviewers actually owned the X3ii at the time of their reviews. The other 28 were promotional loaners.
 
 I'm a firm believer of 'putting your money where your mouth is'. Guess I'll stick to word of mouth on the forums. I'll definitely be ignoring the reviews from now on.
 
Backs out of thread slowly....

 
So your insinuating that all reviewers who don't own the gear can't be trusted because they are biased towards the manufacturer - yet all owners have no bias - even though we know justification bias exists?
 
My advice - find reviewers who own same/similar gear that you have yourself, who describe the gear you both share very similarly, who when reviewing look at both the good and bad points of the gear, and above all, the ones who are prepared to do the basics when reviewing and comparing (ie - list their gear, their known preferences and bias, and volume match when making comparisons). Oh - and avoid anyone who talks about night and day differences, and volume matches by ear 
wink.gif

 
Once you find two or three reviewers who meet your criteria - then you know if they have consensus on a piece of gear, you should be pretty safe looking at it more seriously.  As always though - there is no substitute for listening yourself.
 
Jun 25, 2015 at 9:47 AM Post #122 of 155
   
So your insinuating that all reviewers who don't own the gear can't be trusted because they are biased towards the manufacturer - yet all owners have no bias - even though we know justification bias exists?

 
I'm saying that I personally don't trust reviews where reviewers don't own the gear, yes. I think if you're getting a loaner or a reduced price many reviewers would feel a certain, maybe unconscious, obligation to be grateful and that could influence their reviews. I'm glad you didn't take it as a personal slant but most of the reviews here are not that great, sorry but it's true. The number of 5 star reviews here really isn't that helpful. There's always going to be bias, everyone thinks their gear is the best, that is, unfortunately, human nature nowadays.
 
 
Going back to the X3ii reviews the really interesting thing is that the most damning review came from someone who actually bought it and owned it. (This isn't an attack on the X3ii BTW it's purely an example). There are good reviewers here, that isn't in doubt but they are a very small minority in my honest opinion.
 
Reviews are rarely going to be without bias but under the influence of heavy marketing they often appear as little more than advertisements for the product.
 
I'm firmly staying away from reviews of products, they're (mostly) tainted by the influence of marketing. I'm going to stick to opinions from users who own the product and use it daily. Personally I find it a much better way of gaining opinion on a product. Someone might say it has great bass and then someone says it lacks bass, debate is much better way to gauge opinion than a review. Also reviews, generally are tainted by personal taste, another reason why I prefer the debates on the forum. Concensus of opinion helps with sorting out gear to try out.
 
If the influence of marketing and promotion was taken away from the reviews, I'd trust them a lot more, sadly the stats tell me not to trust them. Also the fact that a lot of reviewers are not very active on the forum tells me the same.
 
I'm not damning head-fi, I love the forum. The reviews, not so much.
 
Just one mans opinion, nothing more.
 
Jun 25, 2015 at 10:00 AM Post #123 of 155
Hey - no problem. And I appreciate the honest debate.

I guess from my own point of view (I've already covered this earlier so I won't restate the details), I put a lot of time into every review I write - most will take 1-2 weeks of research and often 10+ hours to write. Measuring and graphing can take an hour or more by itself. I do it because I genuinely enjoy the process. I always offer to return the gear if the manufacturer wants it back. And a lot of it I have paid for.

I guess as someone who is involved heavily in reviewing, I find it interesting when people talk about trust. If I was doing this for some free gear then it would be putting a really low value on my time.
 
Jun 26, 2015 at 5:31 AM Post #125 of 155
   
So your insinuating that all reviewers who don't own the gear can't be trusted because they are biased towards the manufacturer - yet all owners have no bias - even though we know justification bias exists?
 
My advice - find reviewers who own same/similar gear that you have yourself, who describe the gear you both share very similarly, who when reviewing look at both the good and bad points of the gear, and above all, the ones who are prepared to do the basics when reviewing and comparing (ie - list their gear, their known preferences and bias, and volume match when making comparisons). Oh - and avoid anyone who talks about night and day differences, and volume matches by ear 
wink.gif

 
Once you find two or three reviewers who meet your criteria - then you know if they have consensus on a piece of gear, you should be pretty safe looking at it more seriously.  As always though - there is no substitute for listening yourself.


There obviously is justification bias.  But your reasoning appears fallacious to me to the extent that it seems to imply that the existence of one deficiency could be justification for the another one (bias towards manufacturer).
 
I'm not attacking you in any way, and have no reason to doubt your integrity, let's be clear.
 
 But on principle I'm sceptical as to the neutrality of reviews where gear was made available by the manufacturer (and could or could not be kept by the reviewer after the review).  I don't know what guidelines about this are on head-fi but at the very least it should be made abundantly clear to the reader what the situation is with the loaner unit and if any direct or indirect incentive (keeping the unit, purchasing it or another one at a reduced price, invitation to be a guest at a conference....) was offered to the reviewer.
 
Jun 26, 2015 at 5:47 AM Post #126 of 155
There obviously is justification bias.  But your reasoning appears fallacious to me to the extent that it seems to imply that the existence of one deficiency could be justification for the another one (bias towards manufacturer).
 
I'm not attacking you in any way, and have no reason to doubt your integrity, let's be clear.
 
 But on principle I'm sceptical as to the neutrality of reviews where gear was made available by the manufacturer (and could or could not be kept by the reviewer after the review).  I don't know what guidelines about this are on head-fi but at the very least it should be made abundantly clear to the reader what the situation is with the loaner unit and if any direct or indirect incentive (keeping the unit, purchasing it or another one at a reduced price, invitation to be a guest at a conference....) was offered to the reviewer.

Totally agree on the need of having disclaimers on how the item being reviewed is acquired.
 
There will be bias with loaner units, special reviewing discounts, and even people who spend their hard-earned money to buy the product. Owners will prefer the item they personally picked and purchased. Bias is unavoidable. The goal should be transparency (so readers can be aware of possible biases). 
 
Jun 26, 2015 at 6:14 AM Post #127 of 155
  There obviously is justification bias.  But your reasoning appears fallacious to me to the extent that it seems to imply that the existence of one deficiency could be justification for the another one (bias towards manufacturer).
 
I'm not attacking you in any way, and have no reason to doubt your integrity, let's be clear.
 
 But on principle I'm sceptical as to the neutrality of reviews where gear was made available by the manufacturer (and could or could not be kept by the reviewer after the review).  I don't know what guidelines about this are on head-fi but at the very least it should be made abundantly clear to the reader what the situation is with the loaner unit and if any direct or indirect incentive (keeping the unit, purchasing it or another one at a reduced price, invitation to be a guest at a conference....) was offered to the reviewer.

 
Its all good mate - we're having a civilised discussion - and it's nice to share different POV. For me as a reviewer, I like seeing how others view what I write as well.  At the end of the day - I actually write more for me than for a manufacturer.  I always have.  If they produce good gear they get the kudos.  As to your other points - here is what i posted earlier in this thread.  For me personally it answers your concerns.  I apologise in advance as I know it's a bit wordy.  I've highlighted the important bits this time:
 
  First up - thanks for some of the previous comments - it is nice to know that some of my reviews come in handy for those who share similar tastes.
 
As one of the contributing "semi-regular" reviewers on this site, all I can do is provide a little of my own philosophy and insight - into what I do, how I write, and what drives me.
 
  • I'm not paid, never have been.  I have a real job.  This is a hobby.  I write because I genuinely enjoy doing it.
     
  • I have extensively tested myself (as objectively as possible - a lot of ABX) to understand my own limitations and preferences.  I then preface every review I do with my understanding of my own bias and physical limitations. I don't have 'golden ears' - far from it in fact.  And I also prefer people to know my own views on cables, formats etc so that they can then form their own views on how useful my opinion will be to them.
     
  • I think reviewer bias plays a big part in each review - and I know (for example) that I very much dislike warm bassy headphones.  So I try to state this clearly and give examples of headphones I've owned and liked, and those I haven't.
     
  • One thing I am very conscious of now is that some people may be making buying decisions based on what I like - so I try very hard to make sure I don't overhype something.  The other thing I try to do is test thoroughly (ie put at least 20-30 hours listening into a piece of gear) before I put pen to paper.
     
  • I do always acknowledge if the product being reviewed is mine, borrowed, or a review sample.  I think this is very important.
     
  • I don't solicit reviews - the closest to me doing this is when the occasional manufacturer thread appears asking for interest in reviewing something - and I will generally express an interest in these threads.  But I don't email manufacturers asking for review samples.  Why?  Because it would make me feel uncomfortable.  Most of my contacts have either been "out of the blue" - manufacturers contacting me on this site via PM asking me to review something, or referrals from other reviewers where they have suggested to a manufacturer that I might be a good person to contact.
     
  • With virtually every manufacturer I have offered to either return (generally at my cost), or purchase review units. I do this for my own peace of mind. Most manufacturers have thanked me but explained clearly that the units are review units and no need to return or pay for them.  But the offer is always there.  I have at different times purchased review units from Fiio, Fidue, T-Peos, Beyerdynamic and others - and I have at least offered to do so for everyone else.
     
  • You'll find with my reviews that they aren't all positive.  I try to write exactly what I think about the product in question - and this means often going against the grain / mass opinion of other reviewers. I try to do this respectfully and objectively.
     
  • There is some gear that I have received, listened to, and then given the manufacturer an impression + the option for me not to review (often if the product is in my opinion really bad). I believe this is fair to the manufacturer, and with some it has led to open discussion on flaws with the product in question, and often improvements being implemented.  It is particularly gratifying when this happens, and I think one of the best things about manufacturers being very involved with the Head-Fi community.
 
Hopefully this sheds light into what drives me personally, and reassures those who think a lot of reviewers are shills simply looking for "free stuff".  It wouldn't be worth it given how long it takes me to put a review together, For me it is still all about the joy of the hobby :)

 
If you have a look tomorrow you'll see a review from me on the latest from T-Peos.  Depending how they take it, I may not get another chance to review another item from them.  But it is my honest view, and I have never posted anything with the express intention of trying to get more gear out of a manufacturer.  For a start off, I don't need it.  I own my DAC, amp, full sized headphones, and many of my IEMs.  Anything that's not owned is listed in my profile as a review sample.
 
I don't see what I can do more if I wish to keep reviewing.  I can't afford to buy every piece of gear, and I always offer to return the gear.  So its a catch 22 - do I just hang up my reviewing boots?
 
Jun 26, 2015 at 8:24 AM Post #128 of 155
You've been more than transparent, so no need to even think of giving up reviewing. The fact that you're taking it personally is a good thing, it means you care about doing it and that's why you're why you're defending it
normal_smile .gif

 
From my point of view I've already stated that I regard you as a good reviewer and I can assure you it was a 100% genuine comment. But there definitely needs to be more transparency regarding loaner units, deals etc with regards to reviews.
 
For me it was seeing the rise of the X3ii that really shocked me and made me question things. It was number 1 DAP before it even came out and now everyone will be recommending it for people to buy.  Hell, I don't know but it just appears shady to me with them being sponsors here. It just looks as though reviewers are doing the marketing for these companies. It's probably a great DAP but as a consumer I would feel as if I had been manipulated into buying it. I know that's how marketing works before someone states it but it just makes me very uncomfortable here at head-fi with regards to reviews. That's why I questioned earlier in thread about who head-fi is serving.  I'll be watching the situation with the X5ii here to see if it happens again. (Already #6 DAP, 19 Reviews, All loaners, 1 has since been purchased by reviewer).
 
1000's more people visit the forum than actual members so the potential for sales influence here is HUGE. There's even a reviewer now (A Sponsor & An Industry Insider) who is reviewing gear that his company is selling and linking direct (16 Links in one review) to the product for sale on their site, (both reviews are overwhelmingly positive by the way. EDIT. Both reviews revoked and are currently HELD FOR MODERATION).
 
I'm not trying to turn this into a conspiracy theory but there's things happening here that just don't seem to add up to me. There's reviewers who are not active on the forum in the community. That just seems plain weird to me. Reviewing seems to be a progression move from the forum to me, jumping straight into reviews is strange.
 
Reviews on loaner units appear to be overwhelmingly positive. Yes, there are negative reviews but they do seem to be in the minority. As I stated earlier, everytime I see a negative review it shocks me. Maybe you've inadvertently hit the nail on the head with your post. If someone writes a negative review on a companys product maybe they won't get asked to review by that company again, maybe that's another reason.
 
Maybe hitting a certain post count before being allowed to write reviews is a start so we can get to know reviewers around the forum.
 
All I know is that there needs to be more transparency, at least from my point of view to even begin to trust most reviews on here and to prove that it's something more than a marketing initiative.
 
 
Updated 1st August 2015
 
Jun 26, 2015 at 9:53 AM Post #129 of 155
My opinion:
 
Modern audio equipment is way past the human hearings threshold when it comes to capabilities. A reviewer should be a person that has the equipment to measure frequency response graphs on several different SPL's, ringtimes, impedance changes, phase changes and so on. The human ear is a POS with to many variables to be a useful tool for more than a highly subjective indication. And the human brain is a organ that to easily fools itself to be a good companion to those ears.
 
The sad truth is that there is little one can do to influence a speakers characteristics. If you look at the TS parameters then you quickly see that when one quality is raised some other quality goes down. It doesn't mater if you use expensive materials, the best materials are often some of the cheapest to use. Not much new has happened since the 60's when it comes to speaker design. Sure we have better amplifiers now that covers some of the shortcomings of certain speaker designs, but meh.
 
Its difficult for a "normal" human to quantify the qualities of audio gear, one cant easily measure how it performs without buying loads of extra gear and obtaining skill-set to use said gear.
Imagine that there where no benchmark programs for computers, that we where left to judge by your sight what GPU gave most peak fps, average fps and minimum fps during a play-trough. It would leave loads of room open for argument, especially when it comes to high end GPU's that easily can break 150-200 fps on 1080p when playing a RTS. And we see no place where input lag of a GPU is mentioned, and that's just because there would be extra gear and skills in need for measuring a GPU's frame construction lag, an important and overlooked quality.
 
Now go watch MST3K and be a happy human being.
 
Jul 22, 2015 at 8:05 AM Post #130 of 155
The same thing is happening again with the Fiio X5ii. It's already the number 8 DAP on reviews. This is based on 15 reviews all of which are loaners. The other questionable thing I just noticed is that they seem to be pre-preduction units so the units reviewed may not be the same as ones that are available for purchase.
 
Granted, usually problems are usually sorted when a product comes to market but anyone who has dabbled in hifi equipment (I've been buying for 27 years) knows that this is not always the case. I stand by my conclusion that this is not an open unbiased forum, it's a marketing one and I would recommend that people take the reviews with a HUGE pinch of salt. It's becoming increasingly obvious to me that Headfi is serving the marketing departments of these companies far more than normal members.
 
I think reviews should be limited to actual purchases. It's not going to solve the whole bias debate but it would make things a hell of a lot clearer, upfront and honest.
 
In my experience if it looks like a fish, smells like a fish and tastes like a fish, hell it's a fish and that's what the whole reviews situation is FISHY.
 
Jul 22, 2015 at 10:09 AM Post #131 of 155
I think you are confusing a community full of shills with a community that has a strong working relationship with manufacturers. To see the difference, you need to attend a big head-fi meet, like a CanJam. Community members & manufacturers freely flow from one side of the table to the other. I'm not saying that there are no shills on head-fi - of course there are - but there are also many, many people that are passionate about this hobby and some of those people also happen to work for companies that are also suppliers to this hobby. I don't see that as always a bad thing - I love that there are people here that both love head-fi AND make a living from head-fi. Smart head-fi manufacturers openly involve this community in their product development and testing. Yes, they give out loaners in exchange for reviews - and good head-fiers ALWAYS say that in their reviews. Does that make the review biased? Yes. Does that mean you have to completely dismiss that review or all reviews on head-fi? No. It becomes one more data point in your decision process, along with reviews by Tyll at InnerFidelity, |joker| at theheadphonelist.com, etc, etc. BTW, all of those guys are also head-fiers.

Which raises a more practical point: If you dismiss all reviews of headphones done by head-fiers, then you dismiss *all* reviews of headphones. Because, *all* headphone reviews are by head-fiers or ex-head-fiers.
 
Jul 22, 2015 at 6:31 PM Post #132 of 155
The same thing is happening again with the Fiio X5ii. It's already the number 8 DAP on reviews. This is based on 15 reviews [COLOR=FF0000]all of which are loaners[/COLOR]. The other questionable thing I just noticed is that they seem to be pre-preduction units so the units reviewed may not be the same as ones that are available for purchase.

Granted, usually problems are usually sorted when a product comes to market but anyone who has dabbled in hifi equipment (I've been buying for 27 years) knows that this is not always the case. I stand by my conclusion that this is not an open unbiased forum, it's a marketing one and I would recommend that people take the reviews with a [COLOR=FF0000]HUGE[/COLOR] pinch of salt[COLOR=FF0000].[/COLOR] It's becoming increasingly obvious to me that Headfi is serving the marketing departments of these companies far more than normal members.

I think reviews should be limited to actual purchases. It's not going to solve the whole bias debate but it would make things a hell of a lot clearer, upfront and honest.

In my experience if it looks like a fish, smells like a fish and tastes like a fish, hell it's a fish and that's what the whole reviews situation is [COLOR=FF0000]FISHY[/COLOR].


Mine is now owned. So at least one of those 15 is an owner. And the rest (review units) AFAIK are listen for a review - not giveaway for a review, so I think your grandstanding on the issue sadly misses the point. If you knew how long it takes to write a decent review - then the payback of a week with a unit in return for a review is not really anything most people would regard as a huge reward. What it is instead is a chance to listen to some audio gear that you may not have a chance to spend time with otherwise. I stand by all of my reviews. Funny that most of the comments I get from many, many people regarding my reviews is mostly "thanks".

Personally I find your comments regarding all non-paid for gear reviewers (generalisations) completely contrary to what I see from most reviewers.
 
Jul 22, 2015 at 10:24 PM Post #133 of 155
Which raises a more practical point: If you dismiss all reviews of headphones done by head-fiers, then you dismiss *all* reviews of headphones. Because, *all* headphone reviews are by head-fiers or ex-head-fiers.


The two major audiophile publications - TAS and Stereophile - have yet to make their presence known on this site - not that I've seen - or even mention it with any degree of frequency in their headphone reviews. Different audience, I understand, where the median age is ~55 [my guess] vs here. Different expectations and POV, as well.

:)
 
Jul 22, 2015 at 11:17 PM Post #134 of 155
InnerFidelity is part of the same corporate entity that owns Stereophile and Sound & Vision. As for TAS, well, quite frankly, I'm actually quite happy those guys aren't really part of the head-fi community. If I want to learn new adjectives and poetic phrases that have no basis in science, I read TAS...
 
Jul 23, 2015 at 1:13 AM Post #135 of 155
  The same thing is happening again with the Fiio X5ii. It's already the number 8 DAP on reviews. This is based on 15 reviews all of which are loaners. The other questionable thing I just noticed is that they seem to be pre-preduction units so the units reviewed may not be the same as ones that are available for purchase.
 
Granted, usually problems are usually sorted when a product comes to market but anyone who has dabbled in hifi equipment (I've been buying for 27 years) knows that this is not always the case. I stand by my conclusion that this is not an open unbiased forum, it's a marketing one and I would recommend that people take the reviews with a HUGE pinch of salt. It's becoming increasingly obvious to me that Headfi is serving the marketing departments of these companies far more than normal members.
 
I think reviews should be limited to actual purchases. It's not going to solve the whole bias debate but it would make things a hell of a lot clearer, upfront and honest.
 
In my experience if it looks like a fish, smells like a fish and tastes like a fish, hell it's a fish and that's what the whole reviews situation is FISHY.

 
@taffy2207, I agree that the ranking system on head-fi is flawed. Based more on ownership popularity or whether a product gets more exposure on a review tour. However, I do really applaud companies for offering such opportunities to our community. Not many of us would have the opportunity to buy multiple gear setups to do direct comparison reviews. That sort of practice does allow different users with different gear to write direct comparisons reviews on products. I think review tour reviews are actually quite helpful as owners will inherently be biased towards the gear they picked & purchased. The practice of loaner review units allows a more people share some impressions on different and new products that they may not have personally chosen.
 
I agree that audio reviews in general should always be taken with a HUGE pinch of salt. There is just too many personal variables at play here and everyone has different reference points and personal preferences. I find good reviews made by good reviewers. While there can be some biases with ownership or review tours, I think that reviewers who strongly care about providing comparative information for prospective buyers rather than simply pushing their favorite pick always provide useful information, regardless of whether I personally agree with them. Poor reviews can be found by owners, loaner reviewers, and even demo impressions. Really depends more on the reviewer than the format imo, but I think it is extremely important to include a disclaimer that states what sort of listening conditions and length of time spent with the item.
 

Users who are viewing this thread

Back
Top