When trust disappears, Reviewers are simply “noise”
Nov 3, 2017 at 7:36 PM Post #16 of 198

TheoS53

Headphoneus Supremus
Joined
Mar 17, 2014
Posts
1,986
Likes
1,183
You've hit the nail on the head with much of what you've said there.

I brought up the point of "who I write for" in the Hugo 2 thread.
Using colorful language and otherwise embellishing a product is the job of a company's marketing team, not mine. I write and record reviews based on what is important to me, and hopefully that lines up with what would be important to readers/viewers too.

What I didn't agree with is what you said about the Kinera H3. But, I think the problem here is QC. I actually quite liked the H3 for what it was, but there did seem to be inconsistencies with their QC, and so people had differing experiences with the product.

But you're right, there really does seem to be a culture of trying to please the manufacturers, and ultimately a play on people's trust (and a person's integrity as a reviewer). I suppose this is an issue of ethics, really.
 
Nov 3, 2017 at 7:36 PM Post #17 of 198

Mython

Headphoneus Supremus
Joined
Feb 11, 2008
Posts
8,760
Likes
3,388
On several occasions, both publicly (sometimes in comments section at the end of some reviews), and sometimes via PM, I have discussed exactly this state of affairs, Brooko. I am sick & tired of seeing Head-fiers receiving free product in exchange for their reviews, and believing that as long as they admit they got a free sample, at the beginning of their review, then that makes it all above board and they're in the clear.

It doesn't.

There is a very good reason why in many careers, if one accepts free product, it is viewed as a disciplinary offence; often one warranting dismissal.

Personal audio seems particularly rife with this behaviour, from both reviewers and vendors, and it leaves a very bad taste in the mouth,
 
Nov 3, 2017 at 7:40 PM Post #18 of 198

Tiddlesworth

500+ Head-Fier
Joined
Jun 15, 2013
Posts
997
Likes
86
Great points. And I would fully expect reviews to be different. We are after all unique. I think though that you may have missed the point about the SNR and the fact that I believe that some of the current reviews are not merely different in the fact that they are different, but more in the fact that something which has real issues (like real problems with frequency response), can be glossed over and rated super high. RHA CL1 would be another polarising IEM which I think comes down to not only mere preference. When there is a 20 dB lift from mid-range to lower treble peaks, there are some genuine issues there.

And the Kinera H3 one is something that I really puzzled over. Me choosing it as a review sample is the same as I would be doing as a buyer. i based it on the relentless positivity of almost every review. How can something with such obvious issues be given 4s and 5s, and make front pages when the reviews do not state the issues clearly?

Maybe I'm expecting too much?

That's why it should be the reader's responsibility, as I mentioned a reader either finds a reviewer they really agree with or they read loads of them, no one is gonna do a meta analysis for you. At least you yourself still have a passion for the hobby, it is exactly this noise why I gave up on reading reviews or taking them in any way seriously, I'll just wait till I get a chance to try them, I'm just glad I'm pretty much at "end game" so I have no rush to upgrade, every company seem to be competing on ways to add more zeroes to the price tag as oppose to actually making better sounding headphones.
 
Last edited:
Nov 3, 2017 at 7:41 PM Post #19 of 198

sjb57

100+ Head-Fier
Joined
Dec 21, 2016
Posts
289
Likes
67
Thats actually a good point - but ethically I an't do that. And my point is that i don't want to make anything out of this. i do it because I enjoy listening to new stuff, and I enjoy writing. And it gives me genuine pleasure to contribute and be part of the wider discussion.
Do the reviews you want. Keep the gear they give you for as long as you need it. Then, after say 6 months, sell it on an open forum. So, everybody now knows what you got. Then, give the cash to your church/community/charity and (if you really feel it necessary) post the receipt from your church et al as the notice of item sold. OK some git could twist your motivation to reviewing for piety, but you might not be too hurt by such a person's view.
And, the buyer of such gear gets a unit that they will know inside out from reading your review!
 
Last edited:
Nov 3, 2017 at 7:49 PM Post #20 of 198

ryanjsoo

Reviewer for The Headphone List
ryanjsoo's Reviews
Joined
Oct 5, 2012
Posts
1,052
Likes
3,394
Location
Australia
"Reviewing audio is highly profitable, I'm only in it for the money and the samples"

uKvhglu.png


And Brooko, I can't speak for everyone, but I'm very much still within that cringe period. Give us newer reviewers some time and we won't disappoint!
 
Last edited:
Nov 3, 2017 at 8:35 PM Post #21 of 198

sjb57

100+ Head-Fier
Joined
Dec 21, 2016
Posts
289
Likes
67
Another view:
Head-Fi is clearly a commercial venture - just look at the adverts & many have nothing to do with audio. We are their product and Head-Fi gets paid for allowing advertisers to target us.
Reviewers put in a lot of time & effort & we (me: inexperienced buyer) read some of them + others comments to get information / make better buying decisions & not waste money. There is value in this for the reader. Why should there not be value in it for the reviewer? Rightly, or wrongly, millions of people buy Which? magazine because they trust it & are willing to buy their advice. Head-Fi wants us to keep visiting their site so that sponsors / advertisers pay them more and reviewers (& moderators) are giving Head-Fi tremendous value that Head-Fi can leverage. So, each of us just needs to understand that this is a symbiotic relationship. Punters like me read reviews but I put far more value on the comments & problems that get flagged by community members who have already bought the product. Reviews would count more if I was an eager early adopter & could not wait a few months for the truth to come out or a new product thread to die due to lack of enthusiasm (contrast Fiio X7ii versus Fiio X3iii and watch & wait Shanling M3s for example).
 
Last edited:
Nov 3, 2017 at 8:43 PM Post #22 of 198
Joined
Jan 30, 2011
Posts
13,315
Likes
24,298
What I didn't agree with is what you said about the Kinera H3. But, I think the problem here is QC. I actually quite liked the H3 for what it was, but there did seem to be inconsistencies with their QC, and so people had differing experiences with the product.

I'd believe this if the measurements are all over the place. But check with Crinnacle. His and my measurements are almost a perfect match. I've also seen another pair with the same curve. If Kinera has got some real issues with consistency in tuning, then we need to uncover that. How about getting some of these other pairs of 4/5 star IEMs in for measuring?
 
Nov 3, 2017 at 8:46 PM Post #23 of 198
Joined
Jan 30, 2011
Posts
13,315
Likes
24,298
Do the reviews you want. Keep the gear they give you for as long as you need it. Then, after say 6 months, sell it on an open forum. So, everybody now knows what you got. Then, give the cash to your church/community/charity and (if you really feel it necessary) post the receipt from your church et al as the notice of item sold. OK some git could twist your motivation to reviewing for piety, but you might not be too hurt by such a person's view.
And, the buyer of such gear gets a unit that they will know inside out from reading your review!

Unfortunately we can't do that - we sell, we're banned. And I can't sell what isn't mine. I don't own any of it unless I pay for it. Its all loaner. Third reason - I'm constantly asked to do comparisons with other gear I have. Can't do that if its not here. I like the idea though. If/when I eventually exit the reviewing side, I'd then ask the manufacturers if they are OK with it - and its a great idea of the donations for charity.
 
Nov 3, 2017 at 8:52 PM Post #24 of 198
Joined
Jan 30, 2011
Posts
13,315
Likes
24,298
Another view:
Head-Fi is clearly a commercial venture - just look at the adverts & many have nothing to do with audio. We are their product and Head-Fi gets paid for allowing advertisers to target us.
Reviewers put in a lot of time & effort & we (me: inexperienced buyer) read some of them + others comments to get information / make better buying decisions & not waste money. There is value in this for the reader. Why should there not be value in it for the reviewer? Rightly, or wrongly, millions of people buy Which? magazine because they trust it & are willing to buy their advice. Head-Fi wants us to keep visiting their site so that sponsors / advertisers pay them more and reviewers (& moderators) are giving Head-Fi tremendous value that Head-Fi can leverage. So, each of us just needs to understand that this is a symbiotic relationship. Punters like me read reviews but I put far more value on the comments & problems that get flagged by community members who have already bought the product. Reviews would count more if I was an eager early adopter & could not wait a few months for the truth to come out or a new product thread to die due to lack of enthusiasm (contrast Fiio X7ii versus Fiio X3iii and watch & wait Shanling M3s for example).

This is the bit I can't get my head around though. What its suggesting is that its OK to shill because that is the business we are in? The whole reason why a lot of the reviews now aren't trusted is because of that expectation. And if that is where we are, and there is no way out - then I'd rather just give it all away, stick with my own gear, and participate in loaner programmes (and forget about the comparisons). And if that is what it has come to - then Head-Fi (sadly) is no longer the site for me.

The one thing which could potentially make a huge difference is how the admins here choose their front page content. If they are rewarding accuracy, then they'll get accurate reviews. if they are rewarding "picture and advertising" - thats what they'll get.
 
Nov 3, 2017 at 9:01 PM Post #25 of 198

sjb57

100+ Head-Fier
Joined
Dec 21, 2016
Posts
289
Likes
67
Unfortunately we can't do that - we sell, we're banned. And I can't sell what isn't mine. I don't own any of it unless I pay for it. Its all loaner. Third reason - I'm constantly asked to do comparisons with other gear I have. Can't do that if its not here. I like the idea though. If/when I eventually exit the reviewing side, I'd then ask the manufacturers if they are OK with it - and its a great idea of the donations for charity.
Banned from what? If all review loans are forgiven / forgotten about then it is actually a gift not a loan (you are the one stipulating it a loan, other reviewers say it was gifted). So, you do the review on the basis that you will sell their gift when you deem you are finished with it - at some point it will become irrelevant to most enthusiasts or someone else who still has it will help them. Keep it simple. Keep doing the reviews & raise a bit of cash for charity. Who is going to ban that? And, if they do maybe they are the wrong people to work with?
 
Nov 3, 2017 at 9:08 PM Post #26 of 198

a_recording

Member of the Trade: Lachlanlikesathing
Joined
Mar 20, 2009
Posts
4,705
Likes
784
Hey Brooko, some good thoughts here. Regardless of how we may think differently about crowdfunding, I agree with you that there is an increasing issue with the sheer volume of information and reviews.

Unfortunately, after a long time I have come to think that there really is little possible remedy. The incentives are really aligned against high quality information. Putting aside the issue of review units themselves, at the end of the day most manufacturers regard reviews and reviewers as an extension of PR/media. The larger companies have departments that exist to manage relationships with reviewers. I am sure you would know this from the email footers you receive from them. If manufacturers were interested in product feedback, you would be in contact with engineers rather than PR departments. But engineers are very busy, so most of the time you speak with people whose job it is to maximise exposure for their company. When a reviewer gets to a certain level of clout (like say Tyll) they get direct access to pick the engineers brains. This is great thing, but there is gatekeeping for obvious reasons. And of course it takes a significant level of investment to get to any kind of prominence.

But more then that, there is simply going to be a systemic bias towards positive reviews. Positive reviews and hype generate consumer interest, which generate more purchases, which generate more interest in reviews of that product. Meanwhile negative reviews, unless they are entertainingly controversial, don't tend to create these cycle. And then the superstructure of forums and other platforms thrive on these cycles.

You can draw a line in the sand, but the market will simply pass you by because to a manufacturer and to a forum, all interest is signal and not noise. When there are so many products being released, the resource that manufacturers compete for is time and attention, not quality feedback. In the same way, when forums have to compete against so many other ways of getting information, again the precious resource is time and attention. And the same incentive applies to reviewers, no matter where the money or the hardware is coming from.

For most people this isn't really an important thing. At the end of the day maybe it isn't important. There will be reviews from people who are not fussed, and an audience for these reviews which is not fussed. And then there will be people who do feel the system is flawed, so they will be cautious and selective about their information sources. There will be some people who will value your reviews because they trust you and feel that your opinions align with theirs over time. And then there are people who just want to have a good time geeking out about things they like. At a certain point you have to accept that you are part of a much larger process of information exchange, and you will only address a segment of the audience that relates to you. You can influence things by persistence and example to grow that segment, but there will always be people who are comfortable with different approaches.

And you will always, always, always be called a shill :)
 
Last edited:
Nov 3, 2017 at 9:08 PM Post #27 of 198

sjb57

100+ Head-Fier
Joined
Dec 21, 2016
Posts
289
Likes
67
This is the bit I can't get my head around though. What its suggesting is that its OK to shill because that is the business we are in? The whole reason why a lot of the reviews now aren't trusted is because of that expectation. And if that is where we are, and there is no way out - then I'd rather just give it all away, stick with my own gear, and participate in loaner programmes (and forget about the comparisons). And if that is what it has come to - then Head-Fi (sadly) is no longer the site for me.

The one thing which could potentially make a huge difference is how the admins here choose their front page content. If they are rewarding accuracy, then they'll get accurate reviews. if they are rewarding "picture and advertising" - thats what they'll get.
I am naturally cynical, so I read reviews on a commercial forum with that in mind and Head-Fi is clearly a commercial forum. But, community members generally reveal the truth after a period of ownership. It is only a shill if it is under the counter - transparency is a very powerful disinfectant & kills 99.9% of all lies.
 
Nov 3, 2017 at 9:10 PM Post #28 of 198

ryanjsoo

Reviewer for The Headphone List
ryanjsoo's Reviews
Joined
Oct 5, 2012
Posts
1,052
Likes
3,394
Location
Australia
Banned from what? If all review loans are forgiven / forgotten about then it is actually a gift not a loan (you are the one stipulating it a loan, other reviewers say it was gifted). So, you do the review on the basis that you will sell their gift when you deem you are finished with it - at some point it will become irrelevant to most enthusiasts or someone else who still has it will help them. Keep it simple. Keep doing the reviews & raise a bit of cash for charity. Who is going to ban that? And, if they do maybe they are the wrong people to work with?

You can get banned from Head-fi for flipping review samples. In theory it's supposed to make us more impartial since otherwise, people would get a few samples, write a half-assed review and sell for profit. There is no shortage of brilliant people here in it for passion but a trust based system like that still isn't viable unfortunately.
 
Nov 3, 2017 at 9:20 PM Post #29 of 198
Joined
Jan 30, 2011
Posts
13,315
Likes
24,298
Hey Brooko, some good thoughts here. Regardless of how we may think differently about crowdfunding, I agree with you that there is an increasing issue with the sheer volume of information and reviews.

Unfortunately, after a long time I have come to think that there really is little possible remedy. The incentives are really aligned against high quality information. Putting aside the issue of review units themselves, at the end of the day most manufacturers regard reviews and reviewers as an extension of PR/media. The larger companies have departments that exist to manage relationships with reviewers. I am sure you would know this from the email footers you receive from them. If manufacturers were interested in product feedback, you would be in contact with engineers rather than PR departments. But engineers are very busy, so most of the time you speak with people whose job it is to maximise exposure for their company. When a reviewer gets to a certain level of clout (like say Tyll) they get direct access to pick the engineers brains. This is great thing, but there is gatekeeping for obvious reasons. And of course it takes a significant level of investment to get to any kind of prominence.

But more then that, there is simply going to be a systemic bias towards positive reviews. Positive reviews and hype generate consumer interest, which generate more purchases, which generate more interest in reviews of that product. Meanwhile negative reviews, unless they are entertainingly controversial, don't tend to create these cycle. And then the superstructure of forums and other platforms thrive on these cycles.

You can draw a line in the sand, but the market will simply pass you by because to a manufacturer and to a forum, all interest is signal and not noise. When there are so many products being released, the resource that manufacturers compete for is time and attention, not quality feedback. In the same way, when forums have to compete against so many other ways of getting information, again the precious resource is time and attention. And the same incentive applies to reviewers, no matter where the money or the hardware is coming from.

For most people this isn't really an important thing. At the end of the day maybe it isn't important. There will be reviews from people who are not fussed, and an audience for these reviews which is not fussed. And then there will be people who do feel the system is flawed, so they will be cautious and selective about their information sources. There will be some people who will value your reviews because they trust you and feel that your opinions align with theirs over time. And then there are people who just want to have a good time geeking out about things they like. At a certain point you have to accept that you are part of a much larger process of information exchange, and you will only address a segment of the audience that relates to you. You can influence things by persistence and example to grow that segment, but there will always be people who are comfortable with different approaches.

And you will always, always, always be called a shill :)

Hey Lachlan - nice to see you back. I guess I really am banging my head against the proverbial wall here then.

Here's the question though - if I look at the number of followers, it clearly indicates the way I'm reviewing has more value to a lot of people than a lot of others. If that is the case - then why can't we change the system? Companies want to make better products - lets help them. Consumers want a better idea of the bad ones - lets help them. Head-Fi wants to bring more traffic in - lets do it by accuracy in what is written.

Surely the alternative is that eventually all standards drop, and we're left with a site which has such a low SNR ratio a lot of people no longer see the value in it. This leads to lower numbers of credible reviews. Which leads to lower sponsorship. Which ultimately leads to a self-fulfilling negative cycle.

There is a reason why (until recently) I have chosen to keep virtually all my reviews here, to not look for any rewards or money (although plenty is on offer), and keep the standards I have done.

And I refuse to believe it is impossible to change this cycle.
 
Nov 3, 2017 at 9:20 PM Post #30 of 198

sjb57

100+ Head-Fier
Joined
Dec 21, 2016
Posts
289
Likes
67
You can get banned from Head-fi for flipping review samples. In theory it's supposed to make us more impartial since otherwise, people would get a few samples, write a half-assed review and sell for profit. There is no shortage of brilliant people here in it for passion but a trust based system like that still isn't viable unfortunately.
Punters are not thick & will quickly spot & call out such a corrupt reviewer. Unfortunately, some punters are cruel too. And, it you are selling after say 6 months you are only getting a fraction of the original value. OK might still be good money on expensive kit but TRANSPARENCY is the answer, not pretending gifts are loans or sample.
 

Users who are viewing this thread

Top