When trust disappears, Reviewers are simply “noise”
Nov 5, 2017 at 3:10 PM Post #166 of 198

voxie

500+ Head-Fier
Joined
Jun 5, 2014
Posts
909
Likes
181
Hi Guys.. this a quick message to the "Demi Head-Fi God Reviewers".. This tread has now become a bitch fest. Lighten up, Head-Fi is all about community, not about slagging off one another. The egos on this tread makes me want to give up!!!
 
Last edited:
Nov 5, 2017 at 3:33 PM Post #167 of 198
Joined
Jan 30, 2011
Posts
13,315
Likes
24,298
Great questions :)

I put the max score at 100 simply to make it easier - so in my example 56/100, but the weighted score would be 5.475/10 or 2.7/5. In the star system it would rate either a 2.5 or a 3.0. It matches pretty much what I think of it - really great build and has the basics right. With the right EQ it sounds pretty good, so not a write-off. But unless you're willing to EQ - I wouldn't recommend it. The good news is that they are working on the faults, and I believe the next release will be a lot better. What worries me (and the whole point of the blog article) was that if Kinera had relied on the feedback of the 4-5 star reviews, they may not be addressing all of the (quite obvious) faults.

Personally I'd have no issues with people being paid for reviews - as long as they state it. And in Theo's case its coming from a store and not an actual Manufacturer - so there is little likelihood of bias. There is more likelihood in the current free sample system of bias being introduced.

I don't think a lot of people realise the time that goes into each review either. For me personally, it can take:
- about 8-10 hours casual listening and experimenting with combos (that's the fun part)
- usually 2-3 hours straight measuring and remeasuring
- at least 4-5 hours critical listening and taking notes
- 5-6 hours actually writing the review
- 2 hours photography and editing
- 2 hours preparing to post, posting and editing.

Thats just a rough summary - but shows the work that can go into a single review. So for me, it can take 20-25 hours for a single review. At around 2-3 hours a day - you can see why most of my reviews take at least a week to put together. I'll sometimes have 3 or 4 on the go at once, but it is very time consuming.

On the question of poor reviews getting buried - I think this is where the reputation of the reviewer comes in. Take the ZhiYin QT5 for example. That one had several positive reviews, and the claims of issues from some buyers were being buried by some "reviewers" (as being issues which needed help from burn-in, or cable issues etc). It was still being hyped until finally someone with enough rep stepped in, measured it, and put up an alternative point of view. It happened to be me - but could have been one of half a dozen reviewers with enough rep that people listen. The problem is that a lot of the "new" reviewers are trying to build rep - but they are going the wrong way about it. They have poor methodology, are very subjective, and are writing for the manufacturer and not from a potential buyer's point of view. Until this changes, the SNR on Head-Fi will continue to be poor and dropping (IMO anyway). One of the big issues - is that these poor reviews are being featured (regularly) on the Head-Fi front page. The reviews might look pretty but they have little substance and very little objective fact. In short - a lot are advertising pieces for the manufacturer. How many of the reviewers you could really trust are still here? A lot have left already. I am probably leaving (or at least reducing my input) - simply because I'm tired of seeing things the way they are, and seeing no aptitude to create change. I genuinely like the people and the audio discussions here - but I am finding it increasingly difficult to hold my tongue when I see some of the advice doled out by people who really don't have a clue. If you can't change the system - change the environment. This is my last real shot at changing the system - hopefully because I have enough respect to be listened to. If I can't - then I change the environment (I move on). Its simple really.

I'm also pleased to see you talking about DAPs in terms of features. And that is my aim as a reviewer - to talk about potential user experience. If I can (through a review) supply a holistic example or real world use for a potential buyer - then they can hopefully be more informed when they make those buying decisions. Ultimately you end up finding 1 or 2 reviewers who align with your own thoughts on what is important, and maybe whose preferences align with yours. Then you follow them and read (but largely ignore) the noise from others. Its what I do if I'm thinking of buying.
 
Last edited:
Nov 5, 2017 at 3:39 PM Post #168 of 198
Joined
Jan 30, 2011
Posts
13,315
Likes
24,298
Hi Guys.. this a quick message to the "Demi Head-Fi God Reviewers".. This tread has now become a bitch fest. Lighten up, Head-Fi is all about community, not about slagging off one another. The egos on this tread makes me want to give up!!!

Thanks for that insightful and useful post. Now perhaps (unless you have something to actually contribute) you'd like to run along and go play somewhere else.
 
Nov 5, 2017 at 4:20 PM Post #170 of 198

sjb57

100+ Head-Fier
Joined
Dec 21, 2016
Posts
289
Likes
67
Great questions :)

I put the max score at 100 simply to make it easier - so in my example 56/100, but the weighted score would be 5.475/10 or 2.7/5. In the star system it would rate either a 2.5 or a 3.0. It matches pretty much what I think of it - really great build and has the basics right. With the right EQ it sounds pretty good, so not a write-off. But unless you're willing to EQ - I wouldn't recommend it. The good news is that they are working on the faults, and I believe the next release will be a lot better. What worries me (and the whole point of the blog article) was that if Kinera had relied on the feedback of the 4-5 star reviews, they may not be addressing all of the (quite obvious) faults

Thanks for the full reply. My only response would be if I saw your detailed scoring & something designed for audio scored 2/8 on so many audio qualities then I would write it off. Why on earth would I want to do a load of EQ that the manufacturer should have sorted when there are lots of alternatives for my money? To my mind, such a poor score on audio deserves a Red Flag Warning, irrespective of build / price. Maybe increase weighting to produce < 2 stars. BTW I bought the F9 inspite of the peak you mentioned because it was easy fix & everything else seemed spot on. Actually the peak does not bother me & I prefer no EQ in any case.

Generally, I see some consensus & lots of goodwill above BUT without a published change of policy statement from Head-Fi you are most likely trying to tell the tide (of cash flow) to turn. In the short term this web site is very valuable as things are now and probably worth a lot of money if it were to be sold. Will the owners risk a financial hit by some products getting slated even if such a strategy would increase its longevity & total value? I do not know.
 
Last edited:
Nov 5, 2017 at 4:27 PM Post #171 of 198
Joined
Jan 30, 2011
Posts
13,315
Likes
24,298
I agree with you on the Kinera - which is why I wouldn't personally recommend it. There are some question marks about whether the early review units were tuned differently - but I know the ones Crinnacle and I got measure almost identically, and he has similar opinion to mine on their rating. What would be good is getting one of the "5 star" review units and measuring it. Then we'd know for sure.

Unfortunately last time I suggested this with the ZhiYin QT5 there was at least one well known reviewer (external site - not Head-Fi) who quite neatly avoided taking us up on our offer of measuring his pair.

Sadly - I tend to agree with you on the "policy" decisions at this site. I can't fault them either - its not my business model. If I've got people talking about it though, and recognising there is an issue - then IMO we've taken the right step forward.
 
Nov 5, 2017 at 4:45 PM Post #172 of 198

sjb57

100+ Head-Fier
Joined
Dec 21, 2016
Posts
289
Likes
67
I agree with you on the Kinera - which is why I wouldn't personally recommend it. There are some question marks about whether the early review units were tuned differently - but I know the ones Crinnacle and I got measure almost identically, and he has similar opinion to mine on their rating. What would be good is getting one of the "5 star" review units and measuring it. Then we'd know for sure.

Unfortunately last time I suggested this with the ZhiYin QT5 there was at least one well known reviewer (external site - not Head-Fi) who quite neatly avoided taking us up on our offer of measuring his pair.

Sadly - I tend to agree with you on the "policy" decisions at this site. I can't fault them either - its not my business model. If I've got people talking about it though, and recognising there is an issue - then IMO we've taken the right step forward.
How about peer review prior to publication? That way, the author will get constructive challenge privately before public correction. It might be purely voluntary but a few crucial questions can teach a bright & willing learner a great deal if delivered kindly. Make sure the questions / challenges are agreed by >2/3 peers and nobody becomes de facto quality policeman. Those who care about quality will join.
 
Nov 5, 2017 at 5:02 PM Post #173 of 198
Joined
Jan 30, 2011
Posts
13,315
Likes
24,298
Personally I think that would be a great idea. Not sure how many would join a programme like that - and it would have to be mooted as Head-Fi policy, but definitely worth looking into. Even if it was just to check accuracy.
 
Nov 5, 2017 at 5:09 PM Post #174 of 198

ToroFiestaSol

500+ Head-Fier
Joined
Aug 11, 2015
Posts
774
Likes
532
First post in the thread is nice, we had a few discussions when you were a mod, but nothing personal.
I agree that we can call this forum "Ad-fi", it's not a good place for "correct" info, I use it to discover new products that can interest me (or not) or rant about something I don't like, also, it's very useful to improve my english.

About the "glowing" reviews to not upset manufacturers, this should be expressed in the review, like "I'm allowed to say only good things" or "I work for the manufacturer promoting this product", which, if specified, it's completely fair.
There's a very useful thread in another forum about this, which is called "...as if we didn't know Head-fi was the home of the shills..."

In the 3rd page, second post, there's some interesting legal info about this, I'll copy/paste it:

"I recommend familiarizing yourself with the FTC's 2009 guidelines on endorsements (attached hereto) and the recent guidance it disseminated regarding same. The FTC has made it very clear what the rules are and how to comply. I'll get you started. Example 7 concerning Section 255.5 seems particularly apt to me:

  • Example 7 (section 255.5 disclosure of material connections): A college student who has earned a reputation as a video game expert maintains a personal weblog or “blog” where he posts entries about his gaming experiences. Readers of his blog frequently seek his opinions about video game hardware and software. As it has done in the past, the manufacturer of a newly released video game system sends the student a free copy of the system and asks him to write about it on his blog. He tests the new gaming system and writes a favorable review. Because his review is disseminated via a form of consumer-generated media in which his relationship to the advertiser is not inherently obvious, readers are unlikely to know that he has received the video game system free of charge in exchange for his review of the product, and given the value of the video game system, this fact likely would materially affect the credibility they attach to his endorsement. Accordingly, the blogger should clearly and conspicuously disclose that he received the gaming system free of charge. The manufacturer should advise him at the time it provides the gaming system that this connection should bedisclosed, and it should have procedures in place to try to monitor his postings for compliance."
Writing a review "to not upset the manufacturer" without saying it, like it is an honest user review, it's illegal, at least in the USA.

Manufacturers (at least those who "do not want to be upset") need to do the best product they can, because when you're secure of what you do, you don't have nothing to hide, and don't need to use this kind of practices.
Audio would be a much better world that way...however, not possible, at least today.

Congrats for being honest, it's refreshing to read, and I'm glad you aren't a mod now, you were terrible lmao.
 
Last edited:
Nov 5, 2017 at 6:03 PM Post #176 of 198
Joined
Jan 30, 2011
Posts
13,315
Likes
24,298
First post in the thread is nice, we had a few discussions when you were a mod, but nothing personal.
I agree that we can call this forum "Ad-fi", it's not a good place for "correct" info, I use it to discover new products that can interest me (or not) or rant about something I don't like, also, it's very useful to improve my english.

About the "glowing" reviews to not upset manufacturers, this should be expressed in the review, like "I'm allowed to say only good things" or "I work for the manufacturer promoting this product", which, if specified, it's completely fair.
There's a very useful thread in another forum about this, which is called "...as if we didn't know Head-fi was the home of the shills..."

In the 3rd page, second post, there's some interesting legal info about this, I'll copy/paste it:

"I recommend familiarizing yourself with the FTC's 2009 guidelines on endorsements (attached hereto) and the recent guidance it disseminated regarding same. The FTC has made it very clear what the rules are and how to comply. I'll get you started. Example 7 concerning Section 255.5 seems particularly apt to me:

  • Example 7 (section 255.5 disclosure of material connections): A college student who has earned a reputation as a video game expert maintains a personal weblog or “blog” where he posts entries about his gaming experiences. Readers of his blog frequently seek his opinions about video game hardware and software. As it has done in the past, the manufacturer of a newly released video game system sends the student a free copy of the system and asks him to write about it on his blog. He tests the new gaming system and writes a favorable review. Because his review is disseminated via a form of consumer-generated media in which his relationship to the advertiser is not inherently obvious, readers are unlikely to know that he has received the video game system free of charge in exchange for his review of the product, and given the value of the video game system, this fact likely would materially affect the credibility they attach to his endorsement. Accordingly, the blogger should clearly and conspicuously disclose that he received the gaming system free of charge. The manufacturer should advise him at the time it provides the gaming system that this connection should bedisclosed, and it should have procedures in place to try to monitor his postings for compliance."
Writing a review "to not upset the manufacturer" without saying it, like it is an honest user review, it's illegal, at least in the USA.
Yep I know the rep the site has on other websites. Some of it is deserved, some of it isn't.

Manufacturers (at least those who "do not want to be upset") need to do the best product they can, because when you're secure of what you do, you don't have nothing to hide, and don't need to use this kind of practices.
Audio would be a much better world that way...however, not possible, at least today.
And the other side of the coin is that sometimes manufacturers simply get it wrong. By reviewing accurately you keep them in a feedback loop so they can improve. That's all I'm asking that we do collectively.

Congrats for being honest, it's refreshing to read, and I'm glad you aren't a mod now, you were terrible lmao.
Well its interesting that I've had far more compliments than critiques on the way I moderated. Most of the critiques come from people who don't think the rules are fair. The funny thing is that they associate the rules with the Moderator - when in fact we have nothing to do with them. All the Mods do is enforce the rules of the forum - nothing more, nothing less. So is it any wonder that the people who are most critical of the moderation, are also the ones who most often break the rules - and usually blatantly. Maybe I'll take the fact that you thought I was terrible as a compliment :wink:
 
Nov 5, 2017 at 6:57 PM Post #177 of 198

ToroFiestaSol

500+ Head-Fier
Joined
Aug 11, 2015
Posts
774
Likes
532
Well its interesting that I've had far more compliments than critiques on the way I moderated. Most of the critiques come from people who don't think the rules are fair. The funny thing is that they associate the rules with the Moderator - when in fact we have nothing to do with them. All the Mods do is enforce the rules of the forum - nothing more, nothing less. So is it any wonder that the people who are most critical of the moderation, are also the ones who most often break the rules - and usually blatantly. Maybe I'll take the fact that you thought I was terrible as a compliment :wink:

Yeah I know you only enforced the rules of the forum, don't worry, I get it 110%, it's just my dry humour :wink:

Now on a more serious note, there're a couple articles wrote by Jeff Fritz about this subject:

"An Ugly secret: When Reviews Are Not Really Reviews"
https://www.soundstageultra.com/ind...ly-secret-when-reviews-are-not-really-reviews

"The Failure To Compare"
https://www.soundstageultra.com/index.php/features-menu/opinion-menu/731-the-failure-to-compare

Very interesting reads
 
Nov 8, 2017 at 3:58 PM Post #179 of 198

Dobrescu George

Reviewer: AudiophileHeaven
Joined
Jan 30, 2011
Posts
15,105
Likes
7,722
Location
Bucharest, Romania
BTW my dudes, I tried something quite awesome today. Edifier Luna Eclipse - about 200$. Try them. Sounds awesome.

I think this thread has sparkled some things with reviewers, but at the end of the day, we just want to help people. At least most of us. Our methods might be different, but I think most of us, more established are quite good intended!
 
Nov 8, 2017 at 4:20 PM Post #180 of 198

castleofargh

Sound Science Forum Moderator
Joined
Jul 2, 2011
Posts
9,361
Likes
4,742
I've massively improved the objectivity of my reviews in 2017 by not posting any. the improvement is spectacular, I haven't misguided or lied to anybody. perfect truth telling score. yet, somehow I feel like there might be a tiny flaw in my strategy. I can't really put my finger on it.
 

Users who are viewing this thread

Top