Hi everyone, Lachlan here. As you hopefully know, I do product reviews. Many of the products I have reviewed on my channel have been provided by manufacturers as review units.
Right now I want to make the following argument based on my experience so far as an audio reviewer, and talking to other audio reviewers:
The current system whereby reviewers rely on manufacturers to provide review units produces bad incentives.
These bad incentives unconsciously change the behaviour of reviewers and create an industry wide, systemic bias towards positive reviews.
These biases hurt a community that needs open and transparent information to make decisions.
I am hoping this is a starting point for a community discussion about how we can improve things for everyone.
HOW I CAME TO THINK THIS?
Now to start, I want to make something clear. In the past I have received review units. Some of these review units have been provided on a temporary loan basis, some of these units I have been provided to keep on a permanent basis. I have tried to be very transparent about review units. For some time I have maintained a public Google spreadsheet so that at any point my audience could look up any of my reviews and determine under what basis I acquired a product. I have declared whenever I have received a discount from manufacturers on products, or been invited to manufacturer events, or in other ways received some kind of benefit.
I have been proud of my ethics and my approach, and I think I have conducted things in the most professional manner possible considering that I am just a guy with a camera who started a Youtube channel from his bedroom.
But after doing many reviews, and speaking to many fellow reviewers, I have come to the opinion that the current model for doing reviews is flawed and it produces harmful biases. This is not because of any large degree of dishonesty among reviewers and manufacturers, but because the entire system currently produces a set of incentives that are not aligned with the interests of consumers.
Now when we start talking about bias, people can get very angry and defensive. And to be clear, the kind of bias I am talking about is not shady back alley deals where manufacturers are asking for positive reviews in exchange for free products. I think that kind of gross corruption tends to be very obvious, and I think most manufacturers and reviewers know that making such arrangements could potentially hurt them far more than it would benefit them. Every manufacturer that I have worked with has been open to honest feedback and criticism about their products, though I have personally encountered some instances where there has been shady behaviour behind closed doors.
The problems that I am going to detail are not due to any ill intent among reviewers or manufacturers. The problems that I am talking about arise because we are human beings. I will be the first to admit that I am a human being and as such I am influenced by biases and incentives, and any honest (or modest) reviewer will tell you the same thing - that is, after all, the entire point of seeking subjective opinion from a reviewer in the first place. And despite the availability of graphs and measurements, the value of the reviewer will always serve as a person who gives consistent and easy to understand reference points based on long experience. As the old adage goes, everyone hears things differently.
Problems arise when the information that the reviewer provides is influenced by factors unknown or unclear to the audience, and this influence is directly opposed to the audience’s interest in deciding a product’s merit. Or simply put, problems arise when the interests of the reviewers are not aligned with the interests of the consumer. I think that there is a misalignment of interests in the industry and this is a result of poor incentives.
Human beings respond to incentives. Even Doctors who take the Hippocratic Oath to put their patients well being above all else have been found in studies to respond in subtle ways to external incentives besides optimal patient care. Just like any human being, they tend to become angry and defensive when this is suggested.
So in order to explain what incentives influence a reviewer, let’s break down how the review process works.
REQUESTING REVIEW UNITS
This is how I do my reviews. Individual reviewers may do things differently, but I imagine that this is pretty much the same for every reviewer in broad strokes:
I am interested in a product. I am personally curious about it or my audience has expressed interest in a product.
I send an email to a manufacturer requesting a review unit.
The manufacturer decides to provide a review unit either on a loan basis or a permanent ‘to keep’ basis.
I evaluate the product and ask the manufacturer any questions about the product that arise in the course of my evaluation
I perform a review and return the unit or keep it.
I keep in contact with the manufacturer to arrange future reviews. They may suggest other products to review and I may request other products.
Sometimes I receive unsolicited offers to send review units from manufacturers. This is rare, but because my Youtube channel is now visible enough, it happens every now and then.
Generally speaking, manufacturers do not tell reviewers what to say in reviews. They do not offer free products or payment in exchange for positive reviews. Sometimes they do not even ask that you review the product - they simply provide the review unit for your consideration and you are free to return it or keep it having done no review. A manufacturer that tells reviewers what to do and say is generally going to be a manufacturer that is not confident in their products, has inexperienced or unethical PR and most probably makes poor products anyway. I would not work with such manufacturers and I have not really encountered any like that.
But to say the manufacturer does not provide some kind of incentive to the reviewer with a review unit is false, because for a reviewer the review unit itself IS the incentive.
Why is a review unit a big benefit for a reviewer? Because a review unit means the reviewer can make new content, and new content attracts eyeballs, and new eyeballs mean some kind of payoff for the reviewer. This payoff could be in the form of ad revenue monetisation, like my Youtube channel. Or it could just be in the form of increased recognition from the community, which is why I make videos, since I like talking to people and I like to feel like I’m making a difference. This is something people should not forget, since emotional validation is a very strong payoff in and of itself.
This means that the more popular a product or a brand is, the more incentivised the reviewer is to get a review unit. There is an especially strong incentive to get a review unit early, since being the first on the scene means even more eyeballs.
Review units benefit reviewers because reviewers need review units to survive. This is a voracious market where the consumer always wants something new, something more, something now, even products from even a few years ago still sound perfectly good. The VSonic GR07 still sounds excellent no matter how many balanced armatures people try to cram into an earphone, as if they were running a razor blade company. While there are plenty of examples of products that stand the test of time, by and large people simply stop talking about old things because, you know, that’s sort of boring.
In this market reviewers are like sharks; they need to keep swimming or else they drown.
This also means that audio companies can be very selective about who they provide review units to. It’s different from very large and established low margin markets like computers or smartphones, where competition is extremely fierce and innovation is very fast, and where there are entrenched players like Engadget and The Verge have tremendous clout and editorial independence to get what they want. In audio reviews, particularly in high end, high margin audio, there are only a few big name reviewers. Personally, as a bit of a small fish, I’ve found trying to secure review units exhausting. Most of my emails don’t get a reply, and most that do are refusals.
Once you do secure a review unit from a manufacturer, another incentive kicks in: the incentive to preserve the relationship for future review units.
Again, let me repeat: manufacturers generally do not tell you to produce positive reviews, they do not (if they are a good manufacturer) criticise you for making negative reviews. But at any time they can always stop responding to emails and turn off the tap, or they can concentrate their efforts on other reviewers who they have better relationships with.
There is nothing sinister about this. Manufacturers are businesses, they have the right to pick and choose who they provide review units to, and they will make their decisions based on what benefits them. No one expects them to give products to a reviewer who they know is going to give them a negative review, because that would be silly and get someone fired.
So let me break down some of these incentives into a handy table.
Here are the incentives FOR and AGAINST a reviewer saying positive things about a product.
FOR saying positive things
AGAINST saying positive things
As you can see, there are MANY more incentives to say positive things about a product, but only ONE thing is something that the consumer actually cares about: "a product is good and merits recognition"
Now let’s look at the incentives FOR and AGAINST a reviewer saying negative things about a product.
FOR saying negative things
AGAINST saying negative things
Again, as you can see, there are MANY more incentives to NOT say negative things about a product, but most of these reasons do not particularly benefit a consumer that actually wants information about a product.
Of course, there are other incentives at play. I am not a psychologist or a behavioural economist. But because I run a Youtube channel I do have access to more data and information about how things work than your average punter, and this is what I have observed over time.
So, I’ve broken down some of the incentives that reviewers have. Reviewers have various incentives to produce negative reviews, and various incentives NOT to produce negative ones. But what kind of change in behaviours do these incentives produce for reviewers?
Based on my experience and my discussion with other reviewers, these incentives do not consciously influence reviewers to say positive things. Most reviewers would consider this unethical - but again, like Doctors, they are subject to unconscious bias.
Incentives DO produce certain subtle behaviours among reviewers.
Reviewers may moderate the negative things they may have to say about the product, or signpost the negativity by pointing out it is just a personal opinion.
Reviewers may avoid requesting review units for products they know or suspect they will not like.
Reviewers may avoid reviewing products that they have received and do not like.
Reviewers may avoid saying any other negative things about a company not related to the specific review unit.
What this means is that there is a subtle, industry wide bias towards saying positive things about a product and a brand. Reviewers who tend to say more positive things (within reason) are rewarded with increased exposure because they have access to more things to review. There is strong incentive to moderate negative opinions, but no real incentive to moderate positive ones.
Reviewers might not say this as self consciously as I have. Put in another way, it really isn’t interesting or fun to make negative reviews of products. It’s like being a big party pooper. But the net effect is that reviews tend to be more positive, and many bad products simply never get reviewed or heard of.
Looking at my list of product reviews, out of 62 headphone reviews I have given what I think are 28 positive reviews, 20 neutral / maybe reviews, and 14 negative reviews. Most of them were not review units, I’ve given negative reviews of review units, and I stand by my reviews (especially because I use such broad language, which is another bias in itself). But at the same time I have to question myself. Sometimes people make the joke about my channel that it should be called “Lachlan Likes Everything”. This isn’t true, and I always do point out that I have made negative reviews, but I have to ask if I am subject to a bias which modifies everything I do slightly?
I have the utmost respect for Tyll Hertsens of Innerfidelity as a fair and objective reviewer. I do not mean to single him out as an example of bad practice, but I wanted to point out that even a well respected figure is subject to human biases. This can be observed looking through the products that Tyll measures versus what he chooses to actually write about. He tends only to write about products that he likes, and avoids strongly negative language about any product which he doesn’t. This is of course natural - again there is very little incentive to write a negative review. It’s just not fun.
But Tyll has stated in a Reddit AMA:
"I try not to do negative reviews, I think it's much better to focus on finding new headphones that sound good."
I am not suggesting that Tyll is deliberately hiding anything or that he is dishonest. I am suggesting that he is responding to the same set of incentives all reviewers have when they are deciding what they will spend time and energy on to review. There is simply not enough time or resources to produce negative reviews when there are so many good reasons to produce positive ones.
Obviously this is problematic. If positive reviews are encouraged and negative reviews are suppressed, this distorts the perception of the market and allows manufacturers to escape scrutiny of bad reviews while enjoying the brand enhancement of good ones. Does it really serve the community when reviewers are not really incentivised to produce negative reviews?
RETAIL VS REVIEW
Another issue with review units is that the experience of receiving a review unit is entirely different from the experience of buying something in a retail channel. When you buy something in retail, you are forced (if you are diligent) to do market research, to determine the street price, to select the best vendor, to buy the product and observe the pre-sale and post-sale process, and to evaluate the long term reliability and utility of a product. You are forced to compare among many other options, and learn from the tips and advice of other people who have also gone through the usual retail route.
All of this is useful information to the consumer, but the reviewer has to do none of these things when a review unit turns up at the doorstep, and especially not when the review unit must be returned within a typically 30 day loan period.
Again, this introduces a bias where companies have less incentive to produce good service, since reviewers are generally not even looking at that part of the equation.
It also sustains a market for higher priced products, since reviewers who receive review units may not sufficiently take into account the relative market value of that product. This is highly obvious in reviews like those found on CNET where $1000+ products are regularly awarded 5 stars because the review throws their hands up and says "well I guess this is great if you can afford it." You can of course talk all about how prices are relative till you are blue in the face, but I assure you that if reviewers actually had to pay for more $1000 products, there would be less reviews of them, less visibility for them, and therefore less demand for them.
HURTING THE COMMUNITY
I would argue that it benefits the community when reviewers are transparent about their biases and they produce more information for the community, both positive and negative. I am a strong advocate for free markets, but free markets only work efficiently when people have access to more choices and good information. It takes effort and vigilance to provide that access. I feel that there is currently a bias in the audio industry towards producing positive reviews while disincentivising negative ones, and I think that in the end this hurts the community.
When most reviews tend to be positive, I think this also hurts the manufacturers. Companies do not receive negative feedback. Larger companies look better because they have a larger product portfolio, and thus tend to receive more positive reviews than negative ones. Smaller companies find it difficult to enter the market against the entrenched outfits that have the PR and the budget to give out review units to whomever they choose.
Many biases cannot be removed. They are human nature. As reviewers we have to sheepishly or proudly own up to our biases and let the audience decide whether or not we should be trusted.
In most ways, reviewers do not have any particular expertise or faculty beyond the average consumer. Actually, I would argue that in most ways, a single reviewer is less useful than looking at the aggregate ratings for a product from many “average consumer” reviews on Amazon.com.
What successful audio reviewers do have over the average consumer is the passion or inclination (or insanity) to try many many different types of gear, the ability to know what to look for based on this broad experience and the ability to articulate it to their audiences. The reviewers we tend to respect are the ones who have a broad experience with many different types of gear, and thus can act as a point of reference with which a consumer can triangulate their own likelihood of enjoying a given product.
That’s all there is to it. Good reviewers are good presenters who have managed to try a lot of stuff.
But the problem is, trying a lot of stuff is expensive.
So this brings us back to our original problem. Review units are great for reviewers, because they are FREE. Manufacturers benefit from sending out review units, because for them it is essentially free. The only people who do not benefit from this arrangement are consumers, who as I have said, are largely unaware or resigned to the incentives that are operating in the industry.
The issue is, even if we assume that the individual reviewers that we respect are able to control or admit their biases, the problem of bias is still systemic and it is embedded in the way the industry operates over time.
So in summary, here is the TLDR version of this entire statement.
If manufacturers choose who to allocate review units to, the reviewers who receive review units get increased visibility. If these reviewers tend, even slightly, to give more positive reviews because of this dependance, then this rewards the manufacturers that give out more review units, who in turn reward the reviewers who give more positive reviews. This issue cannot be solved by assuming that reviewers can control their own bias. This is a structural problem because a type of natural selection takes place where the most successful reviewers are the ones who are able to secure the most review units. This skews the available pool of knowledge for consumers towards positive reviews, and it means that manufacturers are not being exposed to sufficient scrutiny. The model of providing review units sustains a bias that is harmful to consumers.
When I first started doing reviews, I was very happy when I received my first review unit, because I took it as a sign of recognition and the idea that I had gone 'legit'. Many people around me congratulated me as well for the same reason. Now that I have done more reviews, I have come to feel that the exact opposite is true. Ordinary consumers do not benefit from the review unit model and they should not congratulate reviewers for having established the relationships necessary to secure them. For my part, I have decided to stop requesting new review units and to stop accepting them.
I am very interested to hear what you, the community, thinks could be alternative models. I have my own thoughts and I acknowledge my own self interest in this topic. But I do feel that this is a problem that needs to be discussed by the community, to produce a healthier market for everyone involved.
Over to you.
Edited by a_recording - 6/28/14 at 12:30am