[a_recording]

Hi everyone, Lachlan here. As you hopefully know, I do product reviews. Many of the products I have reviewed on my channel have been provided by manufacturers as review units.  

 

Right now I want to make the following argument based on my experience so far as an audio reviewer, and talking to other audio reviewers:

 

• The current system whereby reviewers rely on manufacturers to provide review units produces bad incentives.
  
• These bad incentives unconsciously change the behaviour of reviewers and create an industry wide, systemic bias towards positive reviews.
  
• These biases hurt a community that needs open and transparent information to make decisions.
   
  

I am hoping this is a starting point for a community discussion about how we can improve things for everyone.

 

//////////////////////////////////////////////////////////////////////////////////////////////

 

HOW I CAME TO THINK THIS?

 

Now to start, I want to make something clear. In the past I have received review units. Some of these review units have been provided on a temporary loan basis, some of these units I have been provided to keep on a permanent basis. I have tried to be very transparent about review units. For some time I have maintained a public Google spreadsheet so that at any point my audience could look up any of my reviews and determine under what basis I acquired a product. I have declared whenever I have received a discount from manufacturers on products, or been invited to manufacturer events, or in other ways received some kind of benefit.

 

I have been proud of my ethics and my approach, and I think I have conducted things in the most professional manner possible considering that I am just a guy with a camera who started a Youtube channel from his bedroom.

 

But after doing many reviews, and speaking to many fellow reviewers, I have come to the opinion that the current model for doing reviews is flawed and it produces harmful biases. This is not because of any large degree of dishonesty among reviewers and manufacturers, but because the entire system currently produces a set of incentives that are not aligned with the interests of consumers.

 

Now when we start talking about bias, people can get very angry and defensive. And to be clear, the kind of bias I am talking about is not shady back alley deals where manufacturers are asking for positive reviews in exchange for free products. I think that kind of gross corruption tends to be very obvious, and I think most manufacturers and reviewers know that making such arrangements could potentially hurt them far more than it would benefit them. Every manufacturer that I have worked with has been open to honest feedback and criticism about their products, though I have personally encountered some instances where there has been shady behaviour behind closed doors.

 

The problems that I am going to detail are not due to any ill intent among reviewers or manufacturers. The problems that I am talking about arise because we are human beings. I will be the first to admit that I am a human being and as such I am influenced by biases and incentives, and any honest (or modest) reviewer will tell you the same thing - that is, after all, the entire point of seeking subjective opinion from a reviewer in the first place. And despite the availability of graphs and measurements, the value of the reviewer will always serve as a person who gives consistent and easy to understand reference points based on long experience. As the old adage goes, everyone hears things differently.

 

Problems arise when the information that the reviewer provides is influenced by factors unknown or unclear to the audience, and this influence is directly opposed to the audience’s interest in deciding a product’s merit.  Or simply put, problems arise when the interests of the reviewers are not aligned with the interests of the consumer. I think that there is a misalignment of interests in the industry and this is a result of poor incentives.

 

Human beings respond to incentives. Even Doctors who take the Hippocratic Oath to put their patients well being above all else have been found in studies to respond in subtle ways to external incentives besides optimal patient care. Just like any human being, they tend to become angry and defensive when this is suggested.

 

So in order to explain what incentives influence a reviewer, let’s break down how the review process works.

 

REQUESTING REVIEW UNITS

 

This is how I do my reviews. Individual reviewers may do things differently, but I imagine that this is pretty much the same for every reviewer in broad strokes:

 

• I am interested in a product. I am personally curious about it or my audience has expressed interest in a product.
  
• I send an email to a manufacturer requesting a review unit.
  
• The manufacturer decides to provide a review unit either on a loan basis or a permanent ‘to keep’ basis.
  
• I evaluate the product and ask the manufacturer any questions about the product that arise in the course of my evaluation
  
• I perform a review and return the unit or keep it.
  
• I keep in contact with the manufacturer to arrange future reviews. They may suggest other products to review and I may request other products.
  

 

Sometimes I receive unsolicited offers to send review units from manufacturers. This is rare, but because my Youtube channel is now visible enough, it happens every now and then.

 

Generally speaking, manufacturers do not tell reviewers what to say in reviews. They do not offer free products or payment in exchange for positive reviews. Sometimes they do not even ask that you review the product - they simply provide the review unit for your consideration and you are free to return it or keep it having done no review. A manufacturer that tells reviewers what to do and say is generally going to be a manufacturer that is not confident in their products, has inexperienced or unethical PR and most probably makes poor products anyway. I would not work with such manufacturers and I have not really encountered any like that.

 

But to say the manufacturer does not provide some kind of incentive to the reviewer with a review unit is false, because for a reviewer the review unit itself IS the incentive.

 

//////////////////////////////////////////////////////////////////////////////////////////////

 

REVIEWER’S INCENTIVES

 

Why is a review unit a big benefit for a reviewer? Because a review unit means the reviewer can make new content, and new content attracts eyeballs, and new eyeballs mean some kind of payoff for the reviewer. This payoff could be in the form of ad revenue monetisation, like my Youtube channel. Or  it could just be in the form of increased recognition from the community, which is why I make videos, since I like talking to people and I like to feel like I’m making a difference. This is something people should not forget, since emotional validation is a very strong payoff in and of itself.

 

This means that the more popular a product or a brand is, the more incentivised the reviewer is to get a review unit. There is an especially strong incentive to get a review unit early, since being the first on the scene means even more eyeballs.

 

Review units benefit reviewers because reviewers need review units to survive. This is a voracious market where the consumer always wants something new, something more, something now, even products from even a few years ago still sound perfectly good. The VSonic GR07 still sounds excellent no matter how many balanced armatures people try to cram into an earphone, as if they were running a razor blade company. While there are plenty of examples of products that stand the test of time, by and large people simply stop talking about old things because, you know, that’s sort of boring.

 

In this market reviewers are like sharks; they need to keep swimming or else they drown.

 

This also means that audio companies can be very selective about who they provide review units to. It’s different from very large and established low margin markets like computers or smartphones, where competition is extremely fierce and innovation is very fast, and where there are entrenched players like Engadget and The Verge have tremendous clout and editorial independence to get what they want. In audio reviews, particularly in high end, high margin audio, there are only a few big name reviewers. Personally, as a bit of a small fish, I’ve found trying to secure review units exhausting. Most of my emails don’t get a reply, and most that do are refusals.

 

Once you do secure a review unit from a manufacturer, another incentive kicks in: the incentive to preserve the relationship for future review units.

 

Again, let me repeat: manufacturers generally do not tell you to produce positive reviews, they do not (if they are a good manufacturer) criticise you for making negative reviews. But at any time they can always stop responding to emails and turn off the tap, or they can concentrate their efforts on other reviewers who they have better relationships with.

 

There is nothing sinister about this. Manufacturers are businesses, they have the right to pick and choose who they provide review units to, and they will make their decisions based on what benefits them. No one expects them to give products to a reviewer who they know is going to give them a negative review, because that would be silly and get someone fired.

 

So let me break down some of these incentives into a handy table.

 

Here are the incentives FOR and AGAINST a reviewer saying positive things about a product.

 

FOR saying positive things

AGAINST saying positive things

You think a product is good and merits recognition. It feels good to say positive things. People who own the product and like it will also like you. A large number of reviews are read by people who are not necessarily looking for information, but rather affirmation about their purchase. I have learnt over time that people (myself included) like reviews as a form of informational entertainment. People in general like to hear positive things, so they will like content that is positive. This has an effect on the popularity of your reviews. The manufacturers like you and will give you more review units. The manufacturer may promote your review through their channels and give you more exposure. Review units help the future success of your publication. If the product is new, your positive review has a strong influence on community uptake. A snowball / flavour of the month effect occurs and as the reviewer you again stand to benefit.

If you do it too much or it’s unwarranted, people will think you are a shill and stop trusting you.

 

As you can see, there are MANY more incentives to say positive things about a product, but only ONE thing is something that the consumer actually cares about: "a product is good and merits recognition"

 

Now let’s look at the incentives FOR and AGAINST a reviewer saying negative things about a product.

 

FOR saying negative things

AGAINST saying negative things

You think a product is bad and this needs to be mentioned. People will respect your honest opinion. The manufacturer may use this feedback to improve the product.

It feels bad to say negative things. If you do it too much or it’s unwarranted, people will think you are a troll and stop trusting you. You are exposed to criticism from people who like the product and disagree. You will make people disappointed or angry. You risk jeopardising the relationship with the manufacturer. The manufacturer may not promote your review. You may stop receiving review units. If the product is new, your negative review may kill any initial enthusiasm and doom the product.

 

Again, as you can see, there are MANY more incentives to NOT say negative things about a product, but most of these reasons do not particularly benefit a consumer that actually wants information about a product.

 

Of course, there are other incentives at play. I am not a psychologist or a behavioural economist. But because I run a Youtube channel I do have access to more data and information about how things work than your average punter, and this is what I have observed over time.

 

//////////////////////////////////////////////////////////////////////////////////////////////

 

SYSTEMIC BIAS

 

So, I’ve broken down some of the incentives that reviewers have. Reviewers have various incentives to produce negative reviews, and various incentives NOT to produce negative ones. But what kind of change in behaviours do these incentives produce for reviewers?

 

Based on my experience and my discussion with other reviewers, these incentives do not consciously influence reviewers to say positive things. Most reviewers would consider this unethical - but again, like Doctors, they are subject to unconscious bias.

 

Incentives DO produce certain subtle behaviours among reviewers.

 

1. Reviewers may moderate the negative things they may have to say about the product, or signpost the negativity by pointing out it is just a personal opinion.
   
2. Reviewers may avoid requesting review units for products they know or suspect they will not like.
   
3. Reviewers may avoid reviewing products that they have received and do not like.
   
4. Reviewers may avoid saying any other negative things about a company not related to the specific review unit.
   

 

What this means is that there is a subtle, industry wide bias towards saying positive things about a product and a brand. Reviewers who tend to say more positive things (within reason) are rewarded with increased exposure because they have access to more things to review. There is strong incentive to moderate negative opinions, but no real incentive to moderate positive ones.

 

Reviewers might not say this as self consciously as I have. Put in another way, it really isn’t interesting or fun to make negative reviews of products. It’s like being a big party pooper. But the net effect is that reviews tend to be more positive, and many bad products simply never get reviewed or heard of.

 

Looking at my list of product reviews, out of 62 headphone reviews I have given what I think are 28 positive reviews, 20 neutral / maybe reviews, and 14 negative reviews. Most of them were not review units, I’ve given negative reviews of review units, and I stand by my reviews (especially because I use such broad language, which is another bias in itself). But at the same time I have to question myself. Sometimes people make the joke about my channel that it should be called “Lachlan Likes Everything”. This isn’t true, and I always do point out that I have made negative reviews, but I have to ask if I am subject to a bias which modifies everything I do slightly?

 

I have the utmost respect for Tyll Hertsens of Innerfidelity as a fair and objective reviewer. I do not mean to single him out as an example of bad practice, but I wanted to point out that even a well respected figure is subject to human biases. This can be observed looking through the products that Tyll measures versus what he chooses to actually write about. He tends only to write about products that he likes, and avoids strongly negative language about any product which he doesn’t. This is of course natural - again there is very little incentive to write a negative review. It’s just not fun.

 

But Tyll has stated in a Reddit AMA:

 

 

"I try not to do negative reviews, I think it's much better to focus on finding new headphones that sound good."

 

 

I am not suggesting that Tyll is deliberately hiding anything or that he is dishonest. I am suggesting that he is responding to the same set of incentives all reviewers have when they are deciding what they will spend time and energy on to review. There is simply not enough time or resources to produce negative reviews when there are so many good reasons to produce positive ones.

 

Obviously this is problematic. If positive reviews are encouraged and negative reviews are suppressed, this distorts the perception of the market and allows manufacturers to escape scrutiny of bad reviews while enjoying the brand enhancement of good ones. Does it really serve the community when reviewers are not really incentivised to produce negative reviews?

 

//////////////////////////////////////////////////////////////////////////////////////////////

 

RETAIL VS REVIEW

 

Another issue with review units is that the experience of receiving a review unit is entirely different from the experience of buying something in a retail channel. When you buy something in retail, you are forced (if you are diligent) to do market research, to determine the street price, to select the best vendor, to buy the product and observe the pre-sale and post-sale process, and to evaluate the long term reliability and utility of a product. You are forced to compare among many other options, and learn from the tips and advice of other people who have also gone through the usual retail route.

 

All of this is useful information to the consumer, but the reviewer has to do none of these things when a review unit turns up at the doorstep, and especially not when the review unit must be returned within a typically 30 day loan period.

 

Again, this introduces a bias where companies have less incentive to produce good service, since reviewers are generally not even looking at that part of the equation.
 
It also sustains a market for higher priced products, since reviewers who receive review units may not sufficiently take into account the relative market value of that product. This is highly obvious in reviews like those found on CNET where $1000+ products are regularly awarded 5 stars because the review throws their hands up and says "well I guess this is great if you can afford it." You can of course talk all about how prices are relative till you are blue in the face, but I assure you that if reviewers actually had to pay for more $1000 products, there would be less reviews of them, less visibility for them, and therefore less demand for them.
 

//////////////////////////////////////////////////////////////////////////////////////////////

 

HURTING THE COMMUNITY

 

I would argue that it benefits the community when reviewers are transparent about their biases and they produce more information for the community, both positive and negative. I am a strong advocate for free markets, but free markets only work efficiently when people have access to more choices and good information. It takes effort and vigilance to provide that access. I feel that there is currently a bias in the audio industry towards producing positive reviews while disincentivising negative ones, and I think that in the end this hurts the community.

 

When most reviews tend to be positive, I think this also hurts the manufacturers. Companies do not receive negative feedback. Larger companies look better because they have a larger product portfolio, and thus tend to receive more positive reviews than negative ones. Smaller companies find it difficult to enter the market against the entrenched outfits that have the PR and the budget to give out review units to whomever they choose.

 

Many biases cannot be removed. They are human nature. As reviewers we have to sheepishly or proudly own up to our biases and let the audience decide whether or not we should be trusted.

 

In most ways, reviewers do not have any particular expertise or faculty beyond the average consumer. Actually, I would argue that in most ways, a single reviewer is less useful than looking at the aggregate ratings for a product from many “average consumer” reviews on Amazon.com.

 

What successful audio reviewers do have over the average consumer is the passion or inclination (or insanity) to try many many different types of gear, the ability to know what to look for based on this broad experience and the ability to articulate it to their audiences. The reviewers we tend to respect are the ones who have a broad experience with many different types of gear, and thus can act as a point of reference with which a consumer can triangulate their own likelihood of enjoying a given product.

 

That’s all there is to it. Good reviewers are good presenters who have managed to try a lot of stuff.

 

But the problem is, trying a lot of stuff is expensive.

 

So this brings us back to our original problem. Review units are great for reviewers, because they are FREE. Manufacturers benefit from sending out review units, because for them it is essentially free. The only people who do not benefit from this arrangement are consumers, who as I have said, are largely unaware or resigned to the incentives that are operating in the industry.

 

The issue is, even if we assume that the individual reviewers that we respect are able to control or admit their biases, the problem of bias is still systemic and it is embedded in the way the industry operates over time. 
 
So in summary, here is the TLDR version of this entire statement.
 
If manufacturers choose who to allocate review units to, the reviewers who receive review units get increased visibility. If these reviewers tend, even slightly, to give more positive reviews because of this dependance, then this rewards the manufacturers that give out more review units, who in turn reward the reviewers who give more positive reviews. This issue cannot be solved by assuming that reviewers can control their own bias. This is a structural problem because a type of natural selection takes place where the most successful reviewers are the ones who are able to secure the most review units. This skews the available pool of knowledge for consumers towards positive reviews, and it means that manufacturers are not being exposed to sufficient scrutiny. The model of providing review units sustains a bias that is harmful to consumers.
 
When I first started doing reviews, I was very happy when I received my first review unit, because I took it as a sign of recognition and the idea that I had gone 'legit'. Many people around me congratulated me as well for the same reason. Now that I have done more reviews, I have come to feel that the exact opposite is true. Ordinary consumers do not benefit from the review unit model and they should not congratulate reviewers for having established the relationships necessary to secure them. For my part, I have decided to stop requesting new review units and to stop accepting them.
 
I am very interested to hear what you, the community, thinks could be alternative models. I have my own thoughts and I acknowledge my own self interest in this topic. But I do feel that this is a problem that needs to be discussed by the community, to produce a healthier market for everyone involved.
 
Over to you.
 

 

[gikigill]

Great post but then again no one ever claimed audio to be sane.
 
To paraphrase, have faith in God but trust only your ears.

 

[a_recording]

 
To paraphrase, have faith in God but trust only your ears.

 
Well, if we did that, it'd be awfully pointless to have a forum for sharing opinions wouldn't it?

 

[drez]

Agree wholeheartedly a_rec. However the review unit phenomenon is not the only issue here. I know a few reviewers who receive review units yet provide balanced reviews and compare against other products. By far the more common evil is hype style reviews where new toy syndrome is to blame, or reviews which have opinions on absolute timbre, presentation or tonality, or even more common impressions that make lists or just say "X is far better than Y". Add to that reviews which attempt to copy the discursive flowery magazine style of reviews and spend half the time giving a bio of their favourite bass player.

 

[wuwhere]

I used to read reviews from publications. Not anymore. I like the informality of Head-Fiers, general opinions, one-liners, raw. In fact I bought my last headphone based on general opinions here, not from any reviewer's review. And I'm glad I did. Early on I tended to follow reviewer's thinking. Some reviewers tend to have a review system and a music enjoyment system. The review system is just that, a microscope, analytical, a high resolution tool for evaluating a piece of hardware. Then plug that equipment into the music enjoyment system to see if it is enjoyable or not.
 
On the same headphone, this is my favorite one-liner:
 
"Its rolled-off in the highs or/and lows".
 
"Its highs or/and lows are natural".
 
"Natural"? What the hell does "Natural" mean? Give me a break!!! I would believe the roll-off guys.

 

[eke2k6]

My view is similar to yours, but far more cynical.
 
I don't just think it's a matter of review units, but of money being involved. People want to defend their purchases...we want to make ourselves believe that whatever we just spent our cash on is amazing. I catch myself doing it as well.
 
Example...I drive a Lexus that is older than I am. It outputs about 160hp after a good tune up. When I'm at a stop light and a 2014 Camaro pulls up beside me, you can bet your life that I'm going to floor it and beat the Camaro off the line to make myself feel better about my car. 
 
The same principle applies in audio. Review units and audio purchases are essentially currency, especially the purchases. Anything I buy is got with the mentality of maximum resale value, so I can offload it for minimum loss of necessary. In the same way, review units can be tests sometimes. If a review is favorable enough, the manufacturer will often send more and more to the reviewer. So, the reviewer will minimize negatives, or set parameters that skew the results in a way that make the star product shine. That, and they have seemingly condescending replies when their reviews are challenged. 
 
 
Here is a more concise summary...you can choose to ignore the product mentioned.
 

  Quote:

   
That would be true under most circumstances, but not in this echo chamber of a hobby. Allow me to dust off my crystal ball and describe what will take place over the next few months...if the AK240 is not clearly above the competition (and I believe MUST, given that it costs double the price of its competitors).
 
1) Someone will buy an AK240. Likewise, others will receive a sample AK240.
 
2) Said individuals will defend their new purchase, despite how they feel about its performance. Likewise, the selected reviewers will want to talk about the product in a way that ensures they get more in the future.
 
3) People who read these glowing reviews will be convinced to buy these DAPs. Most will, again, try to defend their purchase. Only a few will speak out, but those few will be silenced. I've seen this happen before several times. I remember a certain IEM that had a wonky sounding midrange to a lot of folks. But one member shouted from the top of a rooftop that these iems had no such issue at all. Later, he confided to me via PM that he was using fabric filters to get the IEM to sound normal. This is what the consumer is up against.
 
4) A&K's strategy will have worked, leading to the sale of units for them, but also the impetus to continue with price increases.
 
5) More DAP makers will see this and want in on the action. I predict more $2k+ DAPs within the next 6-12 months.

 

 
 
I call it Emperor's New Clothes syndrome.

 

[Magick Man]

Not too new, really, this has been a common complaint in the audio community for half a century. The problem is, not everyone has unlimited funds to buy everything and test it, not even the larger publications (much less the little guys who are simply trying to make a basic living). So they rely on samples, even those that aren't giveaways, to get a good idea what's out there and how it all compares.

Is there bias? Sure there is. Can you sniff it out with a little due diligence? Yep. There are some people I know who try to do their best to be as forthright as possible, and others whom you can tell don't care and literally say the same things over and over again, shilling for free samples. As for Tyll's silence about certain cans, to me it's deafening. If he won't even review them, then you know they're seriously flawed, and you can go look at his measurements and get a good idea why in a matter of seconds.

Perhaps the fix is to not have any giveaway samples anymore, or severely limit them. But that will only hurt comparison testing with honest reviewers and degrade content from people who love to contribute to the hobby and don't have bottomless pockets.

 

[a_recording]

  My view is similar to yours, but far more cynical.
 
I don't just think it's a matter of review units, but of money being involved. People want to defend their purchases...we want to make ourselves believe that whatever we just spent our cash on is amazing. I catch myself doing it as well.
 
Example...I drive a Lexus that is older than I am. It outputs about 160hp after a good tune up. When I'm at a stop light and a 2014 Camaro pulls up beside me, you can bet your life that I'm going to floor it and beat the Camaro off the line to make myself feel better about my car. 

 
Sadly I believe that kind of bias is always going to be present and impossible to eliminate, and it really seems to depend on the reviewers willingness to admit that they may be wrong or they may have made a mistake. I've gone into many threads on Head Fi with negative opinions on products that were in the middle of a hype cycle and just faced outright hostility to the point of absurdity. People seem to feel personally attacked when their buying purchases are questioned, which is silly but sustainable in an industry when people can always fall back on subjectivity as a pseudo defence.
 
At the very least money introduces the potentially not only be incredibly pleased with a product, but also incredibly disappointed. 
 

Not too new, really, this has been a common complaint in the audio community for half a century. The problem is, not everyone has unlimited funds to buy everything and test it, not even the larger publications (much less the little guys who are simply trying to make a basic living). So they rely on samples, even those that aren't giveaways, to get a good idea what's out there and how it all compares.

Is there bias? Sure there is. Can you sniff it out with a little due diligence? Yep. There are some people I know who try to do their best to be as forthright as possible, and others whom you can tell don't care and literally say the same things over and over again, shilling for free samples. As for Tyll's silence about certain cans, to me it's deafening. If he won't even review them, then you know they're seriously flawed, and you can go look at his measurements and get a good idea why in a matter of seconds.

Perhaps the fix is to not have any giveaway samples anymore, or severely limit them. But that will only hurt comparison testing with honest reviewers and degrade content from people who love to contribute to the hobby and don't have bottomless pockets.

 
I don't think that we can eliminate review units, because there are some people who are always going to give them out and some people who are always going to accept them. And I think skewing the pool of reviewers only towards those who are wealthy enough to afford to review things also introduces its own sets of biases.
 
I think the problem is that even if you can sniff about bias from individual reviewers, review units skew the aggregate visibility of certain products so that the market favours manufacturers can send out the most review units. Over time this means that companies are not necessarily going to be rewarded so much for making good products as they are for having well organised PR, and conversely reviewers are rewarded not for so much for producing good reviews as much as they are rewarded for securing more review units.
 
This kind of thing is obviously almost impossible to see at a low level, especially for experienced members of the hobby who have twigged onto things and only follow a certain set of reviewers, but for the vast majority of consumers out there I think this skew probably hurts them.
 
I just think the community attitude towards review units needs to change so that reviewers are subject to much much greater scrutiny about them. On the flip side the community might want to encourage alternate methods for obtaining review units. Borrowing them from friends for instance, or doing things like the times Head Fi has gotten together to buy Joker a custom IEM. That kind of thing already happens, but what I am suggesting is that these shouldn't be considered a special thing, they should be considered a natural things that the community does to ensure a healthier review system. Unfortunately that culture doesn't exist yet and part of the point of this post is to get people thinking about it.
 
One concern I have about this is that when you don't buy a unit in retail, you end up missing out a lot of information and research about a company. You don't have to experience their pre sales and post sales service, there is no real long term tests of product durability if you borrow products, and things like relative value are a bit harder to assess without having research street prices. With sufficient diligence these issues can probably be dealt with.

 

[TwinQY]

I emphatically agree with everything in this post and I'm going to end up repeating a lot of what has been said.
 
On the topic of alternative models, I think the best alternative that you mentioned, are loans between community members. The roadblock on this model arises from the practical concerns in regards to trust, which is why it might always be stuck between tight-knit social circles. Trust is something that is hard to deal with as a community, because there will always be a bad one out of the bunch that end up ruining the experience.
 
However, we inevitably do end up having to trust one another out of mutual benefit, this is how society works. More than 10,000 years and I think we have gotten to where we can simply accept the reality that bad things happen and we will have to deal with it one way or another.
 
In the end I think many would advocate for a more open and communal culture, as you say.

 

[a_recording]

I see references to the idea that some reviewers are "honest" and some are not. I think I should repeat, the bias that I am suggesting that is introduced by review units is probably subconscious. I'll quote something that someone said on this same topic when I posted it on Reddit:
 

The culture of asking for review units does introduce bias, whether it is consciously perceived or not. The closest analogy is the pharmaceutical industry. Drug reps give freebies to doctors (pens, Post it pads, invitations for dinner + a talk, etc.) with the unwritten expectation that in some way it will influence their prescribing habits. The medical profession has gotten away from this, not just by having docs disclose their relationships with drug companies (for example at the beginning of every talk) but many healthcare organizations now ban drug reps from directly contacting their employed doctors altogether. (my emphasis)

 
You would think that a simple freebie like a free pen wouldn't make any difference to a doctor's prescribing habits, but then the following NY Times article points out that:
 

The restrictions [on promotional products] come as a blow to the makers and distributors of promotional products, an industry with an annual turnover of about $19 billion, according to Promotional Products Association International, a trade group. Such companies, accustomed to orders of up to a million pens a drug, stand to lose around $1 billion a year in sales as a result of the drug industry’s voluntary ban, the group said.
http://www.nytimes.com/2008/12/31/business/31drug.html?_r=0

 
The pharmaceutical industry spends a billion dollars on pens and trinkets alone? They must be doing that for a reason.
 
And indeed they are, because an empirical study in 2000 concluded that these promotional interactions do have an effect.
 

The present extent of physician-industry interactions appears to affect prescribing and professional behavior and should be further addressed at the level of policy and education.
http://www.ncbi.nlm.nih.gov/pubmed/10647801

 
Now you may say that the extent of the bias may be very small, or that perhaps reviewing audio products isn't as serious as say, prescribing medicine. But I do think as a reviewer I should be eliminating potential sources of bias whenever possible, and if that means eschewing review units to avoid subconscious bias I think that is worthwhile.

 

[Hutnicks]

The really key point here is that reviewers are all in competition with each other. If you were not you would all simply create a small consortium to buy samples, and ship them around for review.
Reviewing has become an industry of its own over the years and as such, suffers the biases and foibles of any other competitive industry.
 
If we were to get clear unbiased reviews it would have to be done anonymously and treated as a blind exercise. Manufacturers are not going to go for that, they want a "Name" behind any published opinion of their stuff, as do end users.
 
You need the audio equivalent of "The Stig" where the anonymity is of utmost importance.

 

[billybob_jcv]

This has been the situation and a known issue FAR before the internet. Every possible magazine in every possible consumer goods market has been accused of bias and bribery due to the availability of free review samples. Car & Driver, Motor Trend, Road & Track, Byte Magazine, PC Magazine, Stereophile, Guns & Ammo, etc, etc, etc. Caveman Ogg was probably accused of bias for accepting a review sample of a new flint axe!

Consumer Reports was created as an attempt to eliminate this exact type of bias. Consumers Union was founded in 1936!

 

[RudeWolf]

Why, easy! If you are are a renowned expert in reviewing audio devices the manufacturer pays for someone like you reviewing their product. Then again - what kind of expertise is required from an audio (or almost any consumer product for that matter) reviewer? Can the average reviewer look at a circuit used and give an informed and theoretically grounded opinion on what compromises the engineering solution might have? Probable performance issues when interacting with other adjacent devices or when operating under certain circumstances?
 
The problem is that the entry point into reviewing is laughable. You have to have enough money or have the right connections (best - both). Then you give out an opinion, tell your readers what's it like to use the device in question. The question then is why a certain device operates in a good or bad fashion. What has been done when designing it to make you happy or disappointed with it. Maybe I can look for these properties in other similar devices or what's more important - what should I avoid!
 
Otherwise just about any every "review" can be dismissed by being "just like your opinion, man".

 

[wuwhere]

Like I said before, reviewing is comparative. You must have a reference, be it the SR009, LCD3, what have you. So you have to have a reference system to compare the unit under review. And perhaps not just one reference system. That gets very expensive quickly. As a consumer, my decision would be based on how close is this product to the reference without spending a lot of money. If you read comments after comments here, somebody would ask how does it compare to the best product that cost more than the one under review? Let's be realistic. This is just a hobby. I only have so much disposable income for it. I'm sure most of us are in the same boat.

 

[anetode]

Come on in, dear boy, have a cigar, you're gonna go far...