Had some thoughts I wanted to share. Finally got around to writing them down.
The Bias and Subjectivity of Reviewing
Great to see that you (you personally and THL) are willing to get the discussion about these things going, as I think it is important. A while back I wrote this post in response to the question "
how do professional audio reviewers put food on the table and why should I trust them?":
"From all the reviewers I know, not two of them have the same motivation and the vast majority have regular jobs, they just put in a lot of hours after work (maybe sneak in a listen while working, or checking forums/emails on their phone).
Strictly for myself, I love writing and this hobby is something I got into out of an interest in using music as a way to help me manage my ADHD. Exploring the psychology and physiology of music, signatures, etc., is something I was doing anyway and it only seemed to make sense to write some of it down in my first review. After that I kept on writing because I enjoy it.
How do you stay objective while writing reviews? It is not really that difficult because honest feedback to the manufacturer is important for the manufacturer as well. Personally I do present negatives in a constructive, or sometimes funny way, but that is simply because I consider that the best way. I do not do hyperbole negative criticism because I do not consider it constructive. What then of a bad product? I simply don't waste my time on it and send it back where it came from.
It can be difficult to deal with manufacturers and I often enjoy doing reviews simply based on items I have borrowed from friends, or work with a brick and mortar store to help support them by using their demo units for reviews, but it is really helpful to get review samples for comparisons. What I do is something I know a few other reviewers do as well, and that is view the review samples as being on a long-term loan. Any manufacturer can at any time ask me for their samples back, no questions asked. Also really important is to stay sensible because some manufacturers do implicitly try to exert pressure. They can do that with me as much as they want, I don't care and they will only get my honest opinion. That's the deal. If they are not happy, then they can work with someone else. Those manufacturers that I do feel are trying to put on a little pressure are also put on notice in the back of my mind and I set clear boundaries they should not cross, if they do, I will send back their gear and refuse future reviews.
Those things have to do with personal integrity and that integrity is really important for a sustainable reviewing career/hobby. Equally, manufacturers who are dedicated to making quality products do appreciate a solid review including any criticism.
Like with anything else there are always going to be bad apples and there are always going to be people who make mistakes quite innocently, reviewing is not easy to do well and there is a learning curve. I always urge people to read several reviews and especially reviews by people with different styles because that can lead to interesting insights. Most importantly, never see a review as something truly "objective", it isn't. It is not a science and we can't make it scientific either because at the end of the day, music is a subjective experience and we can't sensibly separate the technology from the psychology." - [
Original post]
As for bias. I don't think a lot of people understand how to deal with it, or even how to spot it. Measurements for instance are often seen as "objective", but that is only because people don't understand experimental design and how uncertainty and bias slips into that. If the setup is not standardised, controlled and calibrated, then the results of one measurement to the next can't be readily compared. If any form of adjustment to the raw result is applied, then there is a chance of bias slipping into the end result, especially if results are made "presentable". The thing is however that people don't question graphs and numbers as easily as descriptive words because those graphs and numbers come across as if those exist independent of the person producing them. But that is not the case and instead those graphs and numbers depend on a wide range of factors directly and indirectly linked to the person producing them. Bias can thus be found everywhere.
My way of dealing with this (and it is only one option of many) is to write my impressions in such a way that people can verify/compare for themselves what I say. That means listing the entire chain of source, cable, IEMs and anything else that might potentially have an influence on the sound, as well as describing the music I listen to. That way any person reading my review can go out and listen to the same music with the same gear and decide for themselves whether or not they agree with me. In that way my reviews can be informative because they are descriptive, and avoid the pitfall of becoming prescriptive ("buy X because...", "Y is crap because...").
Another element I have introduced is transparency about the way I deal with manufacturers and review samples. It is a difficult balance because I can't share confidential information publicly and I won't risk damaging a business by openly declaring what brands I refuse to review. But in generalised terms I can at least share some information about my motivations, how I deal with review samples, how I deal with any perceived pressure from the industry or indeed a friendship with a certain manufacturer.
I think that by doing these things and trying to improve myself in doing them (I am not always as consistent as I would like to be), it can make a positive contribution to this hobby. And, hopefully, go a way to counter the negative impact of disingenuous review spammers.
(I often write during lunch hours, as it turns out...).
