All electronic components in a device generate some level of unwanted noise when it is in operation (as physic tells us that energy is always lost since no system is perfect, and some of the energy lost in a circuit becomes noise). Noise is basically whatever signal that is not part of the original / intended signal and is added by the circuit itself. Typically in the old days, loudspeaker system can pass with a 10% distortion / noise test. On most personal audio devices these days, general consensus is that you need to be 1% or less (about -40dB) to claim to be 'Hi Fi' (as 1% is generally believed to be the threshold of obvious detection on noise/distortion). But for true audiophiles equipment, we usually looking for 0.0x% for minimum or even 0.00x% (-60 to -100 dB range) as norm. Simply put, we want the source to reproduce the music as faithfully as possible.
I am personally on the school that some coloration can be beneficial, but only on a limited scope. Any source device should have it's basic engineering done correctly, then whatever fine-tuning / coloration can take place from that. Coloration is NOT and should NEVER be allowed to be a free pass for bad engineering.
As for how a person's ear can lie and how a reviewer might not even know he /she is biased, here are two examples of mine: First, Shanling UP4 - early reviews and impressions all praised how good it was, but my first two week of listening told me something was way off. I actually ended up building some special cable rig just to measure it and the result was that it has really bad distortion and phase mismatch. I reported the issue here in this thread, Shanling took notice and released a new firmware to address the issue. While UP4 never did measure as great as it's competition, at least it was decent enough after firmware update. That bakes the question of how those early reviewers could have missed distortion and phase mismatch so bad and praised UP4 for it? My guess is that either they never did listen to UP4 before writing or they simply have really crappy hearing. Either way, that's not good for those who are misled by their review. Will however give credit to Shanling as they learned from their mistake and UP5 has really good measurement. Second example is iBasso CF01 - I immediately noticed it has weak output and caused severe coloration to my multi-driver IEM. Again, made a special cable rig to measure it and turned out it has 20 ohm of output impedance (and for reference, any thing that designed to drive multi-driver IEM, such as CF01, should have 1 ohm or less in output impedance in order to avoid any coloration). Almost all the reviews I found on CF01 at the time just praised how great CF01 was, but none mentions it has super weak output power (probably in single digit mW range) and it completely change the FR curve on most if not all multi-driver IEM. Surprisingly no one noticed any of that when it only took me less than 10 minutes to feel something is way off. After my reporting in the forum, iBasso rep actually had the audacity to refute my claim and said my measurement was wrong as they CF01 has only 1 ohm of output impedance and would provide proof. Well, they never did - though I got confirmation from another CF01 owner that his CF01 measured the same as mine.
Measurement is not the end of everything, but it does help us - including users, reviewers, manufacturers and the community as a whole, keep a heathy and honest opinion to our hobby. For me personally, I always take reviews with big grain of salt.
