[xnor]

Let's collect and discuss some evidence that show how subjective, sighted and (therefore) usually biased reviews fail.
 
Note: This thread is not about intolerance or ridicule towards subjectivity per se. See #51.
The following is in no particular order. The reviews are chosen completely randomly.
 
 

Sighted listening tests

http://seanolive.blogspot.co.at/2009/04/dishonesty-of-sighted-audio-product.html ("Audio Musings by Sean Olive: The Dishonesty of Sighted Listening Tests")

[...] if you want to obtain an accurate and reliable measure of how the audio product truly sounds, the listening test must be done blind.
It’s time the audio industry grow up and acknowledge this fact [...]

 
See also:
http://seanolive.blogspot.co.at/2010/12/how-to-listen-course-on-how-to.html
http://seanolive.blogspot.co.at/2012/11/behind-harmans-testing-lab.html
 
 

Bit-perfect audiophile audio players (software)

Reviews/claims:
most "player A sounds better than B" posts in computer-audio
http://www.head-fi.org/t/657920/mqn-minimalist-wasapi-wav-c-memory-player-integrated-with-foobar2000
http://www.head-fi.org/t/444784/cmp-and-cplay-media-player
http://www.head-fi.org/t/634453/jie-extreme-player-2-7-2-released
http://www.head-fi.org/t/340451/lilith-audio-player
...
 
Measurements:
http://archimago.blogspot.ca/2013/06/measurements-part-i-bit-perfect.html
http://archimago.blogspot.ca/2013/06/measurements-part-ii-bit-perfect.html
 
 

(Software induced) Jitter, audibility

Reviews/claims:
see some of the audiophile audio player reviews or product pages
http://www.audioasylum.com/forums/pcaudio/messages/5/56101.html
 
Measurements:
http://www.head-fi.org/t/629729/xonar-d1-measurements (see loaded measurement results)
 
See also:
http://www.head-fi.org/t/668878/jitter-correlation-to-audibility/30
 
 

Wadia 121:

http://archimago.blogspot.com/2013/10/musings-updates-value-of-objective.html
... discusses how the Wadia 121 24-bit DAC (MSRP $1299) received very positive reviews, but Stereophile's measurements concluded only AES/EBU works "correctly" achieving about 17 - 18 bits of performance, S/PDIF 24-bit data was truncated to 16 bits and USB measurements showed a noise floor around the 15-bit level.
 
See also:
#11.
 
 

Earsonics SM64:

Reviews:
http://www.head-fi.org/t/651654/earsonics-sm64-the-impressions-and-appreciation-thread
http://www.head-fi.org/products/earsonics-sm64 (1 review, 4.5/5 stars)
 
Measurements:
http://rinchoi.blogspot.co.at/2013/05/earsonics-sm64-electroacoustic.html ("An electroacoustic abomination born")
 
 

Heir Audio 3.Ai

Reviews:
http://www.head-fi.org/t/622161/heir-audio-3-ai-impressions-comparisons-review-thread
http://www.head-fi.org/products/heir-audio-3-ai (3 reviews, 4.67/5 stars)
 
Measurements:
http://rinchoi.blogspot.co.at/2012/10/heir-audio-3ai.html (huge dip from 2 to 4 kHz, bad channel balance)
 
 

Heir Audio 4.Ai

Reviews:
http://www.head-fi.org/t/621602/heir-audio-4-ai-appreciation-impression-review-thread
http://www.head-fi.org/products/heir-audio-4-ai (9 reviews, 4.72/5 stars)
 
Measurements:
http://rinchoi.blogspot.co.at/2012/10/heir-audio-4ai.html (broader dip than the 3.ai, better matching in the lows though)
 
 

Beyerdynamic DT48 E (25 Ohm)

(maybe other versions too)
Reviews:
http://www.head-fi.org/t/418746/the-beyerdynamic-dt48-arrives
http://www.head-fi.org/products/beyerdynamic-dt-48-e-25-ohm (4 reviews, 4.62/5 stars)
 
Measurements:
http://www.innerfidelity.com/content/historic-beyerdynamic-dt-48-e-25-ohm-sound-and-summary
http://www.innerfidelity.com/content/skullcandys-director-electrical-acoustical-engineering-dr-tetsuro-oishi-visits-innerfidelit-0
 
 

Ultrasone Edition 10

Reviews:
http://www.head-fi.org/products/ultrasone-edition-10 (4 reviews, 4.5/5 stars)
 
Measurements:
http://www.innerfidelity.com/content/ultrasone-edition-10-page-2
 
 

Shure SRH1840 (and SRH1440)

Reviews:
http://www.head-fi.org/t/629143/shure-srh1840-review
http://www.head-fi.org/products/shure-srh1840-professional-open-back-headphones-black (3 reviews, 4.5/5 stars)
 
Measurements:
http://www.innerfidelity.com/images/ShureSRH1840.pdf
 
 
... more to come ...

 

[alpha421]

Interesting blog.  I'll admit that I'm a subjectivist, but I do believe when it comes to the truth, "Without data, you're just another person with an opinion."
 
- Quoted by Andreas Schleicher

 

[xnor]

There's nothing wrong with subjective tests if they're done properly, blind.
 
For example developers of lossy codecs (mp3, aac, opus ...) make extensive use of subjective tests like the good old ABX test or ABC/HR.

 

[alpha421]

Wouldn't you say that there's much more involved in "blind" subjective tests in that there's needs to be a set of criteria in place prior to such tests?  After all, one persons definition of flat, neutral, colored...etc means differently to another set of ears.  Then you have personal preference that automatically creates flavor bias and more so if your an analog-head versus digital-head. When it comes to objective measurements, I found digital-head friends make it much more of an issue over the analog-head friends.
 
I do, however, believe manufactures should be held accountable for all their objective measurements and claims whether it's published or not, and all high profile, subjective published reviews should be backed with objective facts when describing and praising a product (ie. claims it sounds neutral, but graphs proves otherwise).

 

[xnor]

  Wouldn't you say that there's much more involved in "blind" subjective tests in that there's needs to be a set of criteria in place prior to such tests? 

I was thinking of comparisons between DACs or amps. Obviously, matching the levels is extremely important for example.
You would expect sighted reviewers to do at least that but even that is not "standard", rendering any comparison completely futile (even if blinded).
 

After all, one persons definition of flat, neutral, colored...etc means differently to another set of ears.  Then you have personal preference that automatically creates flavor bias and more so if your an analog-head versus digital-head.

I see where you're going .. headphones. Sure, everyone has a slightly different hearing, shape and size of ears and head.
Imho:
- a subjective review saying that the headphone is comfortable, around-ear, etc. may be off for you whereas measurements (dimensions of the ear pads, clamping force etc.) are universally valid. Of course, you still should test the headphone yourself before buying it.
- acoustic measurements are usually made with a dummy head that is based on an average head ear/head shape/size etc.
- for example "bass light" might be the result of an electronica-listening bass head which could be neutral in measurements and would also sound neutral when listening to classical music
- whatever "reference" the reviewer has to compare the device under test to necessarily influences the review; measurements are not completely immune to that either (especially acoustic headphone measurements!) but usually you can buy the same measurement setup and reproduce the results
- Olive & Welti's headphone target equalization curve is based on subjective blind testing and trained listeners had similar preferences - if everyone had completely different preferences they wouldn't have been able to come up with that result
 

When it comes to objective measurements, I found digital-head friends make it much more of an issue over the analog-head friends.

Probably because analog stuff can measure very badly. Doesn't make the measurements invalid but the reviewers' claims that they have golden ears.
And makes the claims of "digital-head friends" that, for example, every DAC sound different (to put it another way: there's no such thing as transparency) even more ludicrous.
 

I do, however, believe manufactures should be held accountable for all their objective measurements and claims whether it's published or not, and all high profile, subjective published reviews should be backed with objective facts when describing and praising a product (ie. claims it sounds neutral, but graphs proves otherwise).

Me too. And what's also very important: describe the test setup (for example how the level matching was done).

 

[Steve Eddy]

What is it exactly that sighted/subjective reviews "fail" at?

se

 

[jcx]

detecting measurable differences already expected to be below human hearing thresholds?

 

[Steve Eddy]

detecting measurable differences already expected to be below human hearing thresholds?

That's a bit of a false premise, isn't it? I mean, that's not really the purpose of sighted/subjective reviews as far as I'm aware.

se

 

[nick_charles]

  Let's collect and discuss some evidence that show how subjective, sighted and therefore usually biased reviews fail.
 
Here's a starter:
http://archimago.blogspot.com/2013/10/musings-updates-value-of-objective.html
 
... discusses how the Wadia 121 24-bit DAC (MSRP $1299) received very positive reviews, but Stereophile's measurements concluded only AES/EBU works "correctly" achieving about 17 - 18 bits of performance, S/PDIF 24-bit data was truncated to 16 bits and USB measurements showed a noise floor around the 15-bit level.

 
One of the distinct sub-patterns you find frequently in Stereophile is for the listening reviewers to gush over a item which John Atkinson then measures and finds to be distinctly low fidelity, then he puzzles why his reviewer likes it when it is so technically deficient - especially when it is high jitter(LOL). On at least one [size=17.33333396911621px]Occasion[/size] he asks if a product is so well liked because it is so awful or despite it being so awful, on another he says "chalk one up to the subjectivists". As he is a bona fide engineer I am a bit dismayed by his approach.
 
But more interesting are the excuses made for some exotic devices and the careful language used when describing terrible technology even stuff that would not make the dismal DIN 45500 standards of the 1960s and the more expensive/exotic the more cautious the language. Generally tube stuff gets away with murder performance-wise and SP do not even measure turntables but will tut-tut about 1ns of jitter in a CD player and allow a TT with spec'd 1% speed error a free pass.
 
But more broadly I've been puzzling over a number of questions:
 
1. Could it be that SP reviewers are so overwhelmed by price, appearance, name, reputation that they simply cannot hear any major faults in something exotic that is technically terrible
2. Could it be that SP reviewers just hear what they see, i.e will faintly praise an item that is cheap but measures superbly while saying how inferior it is to a more expensive item that measures no better or even distinctly worse (often a sub-theme in JA's wrap-ups)
3. Could it be that even technically pretty terrible gear is generally sufficiently high fidelity such that we really cannot hear the difference most of the time unless the differences are grotesque such as with amps and difficult speaker loads and most differences between kit is largely imagined and woukd disappear if we had honest (unsighted) reviews
4. Why is really terrible technical performance shrugged off or just ignored consistently especially when talking about $12K or even $60K components when for far less money really good performance is achievable

 

[Joe Bloggs]

Let's collect and discuss some evidence that show how subjective, sighted and therefore usually biased reviews fail.

Here's a starter:
http://archimago.blogspot.com/2013/10/musings-updates-value-of-objective.html

... discusses how the Wadia 121 24-bit DAC (MSRP $1299) received very positive reviews, but Stereophile's measurements concluded only AES/EBU works "correctly" achieving about 17 - 18 bits of performance, S/PDIF 24-bit data was truncated to 16 bits and USB measurements showed a noise floor around the 15-bit level.

What is it exactly that sighted/subjective reviews "fail" at?

se

detecting measurable differences already expected to be below human hearing thresholds?

Lol indeed--if not reaching 24bit resolution is the only failing of the Wadia, blind testing will do absolutely zilch for uncovering it

 

[xnor]

What is it exactly that sighted/subjective reviews "fail" at?

  detecting measurable differences already expected to be below human hearing thresholds?

Lol indeed--if not reaching 24bit resolution is the only failing of the Wadia, blind testing will do absolutely zilch for uncovering it

 
Have you even taken a look at the reviews?
 
Let me quote from some of the reviews:

 With the high-res files, the Wadia did its best, and its best was fabulous.

Also: Awarded 5/5 stars in the performance category.
 

 Here Wadia, well known for producing some of the finest digital playback equipment available since the earliest days of the medium, has brought its considerable talents to bear in producing a DAC which is operationally bullet-proof at an unexpectedly modest price.

But in the Stereophile review both S/PDIF and USB did not work correctly...
 

The Wadia 121 Decoding Computer is the best affordable digital-to-analog converter that I've ever heard. [...] But given their track record, it is a safe bet that Wadia has not only put a lot of thought into the design of the Wadia 121, but this DAC won't be bested by any DAC for quite a few years to come – unless one is willing to spend quite a bit more than the Wadia 121's very reasonable asking price. [...] This one isn't leaving my system, not for quite a while.

 

 New computer audiophiles seeking their first entry into this wonderful next phase of high end audio can't go wrong by starting with the 121. They may never need another digital to analog converter.

 
 
And please look up what they also write about "high-res" (24-bit) music files, or the 32-bit DSP in the Wadia.
 
It might be the case that SP got a faulty review unit, but so far no follow-up has been posted. But anyway, this is by no means a precedence.

 

[Joe Bloggs]

Have you even taken a look at the reviews?

Let me quote from some of the reviews:
Also: Awarded 5/5 stars in the performance category.

But in the Stereophile review both S/PDIF and USB did not work correctly...





It might be the case that SP got a faulty review unit, but so far no follow-up has been posted. But anyway, this is by no means a precedence.

And blinding these reviewers would have helped them how? Going by the stereophile measurements the Wadia should still be perceptually transparent in any non contrived music listening test. I suppose you could make a case that subjective reviews, sighted or otherwise, are all useless at the current state of the art?

 

[xnor]

Oh I see what you mean, but that's not what I'm talking about.
 
Think of the same hardware being made by a lesser known manufacturer and sold cheaper. Would these sites/magazines even consider reviewing such a DAC? Would they come up with the same "excellent performance" etc. ratings?
 
Especially with the more expensive stuff I'd expect a rigorous test setup and procedure, comparing the device under test to a well known and well measuring device. The subjective opinion should be backed by objective data (measurements).
Regarding listening sighted or blind: well, using a silent 24 bit test track and boosting the volume would have shown the higher noise floor.
 
 
You're right of course that ~15 bit performance will be good enough even for many audiophiles, but that's not what these sites/magazines are writing. Not at all.

 

[Joe Bloggs]

Then you picked the wrong example. The Wadia *does* perform impeccably perceptually, subpar bit depth measurements notwithstanding. In that sense all those rave reviewers are not all off their rocker. Some better examples would be the reviews for the numerous expensive iems / ciems in vogue that have atrocious suckouts around 4kHz and atrocious phase performance. But whaddya know, people may have become so smitten with this colored sound that they would rate more neutral phones lower even in a blind test...

 

[xnor]

Why? What you're basically saying is that no matter if the DAC is 16, 24 or even 32-bits, it doesn't matter as long as it does ~15 bit performance and also that those subjective reviews regarding performance are completely irrelevant. It's completely meaningless if a $1299 item gets a biased "excellent performance" rating and a $199 items gets a biased "only ok" - the cheaper item may even perform better. It conclusively doesn't matter what those reviewers write about "high-res" being a better sounding format either.
 
Well, I kinda agree! In my book that's a fail.
 
 
 
I'm not an in-ear guy at all so please do link to such IEM reviews and measurements if available.