[xnor]

Quote:

 
Tubes vs SS anyone?  

 
If you hang out in these threads too long, it can amount to huge savings and more focus on the music.

 
SS amp A vs SS amp B anyone? haha
 
*thumbs up* aimlink

 

[happyxix]

They should get a real blind guy to do some testing. That why he have no visual bias at all and maybe his hearing is also probably enhanced?
 
That would be interesting.

 

[Prog Rock Man]

I have added another example (no 16) of a blind test, a blog studying the differences between blind and sighted tests. After it is a very interesting discussion by Sean Olive from Harman International, who ran the blind/sighted test and Alan Sircom who is the editor of Hifi Plus magazine....
 
 
 
 

Dear Sean,

As the new editor of Hi-Fi+ in the UK (perhaps one of the most 'out there' of audiophile magazines), I guess I am the Loyal Opposition. As such, I respectfully disagree with your suggestion of dishonesty in sighted tests. 

The word 'dishonesty' implies some kind of deceit in the actions of the reviewer. Although I cannot speak for all subjective reviewers at all times, I suspect most would view their actions as being principally honest, but holding to a different set of values to yours. There's an obvious analogy here; a conservative might be fundamentally opposed to the viewpoint of a liberal (or vice versa), and may even express incredulity at those who support such a position, but still respect the integrity of that stance. Or at least, that used to be the case, but I suspect “I disapprove of what you say, but I will defend to the death your right to say it” is passé now.

For my part, I maintain that sighted tests can reflect the real-world conditions in which people choose and use their products. For example, because blind tests are inherently level-matched in design, they do not take into account the way products are evaluated by listeners in reality. 

Here's an interesting test to explain what I mean: run a blind test a group of products under level-matched conditions. Then run the same test (still blind), allowing the users to set the volume to their own personal taste for each loudspeaker under test. From my (admittedly dated and anecdotal) testing on this, the level-matched group will go for the one with the flattest frequency response, as will those who turn the volume 'down', but those who turn the dial the other way often choose the loudspeaker with the biggest peak at around 1kHz, saying how 'dynamic' it sounds. I wrote on the topic in the early to mid-1990s (I believe it was in Hi-Fi Choice magazine, but the magazine's back-catalog is long gone now).

Unlike some of my colleagues, I am not opposed to blind-testing, in part because my of my previous work with Hi-Fi Choice in the UK (which does still - at least partially - continue to run blind tests). However, I am keen to explore all potential avenues to see if audiophiles are hearing things, or hearing things. As such, I think there might be something other than double-blind ABX that has some degree of scientific credibility, and which might be able to answer this... such as longitudinal testing.

I welcome your comments on the subject.

Kind Regards

Alan Sircom
Editor, Hi-Fi Plus magazine


 
 
 
 

APRIL 16, 2009 4:34 PM
 
 

 
 

Sean E. Olive said...
 
 
 
 

Dear Alan,

Thank you for your response. I appreciate your feedback, and I am sorry if I caused you offense.

It was not my intent to single out audio reviewers for not doing blind tests. Indeed, most audio manufacturers don't do controlled listening tests as part of the product validation and testing. If they have comprehensive perceptually relevant objective measurements in place, then listening may be less important.

I don't think I implied reviewers are intentionally deceitful and dishonest. The word “dishonest” was used to describe the sighted test methodology itself. It fails in measuring the true sound quality of the product due to the influence of listeners’ psychological biases. The listener may not even be conscious of these biases, in which case, they could hardly be accused of being “dishonest” or “deceitful.” I can hardly be blamed as deceitful if I choose the red speaker over the light green loudspeaker, because it sounds louder and more powerful (like a red Ferrari). 

Most audio reviewers I've met are decent, honest, intelligent people trying to do the best job they can given the limited time, budget and resources at their disposal. Most reviewers who visit Harman, tell me they would love to have access our listening facilities or have something like it for reviewing products. Given the choice, I think most reviewers would use a combination of blind and sighted tests. 

I agree that sighted tests have a purpose, particularly to determine the influence of the visual factors (brand, price, design, advertising) on consumers’ perception. This allows audio companies to optimize the right balance of sound quality versus other important design/marketing variables (industrial design, advertising, etc) and predict consumer acceptance in the marketplace. Also, it doesn’t require a blind test to establish that a speaker sounds unacceptable due to audible rub and buzz. 

Your example of having listeners adjust the level of different speakers to their preferred taste, to me, correlates with how much non-linear distortion or power compression the speakers have. Listeners will tend to increase the volume until the speaker and/or their ears begin to produce high-order distortion. Be careful: If the loudspeaker is a JBL Everest - you may find yourself listening at dangerously high SPL levels (>110 dB peak) before you realize it!


Cheers
Sean


 
 
 
 

APRIL 17, 2009 4:51 AM
 
 

 
 

Alan Sircom said...
 
 
 
 

Dear Sean,

No offense taken at all, and from what I gather, I would be one of the envious of your facility.

My goal in writing here is arguably the same as I hold for the magazine; that there needs to some kind of rapprochement between the objective and subjective sides of the business. This is a long-term goal, I need to build a foo-broom with a longer handle first

Kind Regards

Alan


 
 
 
 

APRIL 17, 2009 5:09 AM
 
 
 
 

 
 

 

[revolink24]

The thing I always find interesting about blind testing is that the first people to claim they can tell a difference are the last to step up the plate and prove it. And I think that just says everything.

 

[Prog Rock Man]

I certainly agree with the numerous comments I have read hunting for blind test results, that audiophiles have lost a lot of credibility by ignoring or arguing against blind/ABX testing. 
 
I think that the discussion above is important as to another way to run blind tests which could be a compromise between those who favour sighted and those who say it needs to be blind test to get a true result. Let people adjust the volume to one which they like. i can understand that as I like a bright. forward sound, so I prefer the volume up where the dynamics kick in, which is too loud for others.

 

[aimlink]

Quote:

I certainly agree with the numerous comments I have read hunting for blind test results, that audiophiles have lost a lot of credibility by ignoring or arguing against blind/ABX testing. 
 
I think that the discussion above is important as to another way to run blind tests which could be a compromise between those who favour sighted and those who say it needs to be blind test to get a true result. Let people adjust the volume to one which they like. i can understand that as I like a bright. forward sound, so I prefer the volume up where the dynamics kick in, which is too loud for others.

This is a reasonable point.  The K702's, in particular, have poor dynamics at low listening volumes.  You need the volume nicely up to really get it going.  Not sure why...  So volume corrections could lead to issues if one piece of gear is operating more optimally at a particular volume.

 

[Leny]

Quote:

 
SS amp A vs SS amp B anyone?
 
 

 
That made me smile!

 

 

[Leny]

Some time ago I was at a hi-fi show and sitting in a demonstration that had big speakers playing.
The sound was fine... not world class, but certainly entertaining. 
 
I thought the system and the big speakers were doing fine. Then... half way through the second track the company guy walks over to the big speakers and swings open the fronts... revealing really tiny speaker boxes inside! Then he points out a very small sub woofer in the corner of the room!.
 
I can't remember the name of the company... it may have been Bose... but it was a good demonstration of how sight can be misleading.
 

 

[xnor]

Haha, that made me laugh.

 

[timar]

I'd like to add another empirical study (this time conducted on a scientifical level), which should be of greatest interest here, since it especially deals with headphones.
 
Part one is theory, part two is the empirical study:
http://www.cpt.univ-mrs.fr/~briolle/11thAESpart1.pdf
http://www.cpt.univ-mrs.fr/~briolle/11thAESpart2.pdf
 
The study was conducted for the international Audio Engineering Society and shows, in short, that if the frequency response of different headphones is accurately measured, one pair of headphones can be equalized to give the same subjective impression/quality like any other pair, just by simulating its frequency response.
 
Conclusion is, that the sound of headphones of some basic quality (e.g. THD well below 1%) is determined (almost) exclusively by their frequency response. All those fancy audiophile terms like soundstage, resolution, clarity et al are nothing but functions of the actual frequency response.

 

[Prog Rock Man]

Thanks for the links timar. It was interesting where the study stated the different sounds of different headphones can be re-created in one headphone (lower link, page 2). They also recognise the need to have an equal volume and each person being tested should set the volume to their preferred level. I think that there is evidence that tests can be flawed and 'differences' found between cables or equipment, purely down to differences in volume.
 
The studies by Floyd Toole (first post no15 in the list) also corroborate the view that an equal frequency response is the generally preferred sound. 

 

[Ham Sandwich]

Quote:

I can't remember the name of the company... it may have been Bose... but it was a good demonstration of how sight can be misleading.
 

That was very likely Bose.  I sat through one of their demos and that's exactly what they did.  But more than a demonstration of how sight can be misleading, it was more of a demonstration of how music you are unfamiliar with can be misleading.  Especially if those doing the demo intentionally make it misleading.  The music used for the demo wasn't really music, it was more like movie sound effects and movie soundtrack muzak.  Nothing that you could be familiar with and be able to translate how it should sound on a real stereo.  Sounds that could be EQed and carefully chosen to not demonstrate the hole or discontinuity the Bose system almost certainly has between the sub and the mains.  An interesting demo of how audio tricks can deceive you.

 

[Leny]

Hi Ham Sandwich, I can't remember what was playing at the demo because it was quite some years back. It may well have been as you suggest.

 

[Prog Rock Man]

I have that same demo where the speaker opened up to reveal the Jewel (one little speaker on top of another) speaker in all of its tinyness. It was rock, Queen or Led Zep that was playing and it worked well as a demo to show how powerful the little speakers are. Power as opposed to sound quality is what I also remember.

 

[jwhitakr]

Great thread - subscribed to it.
 
Thanks Prog Rock Man for the effort to compile these various links together!  Have not read through all of them yet, but working my way through them one at a time.  No real surprises, thus far.