Head-Fi.org › Forums › Equipment Forums › Sound Science › What Hifi Magazine - The Big Question, blind testing.
New Posts  All Forums:Forum Nav:

What Hifi Magazine - The Big Question, blind testing.

post #1 of 39
Thread Starter 
What Hifi Magazine - The Big Question, blind testing.

Each month WHF magazine get three or their forum members into their listening studios and put them through various blind tests. The aim is to answer various questions about the validity of various Hifi myths and to help readers decided on what is worthwhile changing or not with their Hifi.

The articles are only available in the magazine, so I thought I would summarise the ones I have. I buy the magazine when there is an article I want to read and don’t subscribe. I am a forum member and I trust the numerous reports from the forum members who have attended that everything is fair and above board. Those who do go are not told what will happen until they get there and are then sworn to secrecy until the magazine comes out.

Feb 2010 - upgrading a budget system. A NAD CDP and amp and Dali speakers are swapped about with a Cyrus CDP and amp and ATC speakers. The results were that each change, replacing a budget with a mid range item was noticeable, but not always worthwhile. The difference between budget and midrange was not as great as expected. The overall conclusion was that upgrading “no longer has a simple answer”.

June 2009 - how low can compression go? Three files at 128kbps, 320 kbps and Apple lossless were played and each was identified correctly. Then a CD versus a ripped Wav file with two to one preferring the CD. Next were three files at 128, 192 and 256kbps. The result was completely mixed with no file performing better than another. Next was WAV versus Apple Lossless and no one expressed a definite preference. All struggled to say that there was a difference, though two said they preferred WAV.

The last test was Apple Lossless versus 320 kbps and lossless was preferred, but not by a much. The conclusion was that compression does make a difference, but is more noticeable between high and low and not so comparing each with something in-between,

Sept 2009 -how much difference do cables make and are they worth it? The set up is blind with both kit (Cyrus, Roksan and Spendor) and the cables. As far as the forum members are concerned each time a change is made it is the kit and no mention is made that it is the cables being changed. They are played the same three tracks with each change (Gershwin, The Chemical Bros and Bonnie Prince Billy).

First time it is with cheap no brand cables and the conclusion is that it all sounds flat with poor dynamics.

Second a Lindy mains conditioner and Audio Copperline Alpha power cables are introduced. The conclusion was that the sound was now clearer, more dynamic with better sound stage.

Third the standard interconnects are changed for Atlas Equator MkIIs. The conclusion is that the Gershwin track is the same, but the other two are better for bass and detail.

Fourth the speaker cable which was 60p per metre is swapped for Chord Carnival Silverscreen at £6 per metre. The Gershwin track was better for one person, but the others said there was a small difference.

Then the systems are mixed between 1 to 4 and it is clear that system 1 is outperformed by 2,3 and 4, but 2,3 and 4 are closer.

When the forum members are told what had been going on they are very surprised. The conclusion is that mains conditioning with the Lindy and mains cables made the biggest difference and after that interconnects and then speaker cable.

I have also flicked through issues testing racks, yes different racks make a difference and there have been quite a few on home cinema, including HDMI cables and they made a difference as well!!!

You will no doubt find flaws in these test, but there is no doubt that there is an effort going on to verify audiophile claims. Could the next Can Jam include some sort of blind test and the results published?
post #2 of 39
Quote:
Originally Posted by Prog Rock Man View Post
I have also flicked through issues testing racks, yes different racks make a difference
I have to say I am sceptical of this. Do you have a link to their testing procedure? Are the testers independent?

This is a What HiFi review of an equipment rack, there are others like it if you care to take a look.
Quadraspire Sunoko Vent T Equipment rack review - from the experts at whathifi.com
'Our test rig really sang through this rack, serving up a refined, well-lit treble, smooth and rich midrange and a tight bass that featured superb grip and bags of detailing.

Dynamically you get a first rate performance, and the production subtleties in Dot Allison’s Tomorrow Never Comes feel like a revelation. What more could you want from a rack?'

Oh Really? What on earth are they comparing it against?

I used to like this mag for the reference guide at the back, but some of their reviews like the above make me cringe. Subsequently I don't pay much attention to their reviews of audio equipment any more. They are a powerful influence in the UK market though, just a few good or bad words from them can make or break a new product.
post #3 of 39
Quote:
Originally Posted by Shark_Jump View Post
?

I used to like this mag for the reference guide at the back, but some of their reviews like the above make me cringe. Subsequently I don't pay much attention to their reviews of audio equipment any more. They are a powerful influence in the UK market though, just a few good or bad words from them can make or break a new product.
I think the information posted by prog rock man is interesting but have to agree with the above, some of their reviews are quite poor and better publications are available.
post #4 of 39
Quote:
Originally Posted by Prog Rock Man View Post
You will no doubt find flaws in these test, but there is no doubt that there is an effort going on to verify audiophile claims.
The biggest flaw seems to be that the participants always know that the items are beng changed between tests, so they always anticipate a difference. If they randomly introduced some no change conditions it would strengthen the tests.

Also it is very messy altering multiple variables. It is better to just change one variable at a time (cabe, CD player, amp and so on).

Beyond that we do not know how many repetitions there were, my guess is just one of each as is the typical hifi mag protocol. Again more repetitions really help statistical power.

A noble effort but without more details it is hard to know how strong the tests are.
post #5 of 39
Quote:
Originally Posted by Shark_Jump View Post
Dot Allison’s Tomorrow Never Comes feel like a revelation. What more could you want from a rack?'
Looks fine to me

P.S. Apologies. I come from Essex, in case you couldn't tell...
LL
post #6 of 39
Thread Starter 
Quote:
Originally Posted by Shark_Jump View Post
I have to say I am sceptical of this. Do you have a link to their testing procedure? Are the testers independent?

This is a What HiFi review of an equipment rack, there are others like it if you care to take a look.
Quadraspire Sunoko Vent T Equipment rack review - from the experts at whathifi.com
'Our test rig really sang through this rack, serving up a refined, well-lit treble, smooth and rich midrange and a tight bass that featured superb grip and bags of detailing.

Dynamically you get a first rate performance, and the production subtleties in Dot Allison’s Tomorrow Never Comes feel like a revelation. What more could you want from a rack?'

Oh Really? What on earth are they comparing it against?

I used to like this mag for the reference guide at the back, but some of their reviews like the above make me cringe. Subsequently I don't pay much attention to their reviews of audio equipment any more. They are a powerful influence in the UK market though, just a few good or bad words from them can make or break a new product.
It is WHF staff who run the tests and as I said the only information is published in the magazine. But forum members who have been are adamant all is above board.

With regards to racks, when I bought one for my old system I was staggered that it did make a difference. But when I made my own one it made no difference. So I am prepared to go with their conclusion that different racks make a difference.

I do agree some of the terminology and descriptives used are OTT, but they are journalists with a remit to sell a magazine by making it interesting.

Quote:
Originally Posted by nick_charles View Post
The biggest flaw seems to be that the participants always know that the items are beng changed between tests, so they always anticipate a difference. If they randomly introduced some no change conditions it would strengthen the tests.

Also it is very messy altering multiple variables. It is better to just change one variable at a time (cabe, CD player, amp and so on).

Beyond that we do not know how many repetitions there were, my guess is just one of each as is the typical hifi mag protocol. Again more repetitions really help statistical power.

A noble effort but without more details it is hard to know how strong the tests are.
From the various articles, they do only change one thing at a time, maybe that did not come over in my original post.

If you want more detail, why not join the WHF forum and ask?
post #7 of 39
Another flaw is that only 3 or 4 people are tested. I understand it's not easy to test a lot of people, but when there's only 3 the results can be pretty fairly skewed. Look at the CD vs. WAV example. Just because 2 of 3 preferred CD means nothing about the difference, because there's so few people. Had 200 out of 300, or even 20 out of 30 preferred it, it would be much more likely that there is a difference. One repetition (assuming), two options, and three testers leaves a lot of room for random guessing appearing statistically significant.

But hey, at least someone's trying.
post #8 of 39
Prog Rock Man, I understand your interest however the methodology and process of the magazine is shallow and has next to no validity. They may mean well with their forum members' testing, but I would not place any value in their results.

You are absolutely right about it being to sell magazines.

As for racks... it's all about vibration being passed to the equipment. The vibration comes from the loudspeaker / sound wave. Some racks pass more vibration, some less. Some equipment is susceptible to vibration, and some not.
post #9 of 39
Quote:
Originally Posted by Leny View Post
As for racks... it's all about vibration being passed to the equipment. The vibration comes from the loudspeaker / sound wave. Some racks pass more vibration, some less. Some equipment is susceptible to vibration, and some not.
Sure, I understand this is the 'consensus' opinion for the reason a rack makes a difference, but has this reason been experimentally determined as a fact?

The reason I say that is this....
An equipment rack is essentially a piece of room furniture in the sense that it will have an effect on the overall frequency response of your room. What are the chances the differences you are hearing from changing your rack are due to the room frequency response changing?

For an example here is a hypothetical situation. You use a big bookshelf full of books for your high fi equipment and replace it with a dedicated frame. Even if you put your hi fi equipment on the floor, the fact you have changed your room furniture around will alter your room frequency response enough to make large changes to the sound you hear. This is due to the rack, but nothing to do with vibration being passed to your equipment, as it isn't even on the rack

Note obviously this only applies to speakers, not headphones. If you hear a change with headphone then it blows my theory out the window.
post #10 of 39
I must agree that the test could and could not be valid. It could not be considered a fact because of some factors given in the test. First, the place of room must be determined. Second, the people they used should be capable on this test. Also, the question should not be answerable by subjective statements.

But what can we do? It is their job to persuade people on a certain products. So, before believing, I think, we should make our own tests or researches.
post #11 of 39
Just to clarify, with my 'rack' theory, I was talking generally, not specifically about the What Hi Fi test.

Although if I may add, regarding the 'consensus' opinion for the reason a rack makes a difference, eg equipment vibrating.

Just how does a vibrating circuit board, or whatever it is that happens to be vibrating (although this is conveniently never discussed), effect the outputted signal?

Unless someone tells me this I would like to put my theory on my previous post out there. Can I do this on this thread, or is that a 'thread-Jack'?
post #12 of 39
What HiFi prescribes to the notion that turntables are an essential part of any self-respecting audiophiles system. It is more than mere coincidence that one of the principle drawbacks of such indulgence is the equipment's susceptibility to vibration. The same logic could be extended to any disc player, assuming the player was manufactured before the days of skip protection and that its owner lives in a house which rests on top of a very active faultline just across from a train station.

This gives me an idea: invite a couple of writers from What HiFi for a weekend on the ol' yacht in the caribbean to demonstrate the effectiveness of my patented memory-foam-padded pure-silver equipment rack. Hell, I'll even let them ABX it with a milk crate!
post #13 of 39
Ohh yes, record players and CD players.

I had forgotten all about them, do some people still use them?

Yacht in the Caribbean! Hell, I'll give you 4 out of 5 stars and I haven't even seen your milk crate.
post #14 of 39
Thread Starter 
Quote:
Originally Posted by Leny View Post
Prog Rock Man, I understand your interest however the methodology and process of the magazine is shallow and has next to no validity. They may mean well with their forum members' testing, but I would not place any value in their results...........
I suspect that even if a very large scale ABX or other test was run there would be those who remain unsure. In any case the results would not be totally conclusive as there will be those who hear no difference and those who do.

WHF have at least contributed to the many smaller scale tests that suggest there is a case for different cables, racks etc.

There are many forum posts here and on the WHF forum where people have run their own mini blind tests. Some find no difference and others do. I did my own with different bit rates on the wife, which she got 100% correct and then when she did on me I did not!

The real aim here is to add actual evidence to the numerous round and round debates that fill this and all other audiophile forums.

I am on the list waiting to get to one of the WHF tests myself.
post #15 of 39
Hey Prog Rock Man.

Perhaps you could you try to contact the What Hi Fi editor on their forum and ask them what their testing procedures are. It would be interesting to know.

Nick Charles posted a great set of links to some ABX testing on page 25 of the 'How do I convince people that audio cables DO NOT make a difference thread'.

Here is a good one on amps:
http://www.bruce.coppola.name/audio/Amp_Sound.pdf

It would be good if you could update us when you do your test. You could compare it against this one if you like to use it as a benchmark.
New Posts  All Forums:Forum Nav:
  Return Home
  Back to Forum: Sound Science
Head-Fi.org › Forums › Equipment Forums › Sound Science › What Hifi Magazine - The Big Question, blind testing.