The flaws in blind testing
Nov 2, 2010 at 3:52 PM Post #31 of 44


Quote:
 

Crazy*Carl, I think you might be crazy.
smily_headphones1.gif

 
To my ears. . . There is an absolutely huge difference between 128 kbps and lossless, an ipod doesn't even sound as good as my meager creative DAPs never mind comparing to my (still modest) EF2A, and the 300 ohm HD 650s sound weird/bad without an amp (not that my amp is that great but on everything else the 650s sound weird). I mean this in the nicest way possible, but I think you may be somewhat deaf. . . 
 
I Think it is unusual that you are against blind tests, usually the objectivists are the ones that say "everything sounds the same!" Aren't the bold statements contradictory tho? You seem to have used failed blind testing as evidence to support your opinion yet claim failures lead to false conclusions!
 
No offense intended, just thought I'd put that out there. . . I'm kind of on the fence/leaning towards blind tests are a good thing, but only to prove things when you actually PASS them, so I think I'm on your side on this point. 
 


What are you talking about.  I am not against blind tests. I am for them.
 
I said when someone fails to blind test their equipment, it leads to false views on it.  This is a promotion of blind tests. Further more I go onto say that with a blind test, I found no deference between an ipod and an audio gd compass.   In fact if someone makes a claim about audio without blind testing, I have no reason to believe it.
 
And no, I am not deaf.  Enjoy your "absolutely huge difference" that you hear with all your snake oil.  Ridiculous people will pay so much for this bull#$#.  When they buy it, they HAVE to defend it because they will look like an idiot if they don't since they payed such a ridiculous amount of money for it.
 
There are even mainstream commercials out there that make fun of audiophiles using tube amps.  What a joke.  I had a tube amp once and guess what... IT DIDN'T SOUND ANY DIFFERENT THAN MY IPOD.
 
Nov 2, 2010 at 4:40 PM Post #32 of 44


Quote:
What are you talking about.  I am not against blind tests. I am for them.
 
I said when someone fails to blind test their equipment, it leads to false views on it.  This is a promotion of blind tests. Further more I go onto say that with a blind test, I found no deference between an ipod and an audio gd compass.   In fact if someone makes a claim about audio without blind testing, I have no reason to believe it.
 
And no, I am not deaf.  Enjoy your "absolutely huge difference" that you hear with all your snake oil.  Ridiculous people will pay so much for this bull#$#.  When they buy it, they HAVE to defend it because they will look like an idiot if they don't since they payed such a ridiculous amount of money for it.
 
There are even mainstream commercials out there that make fun of audiophiles using tube amps.  What a joke.  I had a tube amp once and guess what... IT DIDN'T SOUND ANY DIFFERENT THAN MY IPOD.


Woah, sorry I misinterpreted you. 
 
I haven't spent much money on my little amp, and I wouldn't mind admitting it is pointless if I really felt that way, but for what it does I am very happy with it. I don't think anything I have would be considered snake oil by even most of the more rational people here on head-fi, hey at least I don't buy expensive cables! (and I never will). If you are happy with your ipod I guess that's all that matters. . . . 
 
Nov 2, 2010 at 4:47 PM Post #33 of 44
Where have all the people who regularly dismissed blind test gone?
 
Nov 2, 2010 at 5:59 PM Post #34 of 44
Quote:
 
In theory, DBT could do the job, but in practice would be far too unwieldy - months and months of blinded solo testing in thousands of homes.  Is there an alternative?
 


For starters, get someone with near-perfect auditory memory to play ball, implement these few pointers http://www.head-fi.org/forum/thread/435801/propose-your-protocol-here/15#post_5986220
 
Nov 2, 2010 at 6:37 PM Post #35 of 44
 
Quote:
The amount of debating here is hilarious.  Listen and compare.  If you cant hear a difference, then don't spend money on it.  I think failing to blind test equipment just leads to false views on it (This is seen here all the time with people's bogus claims about their equipment)
 


You make an interesting point here.... 
 
Lets agree that beyond a certain level of electronically measurable quality the human ear can not reliably discern 2 pieces of gear. 

The catch 22 of this is that people wont buy something unless it measures very well, which in the case of lots of gear puts you beyond this threshold. But if it measures better they will often buy the more expensive one because it does have something better going for it.... you can see how this becomes very circular. Well, you could probably see it before I posted, but maybe other people will pull their heads from their asses. Probably not. 
 
As an added problem to continue this issue. People will not listen to your gear at trade shows if they thing something is wrong with it. For example: 
Many people will not listen to a system, or would subconsciously downgrade their opinion of a system if the cables were not up to snuff. You regularly see this at any stereo club meeting where people will flat out state that they wont listen to a speaker system wired with (insert flavor of last month) cables or give recommendations to upgrade certain parts of the system after a few moments of listening in a bad room...
 
Quote:
On the other side, I can not hear a difference using entry level hifi amps and dacs, so I just use my computers motherboard sound.  And boy does it sound good
smily_headphones1.gif

 
Quote:
Thats a joke right.......?

 
Quote:
No its not a joke.  I have blind tested 3 different hifi amps with my ipod shuffle and computer, and me or the other person testing was never able to distinguish them.
 
Head-fi does not wanna hear the truth: an ipod is as good as an audio gd compass.
 


I'd believe it. 
 
What is unfortunate, and where the true problem lies is that DBT is not a method of determining what sounds good, simply whether 2 things sound similar enough that people cant hear the differences. 
 
I would argue that neither an Ipod or an Audio-GD compass sounds particularly good, but this is a matter of opinion. I trust you that they sound similar enough that you cant tell them apart.
 
Quote:
I do think that it is important to look at blind tests especially when a test (Swedish Radio) produces such an unexpected result. I would have expected such a huge blind test which concluded no difference would be accurate. But it was not. So it seems reasonable to go back and check other blind tests and see if there is/can be the same flaw in them.
 
I agree ignoring blind testing is wrong and that it has resulted in many exaggerated and bogus claims being made.
 
I also think that you not being able to differentiate between the lowest and highest bit rate and a dedicated amp/DAC over your PCs soundcard is interesting, as there are blind tests which suggest that both can be differentiated.


I didnt read the whole article...
After the independent tester (sighted) informed the group of how to hear the issue in the coded music, was the group able to differentiate the 2 samples by DBT?
 
It is VERY easy to sway a sighted test. Telling the group what they should expect to happen is a prime example. When you tell people what they should expect to happen a LARGE percentage of the time it will in a sighted test, particularly when there is nothing happening. How often do cable demos start with a guideline of what the listeners should hear? 
 
I do agree that DBT's should be re-confirmed from time to time, or when new methods of listening are brought to the table. By "new methods of listening" I do believe that there are things you can train the ear to hear, even if it does take some time - accordingly I would not be totally surprised if the above sweedish example brought new DBTable results to the table but if a second DBT was not given after the sighted tester told the people what to hear the results are not valid. 
 
There is a fellow in the local audio club who correctly identified the types of 2 turntables on a blind test (same record, same stylus, same phono stage, volume matched...), where I cant personally hear the difference. OTOH he demonstrated to my satisfaction that there is an audible difference and actually preferred the LESS expensive TT!
 
Nov 3, 2010 at 3:53 AM Post #37 of 44

 
Quote:
 
There is a fellow in the local audio club who correctly identified the types of 2 turntables on a blind test (same record, same stylus, same phono stage, volume matched...), where I cant personally hear the difference. OTOH he demonstrated to my satisfaction that there is an audible difference and actually preferred the LESS expensive TT!


I'd imagine some very well could.  TT are mechanical, one may have a bit more rumble that he was listening for that you may not have.  Actually, this is why there's an interesting debate between belt driven and direct drive turntables.  DD are usually cheaper and generally tend to suffer from less rumble in decent implementations.  Some people refuse to give up belts though . . . not entirely sure why.  Only reason I have a belt one is, quite frankly, it was cheap.
 
I like Carl's idea of DBT ones current gear against different ones to see if they can even tell a difference.  I imagine one could save a lot of money and skip unnecessary upgrade paths such a way.  On the other hand, I will disagree with him in regards to motherboard sound - it's variable even today.  For example, if you have a graphics card or fans near the connection it can become noisy due to interference.  Try routing sound to the front of a computer case and you'll realize that PCs are still very very noisy.  Whether you can hear this noise on your headphones is another thing entirely though, some is higher frequency which may roll-off.
 
Nov 3, 2010 at 7:33 AM Post #38 of 44


Quote:
 
 

.........
 


I didnt read the whole article...
After the independent tester (sighted) informed the group of how to hear the issue in the coded music, was the group able to differentiate the 2 samples by DBT?
 
It is VERY easy to sway a sighted test. Telling the group what they should expect to happen is a prime example. When you tell people what they should expect to happen a LARGE percentage of the time it will in a sighted test, particularly when there is nothing happening. How often do cable demos start with a guideline of what the listeners should hear? 
 
I do agree that DBT's should be re-confirmed from time to time, or when new methods of listening are brought to the table. By "new methods of listening" I do believe that there are things you can train the ear to hear, even if it does take some time - accordingly I would not be totally surprised if the above sweedish example brought new DBTable results to the table but if a second DBT was not given after the sighted tester told the people what to hear the results are not valid. 
 
There is a fellow in the local audio club who correctly identified the types of 2 turntables on a blind test (same record, same stylus, same phono stage, volume matched...), where I cant personally hear the difference. OTOH he demonstrated to my satisfaction that there is an audible difference and actually preferred the LESS expensive TT!


I very much get the impression that after the sighted test found the flaw, it was put back to the listeners who now had knowledge of it and then could hear it.
 
Nov 3, 2010 at 10:32 AM Post #39 of 44
 
Quote:
I very much get the impression that after the sighted test found the flaw, it was put back to the listeners who now had knowledge of it and then could hear it.


Thats the key point though. 
If they could not hear it blind, after claiming to hear it sighted...
 
Quote:
I'd imagine some very well could.  TT are mechanical, one may have a bit more rumble that he was listening for that you may not have.  Actually, this is why there's an interesting debate between belt driven and direct drive turntables.  DD are usually cheaper and generally tend to suffer from less rumble in decent implementations.  Some people refuse to give up belts though . . . not entirely sure why.  Only reason I have a belt one is, quite frankly, it was cheap.
 


It was a direct drive that the guy preferred, but what was more important to me was that I didnt tell him exactly what he was listening to. I said "here are 2 needle drops with the same stylus, LMK what you think". He identified the basic type of both TT's. In the test the difference in cost was about 10:1, between a well regarded belt drive TT and an old Sony DD. 
 
I would not say cheaper for DD, as that implies relative worth. When TT's were a common consumer item, like DVD players are today, DD TT's were actually more expensive. The fact that you can find a "stupid good" deal on an older DD table today is simply a result of the fact that the audiophile press almost universally despises DD's. Why? Follow the money trail. What are people who build TT's today and buy add space building? Do you think most modern TT builders could build a DD motor & controller? VERY few have built a new DD system in the past 15 years, but anyone with even medium mechanical can build a belt drive.
 
If your interested in a DD keep tabs on craigslist. Technics 1200's are awesome, but the prices have gone stupid in the past year or 2. Other late 70's/early 80's Japanese DD TT's are outstanding deals though.
 
Nov 3, 2010 at 11:11 PM Post #40 of 44

 
Quote:
I would not say cheaper for DD, as that implies relative worth. When TT's were a common consumer item, like DVD players are today, DD TT's were actually more expensive. The fact that you can find a "stupid good" deal on an older DD table today is simply a result of the fact that the audiophile press almost universally despises DD's. Why? Follow the money trail. What are people who build TT's today and buy add space building? Do you think most modern TT builders could build a DD motor & controller? VERY few have built a new DD system in the past 15 years, but anyone with even medium mechanical can build a belt drive.
 
If your interested in a DD keep tabs on craigslist. Technics 1200's are awesome, but the prices have gone stupid in the past year or 2. Other late 70's/early 80's Japanese DD TT's are outstanding deals though.


I definitely wasn't talking about relative worth, just that they're less expensive to purchase.  In terms of technology they have definite benefits.
 
Nov 5, 2010 at 11:06 AM Post #41 of 44
Ivor Tiefenbrun, founder of Linn and at the height of that company's dedication to the TT as a source, fails three blind tests
 
http://www.bostonaudiosociety.org/bas_speaker/abx_testing2.htm
 
one of which is having a digital processor in the system against no digital processing.
 
Nov 5, 2010 at 12:13 PM Post #42 of 44


Ivor Tiefenbrun, founder of Linn and at the height of that company's dedication to the TT as a source, fails three blind tests


 


http://www.bostonaudiosociety.org/bas_speaker/abx_testing2.htm


 


one of which is having a digital processor in the system against no digital processing.



I especially love Tiefenbrun's claim that an undriven transducer in the room degrades sound quality. Even the piezo buzzer on a digital watch.

Maybe having all these headphones and speakers about is the reason I can't hear the "difference" between cables.
 
Nov 8, 2010 at 4:13 PM Post #43 of 44
There is some evidence to suggest that visual cues might help to boost auditory plasticity, is that some of the reason why blind testing can be somewhat flawed?
 
I also wonder how Contingent Negative Variation affects blind testing, especially considering this..
 
Wikipedia:
Attention also affects the amplitude of the CNV. The following examples from various task conditions and studies show that the CNV is changed when the experimental protocol changes the attention needed to perform the tasks. First, when subjects were told that the imperative stimulus would be removed, the CNV was reduced. Second, in one condition subjects were allowed to choose whether they were going to press the button or not. In trials where the subject chose not to respond, there was no CNV. Third, when the subject was specifically told that there would not be repetitive flashes, no CNV was elicited. Fourth, another condition showed that a CNV was elicited in subjects who were told to estimate when the repetitive flashes would come even when no flashes were presented. Fifth, when subjects were asked pay attention and respond quickly, CNV amplitude was increased. The results of these conditions suggest that the CNV is related to attention and expectancy.
 
When the probability of repetitive flashes is random and the repetitive flashes are removed in about 50% of the trials, the amplitude of the CNV is about half as that of normal.

Some researchers have shown that the intensity of the stimulus may affect the CNV amplitude. It seems that the CNV component has a higher amplitude for stimuli that have low-intensity, i.e. is difficult to see or hear, as opposed to stimuli that have high-intensity. This could be because the subject must pay more attention to perceive the low-intensity stimulus. If the detection of the imperative task becomes too difficult, then the CNV amplitude is reduced. In other words, attention to the imperative stimulus is important for the development of the CNV and increased task difficulties distract the attention.

http://en.wikipedia.org/wiki/Contingent_negative_variation#Attention_and_expectancy
 

Users who are viewing this thread

Back
Top