[krismusic]

Is blind testing a truly neutral tool? One of the abilities that is being tested is that of the listener to quickly identify small differences. Maybe some qualities only reveal themselves in extended listening?

 

[DavidA]

Is blind testing a truly neutral tool? One of the abilities that is being tested is that of the listener to quickly identify small differences. Maybe some qualities only reveal themselves in extended listening?

Good point.  But I think if it takes a extended time then the differences are too small to be noticed/meaningful.  Curious to what do you consider extended listening?  My definition of extended listening is a few hours. 
 
A few blind tests that I've tried was using an A/B switch to switch between DACs, differences were noted within a song or two every time. Also did a A/B test with HD-600 and HD-650, it depends on the individual which they think is the better sounding one or their own preference. 
 
Another problem with doing a blind test is the experiences of the person doing the evaluation, as @ColtMrFire noted above, every one does not have the same hearing and point of reference.
 
Also, if it takes an extended listening then there is also the probability of brain burn-in, I could be wrong, but its my take.

 

[krismusic]

Good point.  But I think if it takes a extended time then the differences are too small to be noticed/meaningful.  Curious to what do you consider extended listening?  My definition of extended listening is a few hours. 

A few blind tests that I've tried was using an A/B switch to switch between DACs, differences were noted within a song or two every time. Also did a A/B test with HD-600 and HD-650, it depends on the individual which they think is the better sounding one or their own preference. 

Another problem with doing a blind test is the experiences of the person doing the evaluation, as @ColtMrFire
 noted above, every one does not have the same hearing and point of reference.

Also, if it takes an extended listening then there is also the probability of brain burn-in, I could be wrong, but its my take.

Well, here we are in our nice shiny new thread!
When I say extended listening I really mean listening to music normally, in the real world rather than the slightly neurotic attempt to choose equipment.
I have been caught out so many times over the years by placebo and expectation bias. Only to realise after a period of time that my listening enjoyment had not been enhanced to the extent that I thought it would be by this "huge improvement "!
I had come to regard blind testing as an insurance against those effects.
Now I am not so sure. The Mojo is academic in point. I have heard it a couple of times and could discern no difference from the HO of my iPhone.
There are hundreds of posts on the Mojo thread singing its praises. Are all those people deluded? Or is it that I cannot pick up on aural cues in a "test" situation but may well come to appreciate the Mojo's qualities over time?

 

[DavidA]

Well, here we are in our nice shiny new thread!

When I say extended listening I really mean listening to music normally, in the real world rather than the slightly neurotic attempt to choose equipment.
I have been caught out so many times over the years by placebo and expectation bias. Only to realise after a period of time that my listening enjoyment had not been enhanced to the extent that I thought it would be by this "huge improvement "!
I had come to regard blind testing as an insurance against those effects.
Now I am not so sure. The Mojo is academic in point. I have heard it a couple of times and could discern no difference from the HO of my iPhone.
There are hundreds of posts on the Mojo thread singing its praises. Are all those people deluded? Or is it that I cannot pick up on aural cues in a "test" situation but may well come to appreciate the Mojo's qualities over time?

Nice new thread indeed

 
I think the adjectives used by some such as "huge difference" can mean many different things to different people.  Trying to quantify a subjective item is not possible, at least to me.
 
I got dinged for saying that I think the BH Crack is 90% of a EC Balancing act, but to me its the laws of diminishing returns, that last 10% is really going to cost you.
 
As for blind testing, one of the best example is the Pono player where the test subjects were listening to very low quality MP3 and very high quality FLAC, I'm pretty sure most of those on Head-Fi can tell the difference between 64K VBR MP3 and a 24/96 FLAC that came from a 24/96 master.  Has anyone tried to blind test 320 CBR MP3 and WAV or FLAC that was made from 16/44.1 original CD?  There are those who say its easy to tell the difference but I'm like @krismusic, its not that easy to me.
 
I have a friend who says he can tell the difference between WAV and any MP3 so I got a few songs and ripped them to WAV, 320cbr MP3, and 256vbr MP3, played back the various tracks with his equipment, AudioGD Master 9, EC Balancing Act and HD-800.  He could only notice the 256vbr files, the 320cbr and WAV were even, most of the time his guess was wrong.
 
With a blind test of DACs, it was very different, between high end DACs like MHDT Pagoda and Yggy it was easy to notice the difference between them, going down the ladder it gets much harder, between Bifrost MB and Uber, really hard to tell the difference for most people (I've tried to see if friends and family could tell the difference, 1 in 20 could but even then most say its really hard) so when someone says there is a "huge difference" between Uber and MB I would take it as that person's definition of huge is very different from mine.

 

[castleofargh]

 welcome to sound science(forcefully) ^_^.
@ColtMrFire I could take your posts so far, replace blind test by "measurement", or simply "review" and get the same result. all you're saying is that because people use different gears in different situations, results are useless. of course there can be some truth to that, but it's like asking why we're even wasting time reading anybody's opinion. when we don't know, we look for cues, and if no perfect answer seems available, we'll take what we can. isn't it what we're all doing on headfi?
 
 
about blind tests, they have one clear purpose, to test if the people tested can notice a difference. any argument about what is better or anything related to taste is potentially a misuse of blind test, and may require to prepare some very particular testing methods.  most of the people strongly opposed to blind test don't have a clue what it's for, or even how to do it right(same as with measurement haters, ignorance is too often the real fuel behind hate). tools are tools, if some use them in a stupid way, that doesn't make the tool wrong.
taste is indeed subjective, and other senses and preconceptions will really make use love something for reasons unrelated to sound. headphones do sound very different for most models, so the question isn't at all about how we can notice the differences. and in my case, when I love the sound of a few LCDs, I don't buy them because of comfort/weight. so I would find it silly not to account for that in a headphone when it's my number one factor.
it would also be very hard to make a system that would make us unable to feel what model it is, yet keep the same sound. and there is the delay to change from one to the other, we know for a fact that such delays need to be almost non existent for memory reasons. we can simulate the sound of other headphones, but perfect reproduction isn't always possible. etc.
so even with the wish for such test, not to check if there are differences, but simply to remove the price bias or some other bias, headphones aren't a simple thing to blind test. it would probably be easier to take people who aren't into the hobby and don't know the models. then lie to them about the price. if price alone really is the question that would probably work fine.

 

[ColtMrFire]

Now I am not so sure. The Mojo is academic in point. I have heard it a couple of times and could discern no difference from the HO of my iPhone.
There are hundreds of posts on the Mojo thread singing its praises. Are all those people deluded? Or is it that I cannot pick up on aural cues in a "test" situation but may well come to appreciate the Mojo's qualities over time?

 
Sampling something is different than listening over time.  Not even getting into what headphones you were using vs. what headphones others had/equipment, etc.  Too many variables involved for the comparison to be of any use.  People too easily come to conclusions about something on an objective level when the listening conditions are very subjective/don't match up.

 

[ColtMrFire]

 
With a blind test of DACs, it was very different, between high end DACs like MHDT Pagoda and Yggy it was easy to notice the difference between them, going down the ladder it gets much harder, between Bifrost MB and Uber, really hard to tell the difference for most people (I've tried to see if friends and family could tell the difference, 1 in 20 could but even then most say its really hard) so when someone says there is a "huge difference" between Uber and MB I would take it as that person's definition of huge is very different from mine.

 
I've said this before, but family and friends may not be well versed in what to listen for in differences.  To most people anything better than their iPod/laptop with mp3s on their crappy earbuds or $50 Sony over-ear, sounds amazing.  And anything above that still "sounds amazing", so to them there is no difference between a Bifrost Uber and MB, they both play music very well, and are subject to confirmation bias all the same.  But do they know what leading edge is?  Transients?  Depth, transparency, atmosphere, know the difference between bloated and tight bass?  Grain?  The difference between analytical and fun?  Their exposure to sound over the course of their lives will have an affect on what to expect.  If they're not exposed to high end equipment on a regular basis, they likely don't have the ability to spot differences in DACs. 
 
Before I bought my HD600/audioengine D1 in 2012, I was one of those people, and wouldn't be able to discern much difference between Bifrost Uber and MB, because I didn't know what to listen for.  I was just overwhelmed by the fact the music got so much better than the crappy headphones/CD players I was using.  Once you go down the rabbit hole, you are first dealing with the fact that you're in the rabbit hole and how amazing it is... you're not seeing the various shades of colors, textures on the wall of that hole, or how the air smells different in certain spots.  It's completely psychological.  Once you're in that hole for a while, and past the point of shock and awe, you start complaining that the walls look weird and want better walls, so you become a bit snobbish about it, and seek better parts of the hole... you are long past "everything is awesome", and now appreciate the subtle differences and enjoy what those differences have to offer.  Same goes with wine tasting, or any other complex hobby.  For someone who's been drinking crappy Boons Farm their entire adult lives wont be able to tell the difference between Merlot and Shiraz right off the bat, especially when they don't even know what to pay attention to.  They're both better red wine, so they're both awesome.
 
These components are made for audiophiles... people who appreciate the difference in music quality, and hence know what to listen for, since they enjoy those little differences as subtle as they maybe and are willing to spend their hard earned money on them.  What is subtle, may actually be substantial depending on the person's subjective tastes.  And your average friend/family member who is new to that rabbit hole are just happy they're there... and are not the most reliable people to use for very delicate/complex testing purposes.  You don't get your brother in law to come help design/decorate your new home, you hire an interior designer who is trained to know the subtle differences in decor that can effect mood and add up to an aesthetically satisfying whole... a Bifrost is a sophisticated piece of equipment and the different upgrades offer appreciable differences in sound quality.  While myself and those who can tell those differences may not be "experts" per se, we have enough experience with hi-fi to know, verify and appreciate those differences.
 
PS - This is in no way meant to offend your friends or family! 

 

[RRod]

  Same goes with wine tasting, or any other complex hobby.  For someone who's been drinking crappy Boons Farm their entire adult lives wont be able to tell the difference between Merlot and Shiraz right off the bat, especially when they don't even know what to pay attention to.  They're both better red wine, so they're both awesome.

 
The issue of blind vs. non-blind testing also comes up wine tasting. Oenophiles also use lots of adjectives to describe what they sense; it's when those adjectives are not consistent when they *don't* know which wine is which that it all becomes a bit hard to believe. Same with audiophiles: if not knowing if a Bifrost is multibit or not completely ruins your ability to assign adjectives, then perhaps the whole show is a bit overblown.

 

[ColtMrFire]

 
The issue of blind vs. non-blind testing also comes up wine tasting. Oenophiles also use lots of adjectives to describe what they sense; it's when those adjectives are not consistent when they *don't* know which wine is which that it all becomes a bit hard to believe. Same with audiophiles: if not knowing if a Bifrost is multibit or not completely ruins your ability to assign adjectives, then perhaps the whole show is a bit overblown.

 
Well, spending more than $100 for a DAC or amp or headphone would qualify as "overblown" to anyone not invested in the hobby.  Does that make them right?  

 

[gregorio]

  These components are made for audiophiles... people who appreciate the difference in music quality, and hence know what to listen for, since they enjoy those little differences as subtle as they maybe and are willing to spend their hard earned money on them.

 
Personally, I don't really care for that definition of audiophile. There is at least one other group of people who do not like being called audiophiles but who do appreciate the difference in music quality and know what to listen for and enjoy the subtle differences and who are willing to spend their hard earned money: The audio professionals who created the recordings in the first place! Typically those components you are talking about are not aimed at these audio pros and also typically audio pros deliberately avoid much of the equipment aimed at audiophiles. Using your wine analogy, much of what goes on in the audiophile community is not Merlot vs Shiraz, it's often effectively two identical bottles of Merlot but one has had it's label replaced by a more impressive looking one and had a significant price hike! Interestingly though, not all audiophile products are shunned by audio pros.
 

  I trust my ears.  I know placebo exists... I also know differences in gear exist.  Neither is necessarily more prevalent than the other.

 
Placebo is only one of many types of bias which affect our audio perception. Indeed, pretty much all of the commercial audio content you listen to absolutely depends on this fact. In other words, unless you are changing your "gear" virtually from second to second, ad infinitum, then audio biases are countless thousands of times more prevalent. The fact that virtually all the commercial audio content you listen to is in one way or another effectively a deliberately manufactured aural illusion of which you are at least partially, if not totally, unaware is proof that you really can't trust your ears. On the other hand, as a professional audio content creator myself, I take it as a compliment that you are largely or completely unaware that we are fooling your ears, which of course is generally our intent.
 
G

 

[ColtMrFire]

 
Personally, I don't really care for that definition of audiophile. There is at least one other group of people who do not like being called audiophiles but who do appreciate the difference in music quality and know what to listen for and enjoy the subtle differences and who are willing to spend their hard earned money: The audio professionals who created the recordings in the first place! Typically those components you are talking about are not aimed at these audio pros and also typically audio pros deliberately avoid much of the equipment aimed at audiophiles. Using your wine analogy, much of what goes on in the audiophile community is not Merlot vs Shiraz, it's often effectively two identical bottles of Merlot but one has had it's label replaced by a more impressive looking one and had a significant price hike! Interestingly though, not all audiophile products are shunned by audio pros.

 
Snake oil absolutely exists.  But so do genuine upgrades.  The trick is knowing which is which.  The problem I see is people tend to slather the entire thing as "snake oil" or vice versa.  Throwing the baby out with the bathwater is popular in our culture.  
 

 Placebo is only one of many types of bias which affect our audio perception. Indeed, pretty much all of the commercial audio content you listen to absolutely depends on this fact. In other words, unless you are changing your "gear" virtually from second to second, ad infinitum, then audio biases are countless thousands of times more prevalent. The fact that virtually all the commercial audio content you listen to is in one way or another effectively a deliberately manufactured aural illusion of which you are at least partially, if not totally, unaware is proof that you really can't trust your ears. On the other hand, as a professional audio content creator myself, I take it as a compliment that you are largely or completely unaware that we are fooling your ears, which of course is generally our intent.

 
And this is what makes blind testing even more dubious.

 

[RRod]

   
Well, spending more than $100 for a DAC or amp or headphone would qualify as "overblown" to anyone not invested in the hobby.  Does that make them right?  

 
If you can't hear a difference blind between $100 and $1000, what makes them wrong?

 

[ColtMrFire]

 
If you can't hear a difference blind between $100 and $1000, what makes them wrong?

 
My point was, it's all relative.  If you can't tell the difference, then you can't tell the difference.

 

[gregorio]

   
Snake oil absolutely exists.  But so do genuine upgrades.  The trick is knowing which is which.

 
Agreed. Although the problem I commonly see is audiophiles reporting "night and day" differences when it's physically impossible to hear even the slightest difference and even on occasion when there are no differences in the first place.
 

  And this is what makes blind testing even more dubious.

 
Why?
 
G

 

[RRod]

   
My point was, it's all relative.  If you can't tell the difference, then you can't tell the difference.

 
I can make up consistent differences all day if I know which piece of equipment is in front of me. So it's not just "can I tell a difference" it's "to what are these differences actually attributable?" If doing the comparison blind removes my ability to differentiate audio gear, then it ain't my hearing that is giving me adjectives. And at the end of the day I am 100% fine with people liking equipment for the broad "feeling" it gives them. But I'm not fine with said people dogging the abilities of others to hear "depth" or "PRAT" or whatever, when they themselves are foiled by the simple trick of "not knowing which piece of gear is which."