[goodyfresh]

   
There are a  number  of "cut to the chase" methodologies for testing DAPs. 
 
One key to effective testing is to use the best standard possible for your reference, and IMO that starts with a good clean rip of the recording being used or its equivalent.
 
IME many DAPs such as the Sansa Fuze have what I consider to be marginal performance when it comes to THD, noise, and maximum voltage output into headphone loads.
 
Ideally one likes to have reference standards that are at least from 10 to 20 dB better than the UUT.
 
Mid-priced headphone amps such as Fiio E11K  seem to measure equal or better that most DAPs  in these areas, so a .wav file played through a pro-audio interface such as a M-Audio AP24192 driving a Fiio E11K has become one of my references for evaluating DAPs. 
 
Doing an ABX test of DAPs the most obvious way would involve the use of some kind of switch box to move the connections headphones or other monitoring system between the two sources being compared.  I am fortunate to have a hardware ABX system at my disposal - the one described here:
 
http://djcarlst.provide.net/abx_hdwr.htm

Nice that you have that ABX setup, man!  Once I have more DAP's in my "collection" in the future I'll almost certainly go and get something similar.

 

[tf10charged]

very nice post.
 
after reading in page 1, saw the name Tera player. did a google and was really really surprised by the price! 

 

[goodyfresh]

  Update: Okay guys so I did alter the title of the thread to better reflect my true intent in starting it, as well as the way the thread has "evolved" over time.
 
To be honest, guys, I never expected this thread to take-off the way it did and become something so long and drawn-out with so much debate, when I first started it back at the end of August!  It has really evolved and grown beyond my expectations and my control.  Certainly interesting, to say the least

And yet  now my thread seems to have died  Haha.

 

[Sal1950]

Ya can only put so much schitt in the out house till the hole gets full. LOL

 

[castleofargh]

 

  Update: Okay guys so I did alter the title of the thread to better reflect my true intent in starting it, as well as the way the thread has "evolved" over time.
 
To be honest, guys, I never expected this thread to take-off the way it did and become something so long and drawn-out with so much debate, when I first started it back at the end of August!  It has really evolved and grown beyond my expectations and my control.  Certainly interesting, to say the least

And yet  now my thread seems to have died  Haha.

because we can only bring up specific cases, DAC X vs DAC Y. and even then what interests us are controlled tests, just this doesn't leave too many people.
then for those who end with statistical significance showing they heard a difference, we would have to check if it's more than just a low pass filter rool off. and even then, many other variables could be the reason instead of the chipset.
this would require a lot of efforts to never really get enough to draw a conclusion(except that they mostly all sound the same when they measure well).
I hoped the Shiit crew could tell us about the stuff they have tested, as they're not the shy engineer type and happen to have gears where they offer the switch between R2R and pulse modulation. but most posts seem to strongly suggest that they try different techs because they can and because it's fun. not really what you're asking about ^_^.

 

[goodyfresh]

  because we can only bring up specific cases, DAC X vs DAC Y. and even then what interests us are controlled tests, just this doesn't leave too many people.
then for those who end with statistical significance showing they heard a difference, we would have to check if it's more than just a low pass filter rool off. and even then, many other variables could be the reason instead of the chipset.
this would require a lot of efforts to never really get enough to draw a conclusion(except that they mostly all sound the same when they measure well).
I hoped the Shiit crew could tell us about the stuff they have tested, as they're not the shy engineer type and happen to have gears where they offer the switch between R2R and pulse modulation. but most posts seem to strongly suggest that they try different techs because they can and because it's fun. not really what you're asking about ^_^.

I'm determined though.  Someday when I have the cash, I'm going to do whatever it takes to get the gear needed to accurately test these questions!!!!  It won't be for years though >_<

 

[Sal1950]

  I'm determined though.  Someday when I have the cash, I'm going to do whatever it takes to get the gear needed to accurately test these questions!!!!  It won't be for years though >_<

By then it will all be as obsolete as a Edison cylinder player.  LOL

 

[castleofargh]

 

  I'm determined though.  Someday when I have the cash, I'm going to do whatever it takes to get the gear needed to accurately test these questions!!!!  It won't be for years though >_<

By then it will all be as obsolete as a Edison cylinder player.  LOL

any time cylinders are mentioned, I have a thought for that poor fellow.

 

[moshen]

I have spent a whole lot of time trying to blind A/B identify closely volume matched Bifrost Multibit & my ODAC (many hours per day for days) with my HD800. I couldn't ever get better than random. It could also mean I just have really bad ears. Either way it put an end to my DAC upgrade craving.

 

[landroni]

  I have spent a whole lot of time trying to blind A/B identify closely volume matched Bifrost Multibit & my ODAC (many hours per day for days) with my HD800. I couldn't ever get better than random. It could also mean I just have really bad ears. Either way it put an end to my DAC upgrade craving.

Did you try Mike Moffat's long-term blind A/B testing method? There are good reasons to avoid instantaneous, back-and-forth switching of DACs as such a workflow is unnecessarily tiring for the human brain. In my opinion instantaneous blind A/B testing in audio is generally biased towards a "no difference" result (and you can apply it to just about everything, whether amps, DACs, formats, cables, etc.), whereas long-term blind testing allows the ear (read: the brain) to fully appreciate all aspects of the sound reproducing system. Try listening to one album at a time (while in a blind testing setting) and see if you can perceive a difference.
 
Check out Moffat's experiences on the subject:
http://www.head-fi.org/t/701900/schiit-happened-the-story-of-the-worlds-most-improbable-start-up/7725#post_11921090
http://www.head-fi.org/t/748067/official-schiit-magni-modi-2-uber-thread/1065#post_11933232

 

[landroni]

Here are some relevant comments by Robert Ludwig on the subject of (instantaneous) A/B testing:
http://tapeop.com/interviews/105/bob-ludwig/

"[Q:] You once said that today's converters, with great clocking, cannot be differentiated from the analog source by anyone you've tested.
 
 
[A:] I'm not saying that no one can ever hear the difference, I'm merely saying when someone comes into the studio for a quick visit and I play the source vs. high resolution digital, a 96 kHz, 192 kHz, or DSD copy, no one can immediately pick out the difference. Don't forget, these are all awesome converters. The quality of the engineering of the analog-to-digital converter and DAC is much, much more important to the musicality of the sound than the sampling rate could ever be. Our $8,000 converters at 16-bit/44.1 kHz sound way, way better than a 192 kHz playback from a $5 chip on a DVD-Audio player. I think the higher resolution sounds reveal themselves not in A/B testing, but in long periods of time. Play an entire album in a relaxed atmosphere at 96 kHz/24-bit, then, at the end, listen to it at 44.1 kHz/16-bit, and you'll get it right away. A/B testing, while the only scientific method we have, does not reveal too much with short-term back-and-forth comparisons due to the anxiety the brain is under doing such a test. The brain becomes very left-brain-technical, rather than right-brain creative and musical." (emphasis mine)


One may think of instantaneous A/B testing as a (stressful) exam setting, which may dull or otherwise overpower senses, or confuse the brain. Anyone recall that test where you knew that that you knew the theorem, but couldn't recall it because of high anxiety? This factor can be alleviated with longer-term listening sessions.

 

[mcteague]

RE: long term listening. Why is it most people can hear differences, when they know which is being listened to, with very short sessions? Only blind ones seem to cause this confusion and stress. Interesting.
 
Tim

 

[RRod]

Originally Posted by landroni /img/forum/go_quote.gif
 
One may think of instantaneous A/B testing as a (stressful) exam setting, which may dull or otherwise overpower senses, or confuse the brain. Anyone recall that test where you knew that that you knew the theorem, but couldn't recall it because of high anxiety? This factor can be alleviated with longer-term listening sessions.
 

 
There are no time limits to the ABX protocol that I know of. So a long-term-listening advocate can just do his 2 hours session on A, then do a switch to X and decide of he hears a change and make his guess. He can then do this guess-at-the-end for 10 sessions then look at the results. It's a test you can be pretty chill about. So go ahead and try it, hi-res advocates.

 

[landroni]

  RE: long term listening. Why is it most people can hear differences, when they know which is being listened to, with very short sessions? Only blind ones seem to cause this confusion and stress. Interesting.

My intuition is that when listening to two sources, the brain "adjusts" to the way it processes sonic information. Think of it as when you're listening to two persons tell you a given phrase in English, but with two different accents. With sighted listening, the brain can quickly adjust to the incoming stream of sounds (you know what "accent" flavor to expect, and focus accordingly). With blind listening, the brain must first decide which flavor it is dealing with, before being able to "optimally" process incoming sounds. With long-term listening you have the time to do this, but not so with instantaneous tests.

 

[landroni]

 
 
There are no time limits to the ABX protocol that I know of. So a long-term-listening advocate can just do his 2 hours session on A, then do a switch to X and decide of he hears a change and make his guess. He can then do this guess-at-the-end for 10 sessions then look at the results. It's a test you can be pretty chill about. So go ahead and try it, hi-res advocates.

Sure, but in most instances only short listening tests are being practised, not least for convenience reasons. It surely is much easier to perform an experiment with two dozen individuals during two days using 10sec or 1 song (i.e. short) sessions. Often such sessions are used to confirm the "no huge differences" or "no differences" POV. But this is not how people listen to their gear at home (from where claims of differences will originate), where long-term listening (at least 1h) on a given gear will often be favored.