I was told by a user in the Schiit Magni thread that sound science readers might have a few questions about my review which will be below and was asked to post it here.
 
There have been a lot of questions around here about the o2 vs magni so here are my impressions of the two. Excuse the awkward transition into the article as it was written for my blog which can be found here http://musicandlistening.wordpress.com/
 
So apparently I was lying a while back when on an audio subreddit, I said that I wouldn’t do a blind abx test. To provide a bit of context, a friend of mine purchased a Schiit Magni, so we decided to compare it to the o2 since one of the things that we’ve seen around some audio sites a lot is the idea that all “transparent” gear should sound the same or at least in a blind abx test. The results were interesting.
 
As far as our method goes, we decided to make all parts of the setup identical (same set of Denon D2000 headphones, cables, and ODAC) except the Magni and O2. Then we started abx testing in two methods. For both methods, we took 30 seconds of listening in which we knew which amp we were listening to based on verbal confirmation (before this we had only spent about 15 minutes on each other’s amps). After that, we started blind testing and left the song playing pulling out the headphone cable between each declaration of an answer until 10 answers were recorded. For the first test, after each pull out of the cable at which either a change or no change could occur, we had to name which was the Magni and which was the O2 based on the 30 second preview that we had. Both of us got between 4 and 6 correct in each of our 3 sets of 10 trials. For the second method (fourth set of 10 trials), we simply had to declare if the amplifier switched. I scored a 9 out of 10 correct and my friend scored a 4 out of 10 correct.
 
I’m sure you’re saying at this point that because the results of the blind abx were inconclusive, that there is no sonic difference. That’s not the case though. During most of the trials, we both had moments in which we knew (and usually exclaimed it out loud in the form of expletives) that we had screwed up which was which and the data showed it in that once the switch got made in our mind, we guessed correctly that there was a change, but got confused on which was the O2 and which was the Magni. The other thing that we noticed was what exactly we were listening for was different — after we finished abxing we debriefed as far as what in the tracks we were listening to for the giveaways of which was which. For a certain track, one of us would say vocals and the other bass kick or one of us guitars and the other piano. Upon re-listening to the track without commenting on which differences we heard, we were able to understand where we had gone wrong. On some tracks there was no difference between the vocal presentation, but there was in the bass and vice-versa. The other problem that we noticed with abxing is the mental pressure aspect. The idea that there could be no change (and the subsequent desire to not look like an idiot by getting it wrong) created a scenario in which the test was not necessarily about what was being heard — it was about the game being played. I had many times in the test where I detected a change or no change — and then the second guessing started. Would he really put a no change there? Would he really switch? Once the doubt in what you heard becomes apparent, the awfulness of auditory memory shows up and you begin to invent differences all in the name of not of the inherent desire to not want to look as if your ears are inferior in terms of detecting differences. It’s equivalent to taking a scantron test and noticing that every single correct answer is A except you aren’t detecting differences on something reliable like knowing a fact. We found that with sighted listening, since the mental game was out of the equation, we were better able to focus on differences in general sound signature and individual details at the same time instead of attempting to focus on mind games and sound at the same time.
 
So without further hesitation, here’s what we heard during our abx trials and sighted listening. Keep in mind that these are generic impressions based on listening to both amps on the same track set of tracks and that these impressions might not apply for every single track that one listens to.
 
1) The O2 has a much cleaner presentation as far as detail retrieval
 
2) On certain songs, the Magni suffered as far as sub bass. It was almost non-existent on some tracks and on others it appeared as a fuzzy fwump noise while the O2 delivered a precise thump
 
3) The Magni’s vocals almost have a veiled warm and thick quality while the O2s were somewhat brighter
 
4) The O2 has more sparkle on instruments such as guitars and piano
 
5) The O2 has a better soundstage as far as width and better placement of things within the soundstage

Hi RAZR,
 
Thanks for linking this here. People get a little touchy when discussing ABX testing outside of the sound science forum.
 
I was wondering a few things while reading your post.
 
1) what method did you use to match the volume of the two amps?
 
2) Perhaps a slight improvement to your test methodology given the same set of resources you had available would be this.
 
letting your friend switch between amps (as quickly as possible)....
1) listen to "A"
2) listen to "B"
3) listen to "X" <- for each trial, your friend chooses the X, and sticks with that choice until the end of the trial
4) You can declare "A", "B", or "X" and your friend pulls and switches to which ever device you request. You can switch between the devices as much as you like without any sense of being rushed and without any sense of pressure that you have to choose an answer immediately. This should be very causal so you can lessen the pressure of doing the abx test.
5) after you feel like you have an answer, declare that "X is A" or "X is B" with your partner secretly recording your answer
6) repeat until statistically converged.
 
Thanks for taking the time and effort to do and share your ABX test!
Here's some beer for you and your friend:
 
Cheers!

Yup I agree with what ab initio wrote. Level matching is extremely important. Quick switching is preferable.
 
How did you decide on the tracks? How did you switch between them?
 
Switching order: use a (virtual) dice, or random function on your calculator or a smartphone app .. to make it truly random and unguessable.
Switching itself: always pull the plug (doh!) and try to take the same time to plug it into either amp. What I mean is that a switch from A to A should take the same time as A to B.
 
And to make it truly double blind you should not only not face the amps (doh!) but also look away from your friend.

Quote:

ab initio said:

Hi RAZR,
 
Thanks for linking this here. People get a little touchy when discussing ABX testing outside of the sound science forum.
 
I was wondering a few things while reading your post.
 
1) what method did you use to match the volume of the two amps?
 
2) Perhaps a slight improvement to your test methodology given the same set of resources you had available would be this.
 
letting your friend switch between amps (as quickly as possible)....
1) listen to "A"
2) listen to "B"
3) listen to "X" <- for each trial, your friend chooses the X, and sticks with that choice until the end of the trial
4) You can declare "A", "B", or "X" and your friend pulls and switches to which ever device you request. You can switch between the devices as much as you like without any sense of being rushed and without any sense of pressure that you have to choose an answer immediately. This should be very causal so you can lessen the pressure of doing the abx test.
5) after you feel like you have an answer, declare that "X is A" or "X is B" with your partner secretly recording your answer
6) repeat until statistically converged.
 
Thanks for taking the time and effort to do and share your ABX test!
Here's some beer for you and your friend:

 
Cheers!

Click to expand...

1) We didn't have an spl meter so we elected to vol match as precisely as possible by ear in the range that both amps sounded their best. Obviously not idea but that's the best that we could pull off. 
 
As for the second point, I agree that that probably would've been a better method to make things a bit more relaxed. The only thing is that for me it still relies on auditory memory to be able to keep which sound is a and which sound is b straight in your head. The other thing is that the question of whether or not we're listening to the same things in a track could provide different results for different people. Do you think it would be a more sound method to before a test establish something like "for this we're going to pay attention to sub bass or mids" and only try to pick up on differences between a, b, and x in those areas?
Quote:

xnor said:

Yup I agree with what ab initio wrote. Level matching is extremely important. Quick switching is preferable.
 
How did you decide on the tracks? How did you switch between them?
 
Switching order: use a (virtual) dice, or random function on your calculator or a smartphone app .. to make it truly random and unguessable.
Switching itself: always pull the plug (doh!) and try to take the same time to plug it into either amp. What I mean is that a switch from A to A should take the same time as A to B.
 
And to make it truly double blind you should not only not face the amps (doh!) but also look away from your friend.

Click to expand...

 
 
As far as tracks, we picked a few that were in FLAC based on a combination of our familiarity with the tracks, and variety of genres/recording styles. One track we used was by Atmosphere, another by the Dear Hunter, and the third by Bear vs Shark
 
We switched amps as quickly as possible and as far as the switching order we did it as randomly as two humans could and neither of us picked up on a pattern (as the results show)
 
And as far as the last couple things we did both of those

I kind of wonder why everybody thinks they need to volume match with an SPL meter. It's easier and more accurate to match the electrical outputs with a $5-10 multimeter (well, something more expensive works too), so there's not much excuse not to use one and do that. Okay, you might also need a splitter to make the measurements when loaded with the headphones in case there's a difference in output impedance.
 
Well, I don't mean to be flippantly dismissive, but that's how it goes. Testing like this is a decent amount of work, so thanks for sharing the results and impressions. Personally I wouldn't want to go through all the trouble just to have to put an asterisk by the results, so to speak.

Yes a DMM (meter) is the way to go.
It's doesn't even need to be accurate, as long as it's repeatable.
It's doesn't need good frequency response as long as it's repeatable. 
But most bottom dollar meters don't has a low voltage AC scale.
 
The only time you need an SPL meter is when you are comparing loudspeakers.
 
I'll skip over comparing headphones.

Or since you only have 1 headphone you can take a small styrofoam block, put a hole and a cheap mic in it, put the headphone on it and play/record something as simple as a 1 kHz tone with an onboard soundcard. Since you don't need to touch the headphone the results should be fairly precise (not accurate though, but that doesn't matter).

Quote:

xnor said:

Yup I agree with what ab initio wrote. Level matching is extremely important. Quick switching is preferable.
 
How did you decide on the tracks? How did you switch between them?
 
Switching order: use a (virtual) dice, or random function on your calculator or a smartphone app .. to make it truly random and unguessable.
Switching itself: always pull the plug (doh!) and try to take the same time to plug it into either amp. What I mean is that a switch from A to A should take the same time as A to B.
 
And to make it truly double blind you should not only not face the amps (doh!) but also look away from your friend.

Click to expand...

Agree with the above, and refining a few points...
 
David Clark's AES paper, "High Resolution Subjective Testing Using a Double-Blind Comparator", 1982, Section 3, "Maximizing Resolution", suggests such points as the test be listener controlled (meaning eliminating the "friend" operator) the time between choices must be short (Clark chose 50ms), no time limit to the test, a minimum of 12-16 individual tests for adequate statistical resolution, randomization of the X choice (essential), and the additional use of non-musical signals such as tones and noise for enhanced resolution.  Clark's ABX Comparator would have been ideal for this particular test as it handled all analog switching and X randomization.  
 
Next time someone wants to do this, PM me, perhaps we can arrange a loan of the Clark system I own, or conduct the test somewhere near me.