Blind test comparison of Magni 3, uDac and Yamaha stereo headphone amplifiers
post-14536838
Thread Starter
Post #1 of 11

eucariote

1000+ Head-Fier
Joined
Oct 15, 2009
Messages
1,474
Reaction score
72
Joined
Oct 15, 2009
Posts
1,474
Likes
72
People have been arguing about amps for a while, whether they all sound the same so long as they don't suck and add distortion (a 'wire with gain') or if better designed, more powerful amps really do sound better. With the help of a friend, I ran blind tests of three different headphone amplifiers- a Magni 3 (160mW @ 300 Ω ), an original uDac (4.3mW @ 300 Ω ) and a Yamaha HTR-5630 integrated amplifier (?) to see if we could hear a difference without the help of sight and expectation. We used HD6XX headphones (300 Ω ), since they would be less susceptible to damping due to the probably high output impedance of the Yamaha.

We used hypothesis testing, where you assume the null hypothesis, that experimental outcomes are random, unless you can show that the outcomes are so consistent that they would be very rare if they were random, occurring below a 5% threshold (P < 0.05) assuming a probabilistic distribution of random outcomes, here a binomial for two correct/incorrect choices. In that case you reject the null hypothesis and accept the experimental hypothesis that there is a real systematic effect (choice) due to an independent variable (amp).

The first test was a blind comparison of the Magni vs the uDac. Sighted, I swear that the Magni has noticeably more slam, soundstage, plankton, I love this thing. I had my back turned to the gear while my friend flipped a coin, unplugged the headphones and plugged them in the Magni (heads) or the uDac (tails). I took all the time I wanted to guess which amp I was listening to, but usually guessed after 5-10 seconds. Since the Magni makes the sound of a resonant box when you plug in headphones, we used a short headphone adapter cord for it (picture). The song was a lossless recording of Olé by John coltrane, which has fairly crisp and constant bass, drums and saxophone to hear all the frequencies. The uDac was the dac for both amps and they were volume matched with pink noise.

In 30 trials, I correctly guessed the amplifier I was hearing 14 times (P = 0.71). For the second test, we switched roles and my friend correctly guessed the amplifier 13 times (P = 0.82). For the third test, I tried to hear the difference between the Magni and the Yamaha. Since they have different inputs we had to use Spotify/Chromecast for the Yamaha, which has its own dac and a matching 256 kbps file for the Magni/uDac and used headphone extension cords for both amps since both make a distinct plug/unplug sound. The test song was M83's Skin Of The Night since it gives me chills when listening on the Yamaha because of its wider, deeper (sighted) sound. In 25 trials, I correctly guessed the amp 15 times (P = 0.21). If these last results were consistent for ~105 trials we might have reached P < 0.05 but there would be a 55% chance of a type 2 error (a false positive).

The null hypothesis was not rejected for any of the three tests (P > 0.05) so in conclusion, neither of us could hear a difference between any of the three amps when relying on hearing alone. It might be the case that with different songs, headphones or better listening ability the results might have been more consistent. On that last point I consider myself to have pretty experienced ears, owning a massive library of lossless music files and having owned/heard many (100+) expensive headphones, dacs and amps. Even though I literally hear differences when not blinded, more than half the inputs to primary auditory cortex come from within the brain and not the cochlea, so in this case it is almost certain that vision and expectation, and not sound alone are making the Yamaha and Magni sound better.

Edit- TL;DR- a buddy and I listened to three amps without seeing what we were listening to and we could not tell them apart at all. Their sound is so similar that any difference is very very subtle to the point of being imperceptible.
 
Last edited:
post-14537523
Post #2 of 11

castleofargh

Sound Science Forum Moderator
Joined
Jul 2, 2011
Messages
9,086
Reaction score
4,425
Joined
Jul 2, 2011
Posts
9,086
Likes
4,425
doesn't seem unreasonable under such conditions. one thing to point out is that the lack of a rapid switching option decreased your ability to notice differences. doesn't mean you would have passed with rapid switching instead of unplugging cables, but the possibility is there.
 
     Share This Post       
  • Like
Reactions: trellus
post-14537599
Post #3 of 11

bigshot

Headphoneus Supremus
Joined
Nov 16, 2004
Messages
21,584
Reaction score
3,628
Location
Hollywood USA
Joined
Nov 16, 2004
Location
Hollywood USA
Posts
21,584
Likes
3,628
Website
www.facebook.com
Your verbiage is impressive.
 
     Share This Post       
post-14543300
Post #4 of 11

Jaywalk3r

100+ Head-Fier
Joined
Jan 15, 2012
Messages
418
Reaction score
50
Location
Seattle
Joined
Jan 15, 2012
Location
Seattle
Posts
418
Likes
50
I applaud your methodology and write-up. I'm not surprised by your results.
 
     Share This Post       
  • Like
Reactions: eucariote
post-14543832
Post #5 of 11

eucariote

1000+ Head-Fier
Joined
Oct 15, 2009
Messages
1,474
Reaction score
72
Joined
Oct 15, 2009
Posts
1,474
Likes
72
I applaud your methodology and write-up. I'm not surprised by your results.
Thanks very much! It was really fun, took about an hour and afterwards we had beers. If anyone wants to try their own test I'd be happy to help with the stats and methodology. The more objective data the better.
 
Last edited:
     Share This Post       
post-14544487
Post #6 of 11

Jaywalk3r

100+ Head-Fier
Joined
Jan 15, 2012
Messages
418
Reaction score
50
Location
Seattle
Joined
Jan 15, 2012
Location
Seattle
Posts
418
Likes
50
Thanks very much! It was really fun, took about an hour and afterwards we had beers. If anyone wants to try their own test I'd be happy to help with the stats and methodology. The more objective data the better.
Stats are my wheelhouse. I doubt I could explain the key concepts for hypothesis testing to a laymen audience as clearly and concisely as you did or without sounding condescending. (To be clear, you did not sound condescending.) It's a struggle, professionally. I'm envious!
 
     Share This Post       
post-14544998
Post #7 of 11

castleofargh

Sound Science Forum Moderator
Joined
Jul 2, 2011
Messages
9,086
Reaction score
4,425
Joined
Jul 2, 2011
Posts
9,086
Likes
4,425
Stats are my wheelhouse. I doubt I could explain the key concepts for hypothesis testing to a laymen audience as clearly and concisely as you did or without sounding condescending. (To be clear, you did not sound condescending.) It's a struggle, professionally. I'm envious!
the issue is that showing to someone who doesn't know that he doesn't know, is seen as condescending nowadays. and with statistics, as a vast majority of people don't know or at least struggle with the ideas, of course you're going to annoy/offend many when discussing the subject. not sure you can do much about it. IMO the fault is on people who want to be treated like they're expert at everything.
just using percentages is apparently a solid hurdle for general comprehension https://arstechnica.com/science/2018/10/fixed-mindsets-might-be-why-we-dont-understand-statistics/ so I can't imagine how many people we lose/offend when we mention P values. even in something as basic as coin flipping, which is really all we need here for most audibility tests.
but the situation is even worst for us than as described in the link. because a great many audiophiles don't understand much about stats, but also don't understand why a statistical approach is relevant in the first place. which is a different can of worms where the potential to be condescending is over 9000. :sweat_smile: even for me who's complete garbage at handling stats.
 
     Share This Post       
post-14546110
Post #8 of 11

Niouke

100+ Head-Fier
Joined
Feb 22, 2017
Messages
403
Reaction score
84
Joined
Feb 22, 2017
Posts
403
Likes
84
excellent, thank you for the time spent
 
     Share This Post       
post-14546289
Post #9 of 11

bigshot

Headphoneus Supremus
Joined
Nov 16, 2004
Messages
21,584
Reaction score
3,628
Location
Hollywood USA
Joined
Nov 16, 2004
Location
Hollywood USA
Posts
21,584
Likes
3,628
Website
www.facebook.com
Math is hard!

But numbers won't help when someone just doesn't want to know the truth. A lot of people online have bubbles built around themselves. They just want to speak for their own benefit and not be questioned. They see the internet as their own personal echo chamber of self validation, and any attempt to communicate with them is seen as an attack and they get all huffy and threatened. Internet forums seem to attract a certain personality type. It isn't the real world. You can't behave like that in the real world without the real world kicking you in the butt on a regular basis.
 
Last edited:
     Share This Post       
post-14549663
Post #11 of 11

eucariote

1000+ Head-Fier
Joined
Oct 15, 2009
Messages
1,474
Reaction score
72
Joined
Oct 15, 2009
Posts
1,474
Likes
72
Hi! How exactly do you match volume? Can you do or refer to some step-by-step guide?
Hi, I rested the microphone of an LG cell phone in one of the ear cups and used the Android 'Sound Meter' app. After finding a comfortable music volume on the uDac, I nestled the phone in the ear cup, played pink noise, measured the dB value, plugged the headphones into the Magni without disturbing the headphones and adjusted the Magni's volume until it reached the same dB. Uncalibrated and cheap but good enough for roughly matching the volume. There were no noises in the house, the fridge and heat pump were turned off and I live on a very quiet street.
 
     Share This Post       
  • Like
Reactions: ChikiChpoki

Users Who Are Viewing This Thread (Users: 0, Guests: 1)

Top