1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.

    Dismiss Notice

Blind test comparison of Magni 3, uDac and Yamaha stereo headphone amplifiers

Discussion in 'Sound Science' started by eucariote, Oct 13, 2018.
  1. eucariote
    People have been arguing about amps for a while, whether they all sound the same so long as they don't suck and add distortion (a 'wire with gain') or if better designed, more powerful amps really do sound better. With the help of a friend, I ran blind tests of three different headphone amplifiers- a Magni 3 (160mW @ 300 Ω ), an original uDac (4.3mW @ 300 Ω ) and a Yamaha HTR-5630 integrated amplifier (?) to see if we could hear a difference without the help of sight and expectation. We used HD6XX headphones (300 Ω ), since they would be less susceptible to damping due to the probably high output impedance of the Yamaha.

    We used hypothesis testing, where you assume the null hypothesis, that experimental outcomes are random, unless you can show that the outcomes are so consistent that they would be very rare if they were random, occurring below a 5% threshold (P < 0.05) assuming a probabilistic distribution of random outcomes, here a binomial for two correct/incorrect choices. In that case you reject the null hypothesis and accept the experimental hypothesis that there is a real systematic effect (choice) due to an independent variable (amp).

    The first test was a blind comparison of the Magni vs the uDac. Sighted, I swear that the Magni has noticeably more slam, soundstage, plankton, I love this thing. I had my back turned to the gear while my friend flipped a coin, unplugged the headphones and plugged them in the Magni (heads) or the uDac (tails). I took all the time I wanted to guess which amp I was listening to, but usually guessed after 5-10 seconds. Since the Magni makes the sound of a resonant box when you plug in headphones, we used a short headphone adapter cord for it (picture). The song was a lossless recording of Olé by John coltrane, which has fairly crisp and constant bass, drums and saxophone to hear all the frequencies. The uDac was the dac for both amps and they were volume matched with pink noise.

    In 30 trials, I correctly guessed the amplifier I was hearing 14 times (P = 0.71). For the second test, we switched roles and my friend correctly guessed the amplifier 13 times (P = 0.82). For the third test, I tried to hear the difference between the Magni and the Yamaha. Since they have different inputs we had to use Spotify/Chromecast for the Yamaha, which has its own dac and a matching 256 kbps file for the Magni/uDac and used headphone extension cords for both amps since both make a distinct plug/unplug sound. The test song was M83's Skin Of The Night since it gives me chills when listening on the Yamaha because of its wider, deeper (sighted) sound. In 25 trials, I correctly guessed the amp 15 times (P = 0.21). If these last results were consistent for ~105 trials we might have reached P < 0.05 but there would be a 55% chance of a type 2 error (a false positive).

    The null hypothesis was not rejected for any of the three tests (P > 0.05) so in conclusion, neither of us could hear a difference between any of the three amps when relying on hearing alone. It might be the case that with different songs, headphones or better listening ability the results might have been more consistent. On that last point I consider myself to have pretty experienced ears, owning a massive library of lossless music files and having owned/heard many (100+) expensive headphones, dacs and amps. Even though I literally hear differences when not blinded, more than half the inputs to primary auditory cortex come from within the brain and not the cochlea, so in this case it is almost certain that vision and expectation, and not sound alone are making the Yamaha and Magni sound better.

    Edit- TL;DR- a buddy and I listened to three amps without seeing what we were listening to and we could not tell them apart at all. Their sound is so similar that any difference is very very subtle to the point of being imperceptible.
    Last edited: Oct 18, 2018
    Philimon, Joe Bloggs, Niouke and 8 others like this.
  2. castleofargh Contributor
    doesn't seem unreasonable under such conditions. one thing to point out is that the lack of a rapid switching option decreased your ability to notice differences. doesn't mean you would have passed with rapid switching instead of unplugging cables, but the possibility is there.
    trellus likes this.
  3. bigshot
    Your verbiage is impressive.
  4. Jaywalk3r
    I applaud your methodology and write-up. I'm not surprised by your results.
    eucariote likes this.
  5. eucariote
    Thanks very much! It was really fun, took about an hour and afterwards we had beers. If anyone wants to try their own test I'd be happy to help with the stats and methodology. The more objective data the better.
    Last edited: Oct 17, 2018
  6. Jaywalk3r
    Stats are my wheelhouse. I doubt I could explain the key concepts for hypothesis testing to a laymen audience as clearly and concisely as you did or without sounding condescending. (To be clear, you did not sound condescending.) It's a struggle, professionally. I'm envious!
  7. castleofargh Contributor
    the issue is that showing to someone who doesn't know that he doesn't know, is seen as condescending nowadays. and with statistics, as a vast majority of people don't know or at least struggle with the ideas, of course you're going to annoy/offend many when discussing the subject. not sure you can do much about it. IMO the fault is on people who want to be treated like they're expert at everything.
    just using percentages is apparently a solid hurdle for general comprehension https://arstechnica.com/science/2018/10/fixed-mindsets-might-be-why-we-dont-understand-statistics/ so I can't imagine how many people we lose/offend when we mention P values. even in something as basic as coin flipping, which is really all we need here for most audibility tests.
    but the situation is even worst for us than as described in the link. because a great many audiophiles don't understand much about stats, but also don't understand why a statistical approach is relevant in the first place. which is a different can of worms where the potential to be condescending is over 9000. :sweat_smile: even for me who's complete garbage at handling stats.
  8. Niouke
    excellent, thank you for the time spent
  9. bigshot
    Math is hard!

    But numbers won't help when someone just doesn't want to know the truth. A lot of people online have bubbles built around themselves. They just want to speak for their own benefit and not be questioned. They see the internet as their own personal echo chamber of self validation, and any attempt to communicate with them is seen as an attack and they get all huffy and threatened. Internet forums seem to attract a certain personality type. It isn't the real world. You can't behave like that in the real world without the real world kicking you in the butt on a regular basis.
    Last edited: Oct 18, 2018
  10. ChikiChpoki
    Hi! How exactly do you match volume? Can you do or refer to some step-by-step guide?
  11. eucariote
    Hi, I rested the microphone of an LG cell phone in one of the ear cups and used the Android 'Sound Meter' app. After finding a comfortable music volume on the uDac, I nestled the phone in the ear cup, played pink noise, measured the dB value, plugged the headphones into the Magni without disturbing the headphones and adjusted the Magni's volume until it reached the same dB. Uncalibrated and cheap but good enough for roughly matching the volume. There were no noises in the house, the fridge and heat pump were turned off and I live on a very quiet street.
    ChikiChpoki likes this.

Share This Page