Lossy Audio Codec's Comparison [HUGE amount of pics] [iTunes UPDATE on p.7]

[Sir Nobax]

Since the day i started 'listening' to music instead of 'hearing' it i'm worried about the quality of lossy codecs like MP3, AAC and WMA. Every time i hear a song i wonder if it is really how it has to sound, because of the loss of data in the compression.
Of couse i could A-B every codec but still this is a very uneffective way to hear how much a file suffered from the lossy compression.

Enter The Spectrograph.

A while back i tested every major lossy codec with encoding a complex wave with an enormous amount of sweeps, because i think some of you would like to see how a MP3 compressed file really "looks". I encoded the file several times to different codecs at different bitrate's with dbPowerAmp and i made spectographs of them with an old version of CoolEdit Pro. (i will upload the sample tomorrow)

[WARNING]
All encodings including MP3 Lame (i could be totally wrong but i think it was 3.47) [UPDATE: The Lame codec is version 3.97] are mostly CBR, this was because VBR wouldnt work well with just a 6 second long sample. I cant tell how much the quality is different compared to "real" MP3 Lame encoded files with VBR.

THE ORIGINAL WAVE

Grab it HERE

AAC

HE-AAC at 96kbps Fast setting


LC-AAC at 96kbps Fast setting with PNS


LC-AAC at 96kbps Fast setting without PNS


LC-AAC at 128kbps Fast setting


LC-AAC at 128kbps Good setting (notice that the "Good" quality setting is worse then the "Fast" setting.)


LC-AAC at ~128kbps Fast setting in VBR


LC-AAC at 196kbps Fast setting


LC-AAC at 320kbps Fast setting (99% perfect copy)

MP3 Blade

MP3 Blade at 96kbps


MP3 Blade at 128kbps


MP3 Blade at 192kbps


MP3 Blade at 256kbps


MP3 Blade at 320kbps (almost as good as perfect)

MP3 Lame

MP3 Lame at 96kbps


MP3 Lame at ~128kbps in ABR (average bitrate)


MP3 Lame at 128kbps


MP3 Lame at 192kbps


MP3 Lame at 256kbps


MP3 Lame at 320kbps (in CBR at 320kbps Lame is worse then Blade, i'm sure that a ~320kbps VBR Lame encoded MP3 would be an near-perfect copy though

MP3Pro

MP3Pro at 64kbps (notice the funny methode to compress tones above 6khz, it causes a lot of jittering and ringing but sounds better then a normal 64kbps MP3 encoding.)


MP3Pro at 96kbps (this encoding is very bad, still the same amount of jittering and ringing as 64kbps, HE- and LC-AAC sounds way better at this bitrate)

.. 30 pic limit reached ...

i love you

[Sir Nobax]

More Pics

OGG Vorbis

OGG at 96kbps (some strange thing happens everytime i encode a OGG file, especially high bitrate ones, the bitrate is WAY lower then others, i sounds fine but the graphs shows its still flawed, probably OGG is forcing VBR)


OGG at 128kbps



OGG at 192kbps



OGG at 256kbps



OGG at 350kbps



OGG at 500kbps (with this file you can see that this can never be 500kbps, the filesize was 170KB, the 320kbps AAC was 190KB, confirming that OGG forces VBR)

WMA 9.1

WMA 9.1 at 96kbps


WMA 9.1 at 128kbps



WMA 9.1 at 256kbps

WMA 9.2

WMA 9.2 at 96kbps


WMA 9.2 at 128kbps


WMA 9.2 at 192kbps


WMA 9.2 at 256kbps


WMA 9.2 at 320kbps (compare this to any other 320kbps file, except OGG, and you'll see that this is really bad)

WMA 10 Pro

WMA 10 Pro at 96kbps, the best 96kbps file


WMA 10 Pro at 128kbps, nothing special


WMA 10 Pro at 192kbps


WMA 10 Pro at 256kbps, this just amazed me, near perfect at 256kbps, trully amazing.



Well this is it so far, i hope you guys enjoy watching the spectographs and get an impression of the quality between the codecs.

And again i think it is a shame that LAME could'nt be encoded this way. Maybe i'll try it again with the newer versions.

If anyone wich to request an codec, just ask.

Originally Posted by Sir Nobax

If anyone wich to request an codec, just ask.

Thank you so much for your time Actually I have one request. AAC-LC @224 with CBR and VBR, since that seems to be the limit before the ipod buffer overfills and there is hard disk interaction.

[*645-k701]

Thats friggin AWSOME..
massive thanks for doing this..
It's neat,explanitory.. you've outdone yourself..

2 (tup's)

[Sir Nobax]

Originally Posted by hYdrociTy

i love you

OMG, that just made me feel all warm and fuzzy inside.



I'm going to reinstall all my stuff and i'll post the results for the AAC-LC @ 224 with CBR and VBR file.

[jinp6301]

Originally Posted by Sir Nobax

OMG, that just made me feel all warm and fuzzy inside.



I'm going to reinstall all my stuff and i'll post the results for the AAC-LC @ 224 with CBR and VBR file.

wow, just wow

thank you so much for dong this

[db597]

If only we could do a similar test for headphones - e.g. feed in a reference signal and then record the output with a mic. Even on the best cans, I bet the graphs would be pretty bad with lots of holes.

Such a test would be hard though - even the microphone would create imperfections in the graph of it's own?

Ps. the graphs are good for technicals. But the reason compression works is because the human ear isn't sensitive to the missing black holes in certain places. A graph full of black holes doesn't mean it will sound bad.

[Febs]

Originally Posted by Sir Nobax

Of couse i could A-B every codec but still this is a very uneffective way to hear how much a file suffered from the lossy compression.

Actually, that is the most effective way to hear how much a file suffered from lossy compression.

These spectrographs are interesting to look at, but as I mentioned in the other thread where you brought this up, they really don't give you any useful information about how the lossy codec sounds.

I think that it is a mistake to look at a spectrograph, see that a codec leaves out significant amount of data, and decide from that spectrograph that the codec sounds bad. I would expect the opposite to be true. All other things being equal, wouldn't the best codec be the one that leaves out that most data and still sounds indistinguishable from the original?

[Genocide]

ncie graphic done there

[zip22]

Originally Posted by Febs

Actually, that is the most effective way to hear how much a file suffered from lossy compression.

These spectrographs are interesting to look at, but as I mentioned in the other thread where you brought this up, they really don't give you any useful information about how the lossy codec sounds.

I think that it is a mistake to look at a spectrograph, see that a codec leaves out significant amount of data, and decide from that spectrograph that the codec sounds bad. I would expect the opposite to be true. All other things being equal, wouldn't the best codec be the one that leaves out that most data and still sounds indistinguishable from the original?

my thoughts exactly. how is looking at pictures more effective than listening when you are trying to judge the quality of an audio reproduction? now, if you would ABX and then explain your findings with visual representations, that would be useful. but simply attempting to judge based on the images is illogical. this line points out the flaw in your method

(notice the funny methode to compress tones above 6khz, it causes a lot of jittering and ringing but sounds better then a normal 64kbps MP3 encoding.)

[Gurra1980]

Originally Posted by Febs

Actually, that is the most effective way to hear how much a file suffered from lossy compression.

These spectrographs are interesting to look at, but as I mentioned in the other thread where you brought this up, they really don't give you any useful information about how the lossy codec sounds.

I think that it is a mistake to look at a spectrograph, see that a codec leaves out significant amount of data, and decide from that spectrograph that the codec sounds bad. I would expect the opposite to be true. All other things being equal, wouldn't the best codec be the one that leaves out that most data and still sounds indistinguishable from the original?

Even if AAC leaves a lot more of data at 128Kbps than Mp3 at 128Kbps the AAC is not a bigger file, I guess this test is about the best codec takes away the most unhearable sounds in music or just not messes up the signal.

One thing that i didn't understand was that WMA9.2 was worse than AAC I can hear the difference from AAC and lossless at much higher bitrates than WMA9.2.

What program did you use to encode with? Did you use WMP11? no you could not have done that because it can't encode higher than 192kbps.

Try to do it with WMP11.

[mofonyx]

Could we do one with .flac just for the sake of comparison?

This is cool stuff, I like. And I wonder how people can actually distinguish 320kbps with lossless, or not.. (placebo.~)

[Filburt]

Originally Posted by Sir Nobax

Of couse i could A-B every codec but still this is a very uneffective way to hear how much a file suffered from the lossy compression.

Based on this statement, it seems as though you don't understand the nature of lossy encoding. It's not designed to be lossy in a perceptual sense; merely in a quantitative sense. That's the whole point of psychoacoustic modeling and its application in the form of so-called 'lossy' encoding. What your charts show is the impact of each codec's psy model quantitatively, but the charts don't show the qualitative impact on their own, and only arguably do so in concert with a working understanding of psychoacoustics and ability to analyze this data through that knowledge and framework.

When you set the bitrate of any of these codecs, you have a determined, finite bandwidth with which the data is represented. Analyzing the data in this manner (a particular frequency-domain analysis) only even gives you part of the quantitative story, let alone qualitative. A great example of the fallacy demonstrated here was old analyses of VQF (a vector quantization based codec developed by NTT and Yamaha) where 96kbps VQF was claimed to exceed the quality of MP3 at 128kbps and beyond (sometimes going so far as claiming better than 192+kbps), and then data was supplied via a spectrogram. I tried VQF, and the qualitative performance of it was relatively poor, even against MP3 at that time (which was in a substantially less refined state compared to today). I suspect there was a substantial amount of quantization noise, based on the overall sound, and it also seemed to lose a lot of positional information...perhaps due to lousy VQ.

The charts are interesting, but I do not recommend choosing a codec based on their superficial appearance. Blade MP3, unless it has been substantially updated and improved in a manner I'm completely unaware of (as far as I know, development was halted in 2002), is not of the refinement of the LAME mp3 encoder and I would not use it in place of LAME even at 320kbps.

[Sparky191]

Another vote for FLAC and how about ATRAC 256/292/352 and lossless.

I'm not entirely sure what I'm looking at in these pictures. Anyone care to explain one of these images?