ABX Test of 320kbps VS FLAC - Results
Jun 27, 2009 at 6:40 AM Thread Starter Post #1 of 66

Berlioz

100+ Head-Fier
Joined
Apr 8, 2009
Posts
249
Likes
14
I realize that there is another thread just like this one in this sub forum, but I didn't want to hijack the thread, and I was curious enough to do my own test.

The structure of the test went like this:

I used the ABX Comparator on foobar2000. It is an optional utility during installation of the program.

How it works:

You choose two files to test. You can pick what section of the song to examine, and how short it is etc.

There are two sets of two choices each: [A and B] & [X and Y]

You can compare any of them to one another at any time, as many times as you like. Basically, you need to match them up in pairs.

One of the [A and B] choices will be lossless, and one will be the 320kbps version.
Same thing goes for the [X and Y] choices.

I would compare the A and B tracks, and decide which one sounded better. Then I would do the same with the X and Y tracks. If B sounded better than A, and X sounded better than Y, I would pick the option "B is X".

Sorry if that's a little confusing for some people; that's the best way I can word it.

I chose four pieces:

1) Symphony No. 1 by Dmitri Shostakovitch - As with all of his work, this dynamic piece is packed with energy. I was curious if I could separate the two versions based on transients.

2) Gold Rush by Dragonette - Electro Pop that is full of compressed sounds, particularly in the low range. I thought it would be a good test of the type of mastering that most people listen to on the radio.

3)
Rambaldo!...Ah! M'aiutate! - A section of an Italian Opera La Rondine by Giacomo Puccini. Vocals and the backing orchestra constantly strive for top spot across the entire piece. This recording won the 1997 Gramophone Award for "Record of the Year".

4) Sunshine of Your Love by Cream - This song got me into music in the first place, so I couldn't leave it out! Clapton's riffs are overdriven yet articulate at the same time. Coupled with his vocals, it's another good test of dynamics in my opinion.

Preparation:


For equipment, I was simply using my Shure SE530's connected to the headphone jack on my computer.

All of the tracks were copied twice using EAC. Once in a 320 kbps .mp3 format using the LAME encoder. The second time, in .flac using the FLAC encoder bundled with EAC.

There would be 40 replicates of each test, per song, so 160 trials altogether. I could compare the tracks as many times as I liked, and I took breaks about every 10 trials (trust me, I needed them!).

All of the MP3's are compressed to have a bitrate of 320 kbps.
The bitrates for the FLAC files are as follows:

Symphony No. 1 - 427 kbps
Gold Rush - 1001 kbps
Rambaldo!...Ah! M'aiutate! - 628 kbps
Sunshine of Your Love - 818 kbps

Foobar2000 also calculates your probability of randomly guessing and getting the same results,although it only goes to 1 decimal place. To calculate the values on your own, with more decimal places, use the binomial probability equation.

Results:

Symphony No. 1 -

Correctly differentiated them 28/40 times. Probability of guessing: 0.8%

I was very pleased with this result. It was extremely hard to differentiate the selections, even though I focused on a variety of sections of the song. The least complex sections that I tested gave me the most trouble.

Gold Rush

Correctly differentiated them 18/40 times. Probability of guessing: 78.5%

This made me sad. I have always sworn that Martina Sorbaras voice sounded sexier on FLAC. Oh well. I honestly couldn't make out any difference between the two versions. About a dozen or so replicates in I was already getting discouraged. It was impossible to hear any difference.

Rambaldo!...Ah! M'aiutate!

Correctly differentiated them 22/40 times. Probability of guessing: 31.8%

I had high hopes for this one. At this point, I began to realize what others have been saying. I tried as hard as I could, but I could not discern any differences whatsoever. I would hear A to have crisper vocals than B, and hear the same quality in Y. I would pair A up with Y....and be wrong!
Oh well, there's still another test left!

Sunshine of Your Love


Correctly differentiated them 24/40 times. Probability of guessing: 13.4%

Again, it was impossible to tell the difference. I took a lot of breaks, and felt well rested, but it seemed to make no difference.

Conclusion:

Only one of the results had enough of a difference to be statistically significant. In doing this test, I discovered just how close 320kbps and flac really are. Even with Symphony No.1, I was focusing on the music so ****ing hard, to the point where it wasn't even enjoyable anymore.

It's important to note, that this is only one set of tests, and really doesn't mean anything in the big picture. Maybe my hearing sucks. Maybe my equipment is so poor I can't hear all the little nuances.

However, the fact alone that it was so hard to tell them apart in the first place...there's not nearly as much difference between the two as I found in sighted "tests".

I take back any statements I have ever made as to whether I can hear a difference between FLAC and 320kbps MP3s. I didn't know what the hell I was talking about. It's funny, because as soon as I play them again the .flac one sounds so much crisper. But I know it's all in my head now.

I hope that anyone reading this enjoyed my experiment, and maybe you'll be curious enough to do a test of your own.

Any comments/questions are welcome.
 
Jun 27, 2009 at 7:54 AM Post #2 of 66
Nice write-up.
smily_headphones1.gif

But to me this result mean that you can indeed hear a difference between 320kbps and lossless.
Quote:

Originally Posted by Berlioz /img/forum/go_quote.gif
Symphony No. 1 -

Correctly differentiated them 28/40 times. Probability of guessing: 0.8%



But of course not on all kind of music/samples, and you need full focus to differentiate them.
 
Jun 27, 2009 at 7:58 AM Post #3 of 66
Oh very true. However, I was straining incredibly hard to focus. If I was in my music listening mood(I'm always in this mood!), it's unlikely that I would be doing this. It wasn't exactly enjoyable.

Now, I still believe it's possible to discern 320kbps from lossless. However, the difference I experienced between the two was very small. That's what really got me. Before, I could hear huge differences between them. I'd play my FLAC files and think to myself, "Yeah, that's why I spent all that time burning those CD's." It was blissful. In reality though, they are almost exactly the same, at least to my ears - which, at least according to those hearing tests someone posted - are pretty good.

When I get my home set up complete, I'll test it again. I expect it to be easier to differentiate the two, using better equipment, but it should still be pretty hard. Harder than I thought anyways.
 
Jun 27, 2009 at 8:21 AM Post #4 of 66
Fair enough!
In most cases you are not able to hear and audible difference, hence why the increased size may not be worth it to you.
 
Jun 27, 2009 at 2:50 PM Post #5 of 66
LAME is good stuff. Most of the problems I've heard with mp3's have been with stuff encoded by WMP (Fraunhoffer?) at 320 CBR.

With LAME I can usually go with VBR4 and be ok with it on portables or in the car but there's just no way I'd go anything less than FLAC on the 'puter.

I've played with the f2k abx'er and most of the time I can hear (Flac v. WMP 320 CBR's) a difference but can't always tell which is which. I applaud your 40 round tests...that's dedication.
 
Jun 27, 2009 at 2:54 PM Post #6 of 66
You might want to try this with a better source. With the headphone jack off my laptop, I only get around 50% correct. With a high end dac, I can get about 90+% right most of the time

^based on previous tests I've done
 
Jun 27, 2009 at 5:16 PM Post #7 of 66
Quote:

Originally Posted by Berlioz /img/forum/go_quote.gif
Oh very true. However, I was straining incredibly hard to focus. If I was in my music listening mood(I'm always in this mood!), it's unlikely that I would be doing this. It wasn't exactly enjoyable.


Some people (such as myself) theorize the important differences between components show themselves mainly in a "music listening mood" and in "enjoyment" rather than during straining.

(Note: some differences do show themselves during extremely close concentrated listening, as we know. However, that doesn't mean that all differences do, or that the most important ones do.)
 
Jun 27, 2009 at 7:56 PM Post #8 of 66
Congrats on your patience, it isn't easy.

I conducted a test of this on my own. at 192, I was pretty much flawless. At 256, I was only noting subtle differences in transparency and dynamics, and only got around 5/7 of the trials correct. At 320, the differences were almost inaudible: I failed 40% of the trials.
 
Jun 28, 2009 at 12:44 PM Post #9 of 66
Quote:

Symphony No. 1 - 427 kbps
Gold Rush - 1001 kbps
Rambaldo!...Ah! M'aiutate! - 628 kbps
Sunshine of Your Love - 818 kbps


It's interesting that higher flac bitrates didn't predict which pieces you'd have trouble ABXing. That's not at all what I would have guessed.

I am curious about a couple of things. Can you describe the type of difference you heard between the Shostakovitch tracks? How long did it take you to do all those trials? Was this all done in one day or spread out over time?
 
Jun 28, 2009 at 1:49 PM Post #10 of 66
Quote:

Originally Posted by jinp6301 /img/forum/go_quote.gif
You might want to try this with a better source. With the headphone jack off my laptop, I only get around 50% correct. With a high end dac, I can get about 90+% right most of the time

^based on previous tests I've done



I'd tend to agree. I appreciate the work you put into this, but if it's done on the typical PC headphone jack I suspect that it's not allowing a lot of details to come through that might otherwise be more revealing of differences.
 
Jun 28, 2009 at 2:09 PM Post #11 of 66
Quote:

Originally Posted by Earwax /img/forum/go_quote.gif
It's interesting that higher flac bitrates didn't predict which pieces you'd have trouble ABXing. That's not at all what I would have guessed.



Actually, while the flac bitrates don't mean anything, they might correlate to loudness. In general, hot CDs I've ripped are usually over 1000 kbps but the classical and jazz can be 400-700 kbps. But the numbers don't mean anything in the end. Just the saved amount of disk space.
 
Jun 28, 2009 at 3:14 PM Post #12 of 66
Quote:

Originally Posted by Berlioz /img/forum/go_quote.gif
Oh very true. However, I was straining incredibly hard to focus. If I was in my music listening mood(I'm always in this mood!), it's unlikely that I would be doing this. It wasn't exactly enjoyable.

Now, I still believe it's possible to discern 320kbps from lossless. However, the difference I experienced between the two was very small. That's what really got me. Before, I could hear huge differences between them. I'd play my FLAC files and think to myself, "Yeah, that's why I spent all that time burning those CD's." It was blissful. In reality though, they are almost exactly the same, at least to my ears - which, at least according to those hearing tests someone posted - are pretty good.

When I get my home set up complete, I'll test it again. I expect it to be easier to differentiate the two, using better equipment, but it should still be pretty hard. Harder than I thought anyways.



I'd also investigate the mp3 encode - which version of lame, what were the settings? from version to version there are often specific pieces of music that remain non-transparent at even the highest settings - these are called "killer samples".

I'd try it with whatever the analog for --alt-preset insane (v0 insane, "=b 320" I think?)

Takes longer to encode, but with todays multicore machines who cares
smily_headphones1.gif



There is supposedly very nearly a negligible difference between high bitrate AAC and MP3 - but they have different killer samples and artifacts. I'd be curious if you could tell between an AAC and FLAC of your identifiable piece (I'd suggest apple's AAC encoder)


Of course, going with high bitrate stuff is a slippery slope - there isn't much difference between 120ish kbps more than transparency and 400 kbps more than transparency, other than space, as long as your formats are supported.
 
Jun 28, 2009 at 3:58 PM Post #14 of 66
Thank you Berlioz ! That's serious testing, the kind that I like to hear about. I'm very interested by this, because I still have to rip my CD collection. Two questions :

1. When you heard a difference between MP3 and FLAC, was it a continuous difference (meaning that you could hear it all time), or was it a ponctual difference (meaning that you could hear it at some places/passages only) ?

2. I understand that MP3-320 is a very good format, but the encoder could be the weak link. It would means that encoding with WMP or LAME would not yield to bit identical MP3 files. Is that tested and confirmed ?

Thanks,
 
Jun 28, 2009 at 4:35 PM Post #15 of 66
Quote:

Originally Posted by Earwax /img/forum/go_quote.gif
It's interesting that higher flac bitrates didn't predict which pieces you'd have trouble ABXing. That's not at all what I would have guessed.


I am not as surprised!
Cause for lossless the bitrate is not a reflection of the sound quality as all (like for lossy), but just a reflection of the complexity of the music and/or the efficiency of the encoder.
 

Users who are viewing this thread

Back
Top