[aQiss]

First of all, this blind test is unique, as not only mp3-encoded files are used for samples. More details below.
 
[size=10.5pt]I'd like to check if there are actually people with 'gold ears'; 4 audio samples of the same track, Daten-Ubertragungs-Kusschen by Dominik Eulberg, are going to help me with that.[/size]
[size=10.5pt] As I said, you are not going to face a simple mp3-vs-FLAC comparison. The test consists of 128 Kbps MP3, 256 Kbps MP3, FLAC (1406 Kbps) and, for good measure, HE-AAC (64 Kbps, iTunes VBR) encodings.[/size]
 
[size=10.5pt]Let's get started, shall we? The samples are uploaded to Mediafire, they are 16 bit/44100 Hz WAVs of the same size (approx. 10,3 Mbytes). Match the codec/bitrate and the sample number.[/size]
 
[size=10.5pt]Good luck with your guesses![/size]

 
Sample #1,  Sample #2,  Sample #3,  Sample #4
 
--- --- --- --- ---
Hats off to everyone who tried.

The results seem to be very interesting; they made me think whether or not we really need lossless audio formats. The most cool thing is that HE-AAC (encoded with iTunes, not Nero Encoder) is a surprisingly good lossy codec which is able to provide impressive quality at ultra-low bitrates. 
 

Spoiler: File%20formats%20%26%20bitrates%20uncovered

 
 
Sample 1 = 256 kbps  CBR .mp3
Sample 2 = 64 kbps  VBR .m4a (aac)
Sample 3 = 1404 kbps  .flac
Sample 4 = 128 kbps  CBR .mp3

 
 

 

 

[Bizzlebin]

This was very tough indeed! Even on my DT770 Pro's, however, I could pick out more details on Sample 4. Distinguishing between the others was much more difficult, but I think I've got them in a reasonable order:
 
Sample 4: FLAC
Sample 2: 256kbps MP3
Sample 1: 128kbps MP3
Sample 3: 64kbps HE-AAC
 
Even though I am quite new to headphones, I was starting to like these sorts of tests; this was a whopper, though!
 
-Bizzlebin

 

[deadlylover]

Nice song bro.
 

 
1. 256kbps mp3
2. 128kbps mp3
3. Lossless
4. 64kbps aac
 
 

Spoiler: ABX

 
foo_abx 1.3.4 report
foobar2000 v1.1.11
2012/06/12 11:58:17
 
File A: C:\Users\Alice\Desktop\Sample_1.wav
File B: C:\Users\Alice\Desktop\Sample_2.wav
 
11:58:17 : Test started.
11:59:50 : 01/01  50.0%
11:59:58 : 02/02  25.0%
12:00:06 : 03/03  12.5%
12:00:43 : 04/04  6.3%
12:00:48 : 05/05  3.1%
12:00:53 : 06/06  1.6%
12:01:04 : 07/07  0.8%
12:01:14 : 07/08  3.5%
12:01:29 : 08/09  2.0%
12:01:36 : 09/10  1.1%
12:01:46 : 10/11  0.6%
12:01:59 : 11/12  0.3%
12:02:17 : 12/13  0.2%
12:02:28 : 13/14  0.1%
12:02:36 : 14/15  0.0%
12:02:40 : Test finished.
 
 ---------- 
Total: 14/15 (0.0%)
 
foo_abx 1.3.4 report
foobar2000 v1.1.11
2012/06/12 12:12:51
 
File A: C:\Users\Alice\Desktop\Sample_1.wav
File B: C:\Users\Alice\Desktop\Sample_3.wav
 
12:12:51 : Test started.
12:13:36 : 00/01  100.0%
12:14:11 : 01/02  75.0%
12:14:26 : 02/03  50.0%
12:14:40 : 03/04  31.3%
12:15:22 : 03/05  50.0%
12:15:41 : 04/06  34.4%
12:16:17 : 05/07  22.7%
12:16:24 : 06/08  14.5%
12:16:35 : 07/09  9.0%
12:16:51 : 08/10  5.5%
12:17:16 : 09/11  3.3%
12:17:31 : 10/12  1.9%
12:17:49 : 11/13  1.1%
12:18:06 : 12/14  0.6%
12:18:58 : 13/15  0.4%
12:19:15 : 13/16  1.1%
12:19:28 : 14/17  0.6%
12:19:35 : 15/18  0.4%
12:19:44 : 16/19  0.2%
12:19:49 : 17/20  0.1%
12:19:51 : Test finished.
 
 ---------- 
Total: 17/20 (0.1%)

 
foo_abx 1.3.4 report
foobar2000 v1.1.11
2012/06/12 11:51:49
 
File A: C:\Users\Alice\Desktop\Sample_3.wav
File B: C:\Users\Alice\Desktop\Sample_4.wav
 
11:51:49 : Test started.
11:53:14 : 01/01  50.0%
11:53:22 : 02/02  25.0%
11:53:25 : 02/03  50.0%
11:53:40 : 03/04  31.3%
11:53:49 : 04/05  18.8%
11:53:52 : 05/06  10.9%
11:53:55 : 06/07  6.3%
11:54:03 : 07/08  3.5%
11:54:07 : 08/09  2.0%
11:54:17 : 09/10  1.1%
11:54:26 : 10/11  0.6%
11:54:29 : 11/12  0.3%
11:54:42 : 12/13  0.2%
11:54:49 : 13/14  0.1%
11:54:52 : 13/15  0.4%
11:55:05 : Test finished.
 
 ---------- 
Total: 13/15 (0.4%)

 
foo_abx 1.3.4 report
foobar2000 v1.1.11
2012/06/12 12:38:35
 
File A: C:\Users\Alice\Desktop\Sample_2.wav
File B: C:\Users\Alice\Desktop\Sample_3.wav
 
12:38:35 : Test started.
12:39:00 : 00/01  100.0%
12:39:11 : 01/02  75.0%
12:39:38 : 02/03  50.0%
12:39:49 : 03/04  31.3%
12:39:53 : 04/05  18.8%
12:39:55 : 05/06  10.9%
12:39:59 : 06/07  6.3%
12:40:01 : 07/08  3.5%
12:40:04 : 08/09  2.0%
12:40:07 : 09/10  1.1%
12:40:09 : 10/11  0.6%
12:40:12 : 11/12  0.3%
12:40:14 : 12/13  0.2%
12:40:16 : 13/14  0.1%
12:40:19 : 14/15  0.0%
12:40:20 : Test finished.
 
 ---------- 
Total: 14/15 (0.0%)

 

[OJNeg]

Really tough. I didn't use a ABX comparator, just listened to them through about 5 times. The 4th I definitely zeroed out as the lowest quality, but between the second and third it was pretty much a toss-up for me
 
1: 128
2: Lossless
3: 256
4: 64

 

[aQiss]

Here is the correct answer to the test. Only a few of your guesses match the reality, which is somewhat surprising.
Thanks to everyone for trying anyways!
 
 

Spoiler: File%20formats%20%26%20bitrates%20uncovered

 
[size=medium]Sample 1 = 256 kbps  CBR .mp3[/size]
[size=medium]Sample 2 = 64 kbps  VBR .m4a (aac)[/size]
[size=medium]Sample 3 = 1404 kbps  .flac[/size]
 
[size=medium]Sample 4 = 128 kbps  CBR .mp3[/size]

 
 

 

 

[Audio-Omega]

Using my laptop and Sennheiser HD600, I thought they all sounded the same !  

 

[ProtegeManiac]

Quote:

Using my laptop and Sennheiser HD600, I thought they all sounded the same !  

For the most part yes (I'm using the same headphone) but one has looser bass and one has slightly sharper cymbals (not necessarily more accurate though; it sounds a little tinny, not sharper as in more body or reverb to it). Everything else is probably far enough from the extremes that they aren't affected enough by the compression algorithm (ie, most compression works by reducing the freqs at the extremes, since the initial assumption was that people don't hear them well enough).

Just for the heck of it I did somewhat of a blind test with my own audio and compressed/converted the FLAC into MP3, mixed them and the originals in the same folder, then put them in a playlist set to random so I wouldn't know which one is which. Aside from similar observations as above wherever these can be observed the vocals tend to sound thinner, and so are the rest of the percussion. Put the same files in a MicroSD and used it with my mobile phone and a Westone2 - I had to strain to hear any difference but given the file sizes and even if I got a higher capacity microSD, even if the gap was larger I'd much rather cram more music into it than larger files of the same songs.
 
EDIT : Just saw the actual bitrates are already posted, and the track with looser bass from my listening was the FLAC. Not surprised though since it's supposedly lossless, the bass must have been recorded that way and/or the headphones were adding to it. Never observed it with my own audio though but for the most part they either don't have bass or have fast bass, so they probably didn't have it recorded like that. What surprised me was that the cymbals I thought were most likely 64kbps was actually 246kbps; maybe it would have lacked body but it would have takes out enough of it to not sound too sharp?

 

[Audio-Omega]

One sample had tinny treble and the other had fuller bass but I couldn't be certain.  I felt that the differences were minute.  

 

[Bizzlebin]

@ProtegeManiac: I also pinned the FLAC as poorer quality, but since I'm going to reference some tell-tale signs which can be used to more easily differentiate some of the samples, I'll go ahead and put it in a spoiler box:
 

Spoiler: Warning%3A%20Spoiler!

After listening to all the samples a few times, I could tell some differences but, as noted in my original response, they were very hard to pick out. I decided to try another method which I though would make the differences clearer: using a 10-band EQ. I randomly decided to up the 16k band by 12 dB and lower all of the others by 12 dB. And one of the differences was much clearer: I very easily heard some background noise beginning at 38-39 seconds. This noise lasted from then until the end of the samples and, since it was the most obvious difference to me, I used it to pick out which sample was which. Unfortunately, this is where I went wrong.
 
As all of the sounds in the sample are digital, there is no point of reference for what is "right" and "wrong". The only thing I could do was pick what I thought sounded best. In the 64 kbps HE-AAC and 128 kbps MP3 samples (2 and 4, respectively), I heard a very pleasant oscillation in the aforementioned background noise. The oscillation was clearer in 4 (128 kbps MP3), so I picked that as the best. The oscillation was less consistent and harder to hear in 2 (64 kbps HE-AAC), so I picked it as second. The other two samples had almost no oscillation, only a constant noise, so I assumed that they were the lowest bitrate samples. Being very similar (i.e. they both sounded like noise), I was only barely able to pick sample 1 (256 kbps MP3) as my third choice; there was just a hint of that ever-so-wonderful oscillation I was after! That left sample 3 (what turned out to be the FLAC) as last on my list; I thought surely the loss of that oscillation was produced by compression and that this must be the worst sample, the encoder leaving nothing but boring noise in the background!
 
Unfortunately for my list, the oscillation itself was the compression artifact and the noise was what the composer intended.

 
Based on my experiences, then, I think this test could be improved by having a fifth sample, also FLAC, with which to compare the other samples to. The control sample would be an undisguised FLAC and the other four could be the same types as were already used. This way, our differing perceptions can be neutralized better; it won't matter what I think sounds better or more pleasing, but what I think sounds closest to the control sample. If this was non-electronic music, I think the having a control would still be quite helpful (especially for those who don't have very much experience going to live performances and hearing many instruments played in person) but maybe not imperative.
 
One part of the test that surprised me, though, was my mixing up the lower-quality MP3 and the HE-AAC samples. As that sound which I liked (which turned out to be the compression artifact) was actually less pronounced on the 64 kbps HE-AAC than the 128 kbps MP3, I wonder if that will translate to less artifacts in other music. It uses, after all, a newer compression scheme and was designed in order to be better than MP3 compression. Comparing them just once in a very specific type of music isn't enough for me to make a broader judgment on them, but I am intrigued and would like to listen more into the differences between the two formats.
 
So, to conclude, this was an extremely interesting test and I am disappointed more people didn't participate. However, aQiss, to use this test of a very carefully selected piece of simpler electronic music with no clear control and no other standard of reference and say we don't "really need lossless audio formats" is totally unfair. If you wanted to design a test that was very difficult, then yes, you did a top-notch job. But seeing that myself and some others were able to distinguish pretty accurately between most of the samples (even though we disagreed as to which ones sounded subjectively better and thus about which bitrate was which, having no standard of comparison), this hardly makes the point that the samples were indistinguishable.
 
-Bizzlebin

 

[Audio-Omega]

Do you think it's much harder to differentiate electronic music than orchestra ?

 

[ProtegeManiac]

@Bizzlebin,
 
Yes I agree that without knowing the track intimately the best we can do really is pick out what "sounds best" vs "what sounds like the original (lossless)." Plus, was that track fully synthesized? I'd like to try a similar test using a recording of acoustic and even electronically amplified instruments, but instead of guessing the bitrates based on multiple choice, just what each thinks is "best" depending on the equipment used. Let's face it, this will play a part in what people hear - I'm not delusional in thinking that maybe it's my headphones that have more bass, or adds to the excessive bass, of the test track. Kudos to the aQiss for the inspiration for a test like this; I'll take a few nights this week to select a couple of tracks to convert to lossy and post for another round of tests. Also, I actually won't be surprised if some people end up picking the lossy files as "better" since our equipment aren't absolutely colorless, which is why I won't post the bitrates of the tracks so people won't be distracted by that.
 
 
 
@Audio-Omega,
 
From my own experience, yes. As per my post above, the body/weight of lower freq percussion instruments are the first things I notice if I listen to a lossy copy of a track I'm familiar with, followed by the vocals. Like I said, I'll work on another one, and post here when it's up so there's another way to find the link, and I hope the few participants here will take that test too.

 

[Audio-Omega]

Could bit rate be measured via headphones ?

 

[ProtegeManiac]

Quote:

Could bit rate be measured via headphones ?

 
No, of course not; what comes out of the headphone is analog. That will be kind of like scanning a printout from a Nikon D3 35mm camera and guessing how many megapixels it has - probably not even a microscope can give a close enough answer. However you can still see which photo may have been downscaled, or printed out a large enough size that you begin to see pixellation (and conclude, for example, that the camera that took it wasn't a medium format). You can however measure frequency response if you have, say, a USB microphone, but you won't be able to begin guessing bitrates if you don't have the same data off the lossless copy since not all songs will register the same (ie, some will have louder lows, softer highs, etc depending on the instruments and the recording). A specially designed frequency sweep might be better for that though.

 

[Audio-Omega]

I was meant to ask if the differences we heard could be measured via headphones and plotted on a graph. Each bit rate would show a different frequency response in speculation anyway.

 

[ProtegeManiac]

Quote:

I was meant to ask if the differences we heard could be measured via headphones and plotted on a graph. Each bit rate would show a different frequency response in speculation anyway.

 
Put that way, theoretically, it should be possible.