An experiment in lossy vs lossless A/B testing

[devhen]

This is my initial quick run-through. This is really difficult. I've never listened to this song before today and this is exactly the kind of music I've had a hard time telling the difference between in the past. I really think we'd be better off with something more ambiant and realistic like a live recording. But **** it. Obviously you'll think this is me making excuses. You'll see I still did pretty well.
 
 

Spoiler: Warning%3A%20Spoiler!

 
foo_abx 1.3.4 report
foobar2000 v1.2.2
2013/02/08 18:45:41
 
File A: C:\Users\Devin\Music\ABX\A1.wav
File B: C:\Users\Devin\Music\ABX\B1.wav
 
18:45:41 : Test started.
18:46:28 : 00/01  100.0%
18:46:33 : Trial reset.
18:50:46 : 01/01  50.0%
18:51:00 : 02/02  25.0%
18:51:13 : 02/03  50.0%
18:51:42 : 03/04  31.3%
18:51:54 : 04/05  18.8%
18:52:22 : 05/06  10.9%
18:52:57 : 06/07  6.3%
18:53:00 : Test finished.
 
 ---------- 
Total: 6/8 (14.5%)
 

 

[soundstige]

Nice results! What headphone were you using? This music is really hard, not familiar to me at all either. Your results are about the same as mine, so you can draw conclusions from that. I know I want to do a test of at least 20 results -- do you have any interest in that? It's definitely taxing on your ears but hey, why not? 

Quote:

This is my initial quick run-through. This is really difficult. I've never listened to this song before today and this is exactly the kind of music I've had a hard time telling the difference between in the past. I really think we'd be better off with something more ambiant and realistic like a live recording. But **** it. Obviously you'll think this is me making excuses. You'll see I still did pretty well.

 

[devhen]

Yes its taxing and yes I'm interested in doing a more extensive test. I need more time. I need to get familiar with this track and I do need to give my ears a bit of a break since its particularly difficult unless the volume is quite high haha.
 
I'm on foobar > ASIO > HRT Music Streamer II > Lake People G109 > HD600

 

[soundstige]

I know, I keep telling myself that I shouldn't turn it up, to save my hearing, but it helps!

Also HD650 and 600 are the only two Sennheiser headphones I'm really interested in trying soon (unobtainium, aka HD800, notwithstanding). I used my savings on the HE-400 though and love it. Just need a better amp!
 
Quote:

Yes its taxing and yes I'm interested in doing a more extensive test. I need more time. I need to get familiar with this track and I do need to give my ears a bit of a break since its particularly difficult unless the volume is quite high haha.
 
I'm on foobar > ASIO > HRT Music Streamer II > Lake People G109 > HD600

 

[devhen]

I hear the HE-400 is pretty easy to drive for an ortho but if you're looking for something with plenty of power for the $ checkout the Lake People G103 its only marginally less powerful than my G109 (which is quite a beast) and runs only ~ $250.

 

[jimmylee1997]

Spoiler: Warning%3A%20Spoiler!

I think the lossless track is B1. It has less noise in the background, and it sounded a bit fuller.

 

[chewy4]

Quote:

This is my initial quick run-through. This is really difficult. I've never listened to this song before today and this is exactly the kind of music I've had a hard time telling the difference between in the past. I really think we'd be better off with something more ambiant and realistic like a live recording. But **** it. Obviously you'll think this is me making excuses. You'll see I still did pretty well.
 
 

Spoiler: Warning%3A%20Spoiler!

 
foo_abx 1.3.4 report
foobar2000 v1.2.2
2013/02/08 18:45:41
 
File A: C:\Users\Devin\Music\ABX\A1.wav
File B: C:\Users\Devin\Music\ABX\B1.wav
 
18:45:41 : Test started.
18:46:28 : 00/01  100.0%
18:46:33 : Trial reset.
18:50:46 : 01/01  50.0%
18:51:00 : 02/02  25.0%
18:51:13 : 02/03  50.0%
18:51:42 : 03/04  31.3%
18:51:54 : 04/05  18.8%
18:52:22 : 05/06  10.9%
18:52:57 : 06/07  6.3%
18:53:00 : Test finished.
 
 ---------- 
Total: 6/8 (14.5%)
 

Not bad, there's of course still an uncertainty (14.5% uncertainty to be exact) with those results, but IMO that's OK correlation that you can at least hear the extra details that are there in the lossless. Any opinion on which file is which? I believe when ABXing that the first file in your playlist is always going to be A, and the second is B.

 

[devhen]

Quote:

Not bad, there's of course still an uncertainty (14.5% uncertainty to be exact) with those results, but IMO that's OK correlation that you can at least hear the extra details that are there in the lossless.

 
Lets not forget that no one is going argue that there is a night and day difference so it would be unreasonable to expect 100% certainty.

Any opinion on which file is which? I believe when ABXing that the first file in your playlist is always going to be A, and the second is B.

 

Spoiler: Warning%3A%20Spoiler!

That was my assumption as well and so far my guess is that B is the lossless copy. I need some more time with the track though, after I give my ears a good rest, and a more extensive test I think would be good (maybe 20 results as soundstige mentioned). At that point I should have a more solid guess on which is the lossless.

 

[chewy4]

Quote:

 
Lets not forget that no one is going argue that there is a night and day difference so it would be unreasonable to expect 100% certainty.

Definitely. I know some people say that you can't tell a difference at all unless you can get 15/15 or higher, but honestly given how flawed auditory memory is and how impatient people are(or at least I am), I don't expect those results unless you're trying to make a claim that it's a night and day difference. But at the same time be skeptical of small amount of trials, as I have personally failed 4 or 5 in a row before right off the bat, someone could just as easily pass that many by chance.
 
In the end I think it's best to just use your own judgement when doing these tests to decide how significant the change is to you. 

 

[Satellite_6]

10-20 trails with a probability of guessing of less than 5%. Don't use any weird settings. Then you WIN.
 
I've only done some 5/5 ones myself tho, but several in a row which changes the stats. 

 

[SandvichDISH]

Sorry for being late! I was a bit distracted by life stuff and didn't check head-fi for a while. But I'll gladly give this test a go, if I could at the minute. I am waiting for my Objective2 as I had some amp trouble with my new HD 600 (trouble being, my E17 seems to have decided it doesn't want to work right anymore. Although it wasn't a good amp for the HD 600 anyway).
So in (hopefully) a week I'll get it, and then I'll do this test with some proper audio. 

 

[ferday]

that was fun, i like this it entertains me.  the music was unfamiliar which is probably a good thing
 
here's my first try.  i'm using HE400's through a FiiO E17...i'm going to play with the files with some different setups too.  i don't do a lot of A/B but my initial thoughts are that i could "get better" at it after a while
 

Spoiler: Warning%3A%20Spoiler!

 
foo_abx 1.3.4 report
foobar2000 v1.1.15
2013/02/12 15:24:22
 
File A: C:\Users\ferday\Downloads\A1.wav
File B: C:\Users\ferday\Downloads\B1.wav
 
15:24:22 : Test started.
15:25:01 : 01/01  50.0%
15:25:32 : 02/02  25.0%
15:25:55 : 03/03  12.5%
15:26:24 : 03/04  31.3%
15:26:42 : 03/05  50.0%
15:27:10 : 04/06  34.4%
15:27:35 : 05/07  22.7%
15:28:02 : 06/08  14.5%
15:28:44 : 06/09  25.4%
15:29:32 : 06/10  37.7%
15:29:39 : Test finished.
 
 ---------- 
Total: 6/10 (37.7%)
 
i think B1 was the flac, but not for any other reason than an occasionally perceived "fullness".  obviously this perception was not as noticeable in the A/B setting 
 

 

[thoughtcriminal]

Has everyone forgotten about this? I for one would like to know which is which

 

[chewy4]

It appears so. Here's the answer:
 
 

Spoiler: Warning%3A%20Spoiler!

B is the MP3

 

[Xinze]

Results here pretty much proves that when even if you can tell a difference in MP3 vs FLAC, the difference is so minimal that you actually have to repeatedly play a song and it's different bitrates one after the other to actually hear it. Now to a real challenge: People do the blind test, and figure out which one is the FLAC and which one's the MP3.
 
 
 
Think a safe conclusion is for most people out here, you're not missing out on anything out on anything if you're listening to 320 rather than FLAC.