Head-Fi.org › Forums › Equipment Forums › Sound Science › An experiment in lossy vs lossless A/B testing
New Posts  All Forums:Forum Nav:

An experiment in lossy vs lossless A/B testing - Page 3

post #31 of 48
2/8/13 at 6:02pm
Thread Starter 

This is my initial quick run-through. This is really difficult. I've never listened to this song before today and this is exactly the kind of music I've had a hard time telling the difference between in the past. I really think we'd be better off with something more ambiant and realistic like a live recording. But **** it. Obviously you'll think this is me making excuses. You'll see I still did pretty well.

 

 

Warning: Spoiler! (Click to show)

 

foo_abx 1.3.4 report
foobar2000 v1.2.2
2013/02/08 18:45:41
 
File A: C:\Users\Devin\Music\ABX\A1.wav
File B: C:\Users\Devin\Music\ABX\B1.wav
 
18:45:41 : Test started.
18:46:28 : 00/01  100.0%
18:46:33 : Trial reset.
18:50:46 : 01/01  50.0%
18:51:00 : 02/02  25.0%
18:51:13 : 02/03  50.0%
18:51:42 : 03/04  31.3%
18:51:54 : 04/05  18.8%
18:52:22 : 05/06  10.9%
18:52:57 : 06/07  6.3%
18:53:00 : Test finished.
 
 ---------- 
Total: 6/8 (14.5%)

 

Reply

Head-Fi's Best Sellers

post #32 of 48
2/8/13 at 6:08pm

Nice results! What headphone were you using? This music is really hard, not familiar to me at all either. Your results are about the same as mine, so you can draw conclusions from that. I know I want to do a test of at least 20 results -- do you have any interest in that? It's definitely taxing on your ears but hey, why not? tongue_smile.gif

Quote:
Originally Posted by devhen View Post

This is my initial quick run-through. This is really difficult. I've never listened to this song before today and this is exactly the kind of music I've had a hard time telling the difference between in the past. I really think we'd be better off with something more ambiant and realistic like a live recording. But **** it. Obviously you'll think this is me making excuses. You'll see I still did pretty well.

Reply
post #33 of 48
2/8/13 at 6:10pm
Thread Starter 

Yes its taxing and yes I'm interested in doing a more extensive test. I need more time. I need to get familiar with this track and I do need to give my ears a bit of a break since its particularly difficult unless the volume is quite high haha.

 

I'm on foobar > ASIO > HRT Music Streamer II > Lake People G109 > HD600

Reply
post #34 of 48
2/8/13 at 6:13pm

I know, I keep telling myself that I shouldn't turn it up, to save my hearing, but it helps!


Also HD650 and 600 are the only two Sennheiser headphones I'm really interested in trying soon (unobtainium, aka HD800, notwithstanding). I used my savings on the HE-400 though and love it. Just need a better amp!

 

Quote:
Originally Posted by devhen View Post

Yes its taxing and yes I'm interested in doing a more extensive test. I need more time. I need to get familiar with this track and I do need to give my ears a bit of a break since its particularly difficult unless the volume is quite high haha.

 

I'm on foobar > ASIO > HRT Music Streamer II > Lake People G109 > HD600

Reply
post #35 of 48
2/8/13 at 6:15pm
Thread Starter 

I hear the HE-400 is pretty easy to drive for an ortho but if you're looking for something with plenty of power for the $ checkout the Lake People G103 its only marginally less powerful than my G109 (which is quite a beast) and runs only ~ $250.


Edited by devhen - 2/8/13 at 6:17pm
Reply
post #36 of 48
2/8/13 at 6:18pm
Warning: Spoiler! (Click to show)

I think the lossless track is B1. It has less noise in the background, and it sounded a bit fuller.

Reply
post #37 of 48
2/8/13 at 6:22pm
Quote:
Originally Posted by devhen View Post

This is my initial quick run-through. This is really difficult. I've never listened to this song before today and this is exactly the kind of music I've had a hard time telling the difference between in the past. I really think we'd be better off with something more ambiant and realistic like a live recording. But **** it. Obviously you'll think this is me making excuses. You'll see I still did pretty well.

 

 

Warning: Spoiler! (Click to show)

 

foo_abx 1.3.4 report
foobar2000 v1.2.2
2013/02/08 18:45:41
 
File A: C:\Users\Devin\Music\ABX\A1.wav
File B: C:\Users\Devin\Music\ABX\B1.wav
 
18:45:41 : Test started.
18:46:28 : 00/01  100.0%
18:46:33 : Trial reset.
18:50:46 : 01/01  50.0%
18:51:00 : 02/02  25.0%
18:51:13 : 02/03  50.0%
18:51:42 : 03/04  31.3%
18:51:54 : 04/05  18.8%
18:52:22 : 05/06  10.9%
18:52:57 : 06/07  6.3%
18:53:00 : Test finished.
 
 ---------- 
Total: 6/8 (14.5%)

 

Not bad, there's of course still an uncertainty (14.5% uncertainty to be exact) with those results, but IMO that's OK correlation that you can at least hear the extra details that are there in the lossless. Any opinion on which file is which? I believe when ABXing that the first file in your playlist is always going to be A, and the second is B.


Edited by chewy4 - 2/8/13 at 6:24pm
Reply
post #38 of 48
2/8/13 at 6:28pm
Thread Starter 
Quote:
Originally Posted by chewy4 View Post

Not bad, there's of course still an uncertainty (14.5% uncertainty to be exact) with those results, but IMO that's OK correlation that you can at least hear the extra details that are there in the lossless.

 

Lets not forget that no one is going argue that there is a night and day difference so it would be unreasonable to expect 100% certainty.


Quote:
Originally Posted by chewy4 View Post

Any opinion on which file is which? I believe when ABXing that the first file in your playlist is always going to be A, and the second is B.

 

Warning: Spoiler! (Click to show)

That was my assumption as well and so far my guess is that B is the lossless copy. I need some more time with the track though, after I give my ears a good rest, and a more extensive test I think would be good (maybe 20 results as soundstige mentioned). At that point I should have a more solid guess on which is the lossless.

Reply
post #39 of 48
2/8/13 at 6:35pm
Quote:
Originally Posted by devhen View Post

 

Lets not forget that no one is going argue that there is a night and day difference so it would be unreasonable to expect 100% certainty.

Definitely. I know some people say that you can't tell a difference at all unless you can get 15/15 or higher, but honestly given how flawed auditory memory is and how impatient people are(or at least I am), I don't expect those results unless you're trying to make a claim that it's a night and day difference. But at the same time be skeptical of small amount of trials, as I have personally failed 4 or 5 in a row before right off the bat, someone could just as easily pass that many by chance.

 

In the end I think it's best to just use your own judgement when doing these tests to decide how significant the change is to you. 

Reply
post #40 of 48
2/8/13 at 7:24pm

10-20 trails with a probability of guessing of less than 5%. Don't use any weird settings. Then you WIN. :D

 

I've only done some 5/5 ones myself tho, but several in a row which changes the stats. 


Edited by Satellite_6 - 2/8/13 at 7:25pm
Reply
post #41 of 48
2/12/13 at 7:47am

Sorry for being late! I was a bit distracted by life stuff and didn't check head-fi for a while. But I'll gladly give this test a go, if I could at the minute. I am waiting for my Objective2 as I had some amp trouble with my new HD 600 (trouble being, my E17 seems to have decided it doesn't want to work right anymore. Although it wasn't a good amp for the HD 600 anyway).

So in (hopefully) a week I'll get it, and then I'll do this test with some proper audio. 

Reply
post #42 of 48
2/12/13 at 2:32pm

that was fun, i like this it entertains me.  the music was unfamiliar which is probably a good thing

 

here's my first try.  i'm using HE400's through a FiiO E17...i'm going to play with the files with some different setups too.  i don't do a lot of A/B but my initial thoughts are that i could "get better" at it after a while

 

Warning: Spoiler! (Click to show)

 

foo_abx 1.3.4 report
foobar2000 v1.1.15
2013/02/12 15:24:22
 
File A: C:\Users\ferday\Downloads\A1.wav
File B: C:\Users\ferday\Downloads\B1.wav
 
15:24:22 : Test started.
15:25:01 : 01/01  50.0%
15:25:32 : 02/02  25.0%
15:25:55 : 03/03  12.5%
15:26:24 : 03/04  31.3%
15:26:42 : 03/05  50.0%
15:27:10 : 04/06  34.4%
15:27:35 : 05/07  22.7%
15:28:02 : 06/08  14.5%
15:28:44 : 06/09  25.4%
15:29:32 : 06/10  37.7%
15:29:39 : Test finished.
 
 ---------- 
Total: 6/10 (37.7%)
 
i think B1 was the flac, but not for any other reason than an occasionally perceived "fullness".  obviously this perception was not as noticeable in the A/B setting 

 


Edited by ferday - 2/12/13 at 2:34pm
Reply
post #43 of 48
2/26/13 at 5:13pm
Has everyone forgotten about this? I for one would like to know which is which
Reply
post #44 of 48
2/26/13 at 5:18pm

It appears so. Here's the answer:
 

 

Warning: Spoiler! (Click to show)

B is the MP3

Reply
post #45 of 48
2/26/13 at 8:37pm

Results here pretty much proves that when even if you can tell a difference in MP3 vs FLAC, the difference is so minimal that you actually have to repeatedly play a song and it's different bitrates one after the other to actually hear it. Now to a real challenge: People do the blind test, and figure out which one is the FLAC and which one's the MP3.

 

 

 

Think a safe conclusion is for most people out here, you're not missing out on anything out on anything if you're listening to 320 rather than FLAC.

 

 

Reply
New Posts  All Forums:Forum Nav:
  Return Home
  Back to Forum: Sound Science

Head-Fi's Best Sellers

Head-Fi.org › Forums › Equipment Forums › Sound Science › An experiment in lossy vs lossless A/B testing