24bit vs 16bit, the myth exploded!

[castleofargh]

 

   
I'm learning to fly like that. I started by standing on a chair and flying to the ground. I'm working my way up to jumping off the top of the Empire State Building.

Oh how humorous we are .... but thanks for proving my point.  If you practice flying by increasing 3 feet at a time, at least we learn how high you can fly.  With ABX testing all I see is a bunch of people jumping from different skyscrapers - all with the same predictable result.  That is why you don't learn that much, except what you already expected. 
 
Remember the only thing you can absolutely learn from ABX testing is that if you pass you were absolutely NOT guessing (down to extremely unlikely probability).  A fail tells you close to nothing - A could have been Bach and B Mozart and maybe the listener was just not paying attention.

I like how you went from first timer in ABX, to telling us what abx can and cannot do in a week.

I would think that most people here have a little understanding of statistical significance and have looked at most of Arny's explanations about the ups and downs of abx on his website or now on hydrogen.
 
anyway the logical "solution" for you then if you're looking for positive samples and progressive evaluation, would be to try abx starting with 6 or 8bits and go up. if all you're asking for is when does the quantization noise is audible or when is the track eaten too much. but we already have a pretty good idea about that, it would be a little under 10 or 12bits for most people at normal listening level. which was about what we had when 16bit dacs where only 16bit.
 
but as RRod said, it a matter of what you're trying to achieve with abx. looks like you're trying to find any way to succeed, your not trying to know if you would succeed under normal conditions. just get the volume loud enough on a calm enough passage and you'll win 100%. the end. 
but that's as fake as trying to show you can hear 20khz by pushing the tone at 140db. sure you can succeed the test, but what did that tell you that you didn't already knew know?

 

[bigshot]

I guess it depends on what the purpose of the test is in the first place... whether it's just to find out what does and doesn't make a difference in practical real world use, or whether it's that you want to beat the test.

 

[sonitus mirus]

I went through a similar issue when attempting to find an acceptable bit rate with a lossy codec to use for my music.  After reading about a dozen or more different descriptions of what an artifact sounded like and what types of audio differences that might occur with lossy files, I simply made a very low bit rate mp3 that had obvious artifacts so that I could hear these for myself.

 

[RRod]

  I guess it depends on what the purpose of the test is in the first place... whether it's just to find out what does and doesn't make a difference in practical real world use, or whether it's that you want to beat the test.

 
Yeah, we're in "Kirk taking the Kobayashi Maru Test" phase, it seems.

 

[sonitus mirus]

   
Yeah, we're in "Kirk taking the Kobayashi Maru Test" phase, it seems.

 
Clearly Kirk cheated and lacks ethics.

 I thought the point of the test was to determine how one deals with failure.  I was the only freak in an entire business team of over 30 people that felt this way when this was used as a group exercise at some seminar.  Everyone else thought it was innovative thinking.

 

[Krutsch]

   
Clearly Kirk cheated and lacks ethics.

 I thought the point of the test was to determine how one deals with failure.  I was the only freak in an entire business team of over 30 people that felt this way when this was used as a group exercise at some seminar.  Everyone else thought it was innovative thinking.

You attended a business seminar where one of the tasks was the Kobayashi Maru test from the fictional Star Trek universe?  
Did they build a working replica of the bridge with simulated explosions? 

 

 

[sonitus mirus]

One of the team building exercises was to see how we felt about the actions that Kirk took in order to pass a test that was designed to fail.    We didn't actually take the Kobayashi Maru test.   The idea was that it was supposed to bring about a discussion, all in good fun.  I was the only one in the group that thought what Kirk did was wrong.  

 

[Krutsch]

  We didn't actually take the Kobayashi Maru test.

 
But... everyone in that group knew to what that referred? Was this at Comic-con?

 

[sonitus mirus]

   
But... everyone in that group knew to what that referred? Was this at Comic-con?

 
I didn't have a clue to what it was before.  It was a Power Point presentation; just a paragraph describing what it was all about.  We answered some question along the line of whether we thought Kirk's action was commendable or wrong, and then shared our results and discussed it in the group.  It was not that detailed.  It was a simple exercise about a no-win situation and creative thinking.  My employer sent 30 of us to this seminar, and it was just one small exercise of many.
 
Nobody had to be a Trekkie to grasp the concept and participate.

 

[RRod]

   
Clearly Kirk cheated and lacks ethics.

 I thought the point of the test was to determine how one deals with failure.  I was the only freak in an entire business team of over 30 people that felt this way when this was used as a group exercise at some seminar.  Everyone else thought it was innovative thinking.

 
"it's a test of character", Kirk says, and it wouldn't surprise me if the character of business is to cheat to win :3 The thing with ABXing hi-res vs. Redbook is that those of us who have tried it ourselves and verse ourselves, to varying degrees, in the theory (both audio and statistical) know what the outcome will be when the test is done correctly. So when a Kirk "passes" the test, we know something is up. Once the truth comes out, there's always some "cheat" that occurred, and if it "had the virtue of never having been tried", as with Kirk, we might give a pat on the back. But we've seen this stuff too many times now for that

 

[Greenears]

Wow. Tough crowd. All I did was use minimal flags and default settings and dared to share interim results. Before I could shake my audiophile tail I'm jumping off skyscrapers into a fictional universe. Nice to see everyone has an open mind.

 

[Greenears]

The track is the 2nd clarinet track test192 from Linn sampler (I put it in the HD tracks directory with the others).
 
I mainly focused on the 6th note, listening for a little distortion or texture on the start of that note that I thought was in the 24 bit.
 
sox -G test192.flac -b 16 test192_16.flac
C:\Users\Public\Music\HDtracks\Various Artists HDtracks Sampler\HDtracks 2014 Sa
mpler>sox --i test192.flac
Input File     : 'test192.flac'
Channels       : 2
Sample Rate    : 192000
Precision      : 24-bit
Duration       : 00:00:29.28 = 5621672 samples ~ 2195.97 CDDA sectors
File Size      : 17.2M
Bit Rate       : 4.69M
Sample Encoding: 24-bit FLAC

C:\Users\Public\Music\HDtracks\Various Artists HDtracks Sampler\HDtracks 2014 Sa
mpler>sox --i test192_16.flac
Input File     : 'test192_16.flac'
Channels       : 2
Sample Rate    : 192000
Precision      : 16-bit
Duration       : 00:00:29.28 = 5621672 samples ~ 2195.97 CDDA sectors
File Size      : 6.25M
Bit Rate       : 1.71M
Sample Encoding: 16-bit FLAC
Comment        : 'Comment=Processed by SoX'

C:\Users\Public\Music\HDtracks\Various Artists HDtracks Sampler\HDtracks 2014 Sa
mpler>c
 
foo_abx 1.3.4 report
foobar2000 v1.3.6
2014/12/15 22:48:27
File A: C:\Users\Public\Music\HDtracks\Various Artists HDtracks Sampler\HDtracks 2014 Sampler\test192.flac
File B: C:\Users\Public\Music\HDtracks\Various Artists HDtracks Sampler\HDtracks 2014 Sampler\test192_16.flac
22:48:27 : Test started.
22:50:09 : 00/01  100.0%
22:51:17 : 00/02  100.0%
22:51:37 : 00/03  100.0%
22:51:58 : 01/04  93.8%
22:53:31 : 02/05  81.3%
22:54:06 : 03/06  65.6%
22:55:00 : 04/07  50.0%
22:56:28 : 04/08  63.7%
22:56:50 : 05/09  50.0%
22:57:17 : 06/10  37.7%
22:58:59 : 07/11  27.4%
22:59:27 : 08/12  19.4%
23:00:42 : 08/13  29.1%
23:01:11 : 09/14  21.2%
23:01:52 : 09/15  30.4%
23:02:45 : 10/16  22.7%
23:03:17 : 11/17  16.6%
23:03:52 : 11/18  24.0%
23:04:46 : 12/19  18.0%
23:05:34 : 13/20  13.2%
23:06:23 : 14/21  9.5%
23:07:54 : 15/22  6.7%
23:09:07 : 15/23  10.5%
23:09:27 : 15/24  15.4%
23:09:50 : 16/25  11.5%
23:10:25 : 17/26  8.4%
23:10:45 : 18/27  6.1%
23:11:20 : 18/28  9.2%
23:11:57 : 18/29  13.2%
23:12:23 : 19/30  10.0%
23:12:43 : 20/31  7.5%
23:13:20 : 20/32  10.8%
23:13:48 : 21/33  8.1%
23:14:47 : 21/34  11.5%
23:15:48 : 22/35  8.8%
23:17:17 : 23/36  6.6%
23:18:04 : 23/37  9.4%
23:18:17 : 23/38  12.8%
23:18:43 : 23/39  16.8%
23:19:17 : 24/40  13.4%
23:20:14 : 24/41  17.4%
23:20:35 : 25/42  14.0%
23:21:04 : 25/43  18.0%
23:21:24 : 26/44  14.6%
23:21:54 : 26/45  18.6%
23:23:02 : 26/46  23.1%
23:23:37 : 27/47  19.1%
23:24:06 : 28/48  15.6%
23:24:32 : 28/49  19.6%
23:25:24 : 29/50  16.1%
23:26:02 : 30/51  13.1%
23:26:46 : 30/52  16.6%
23:27:39 : 31/53  13.6%
23:28:07 : 31/54  17.0%
23:28:28 : 32/55  14.0%
23:28:47 : 33/56  11.4%
23:29:14 : 33/57  14.5%
23:29:40 : 34/58  11.9%
23:30:15 : 35/59  9.6%
23:31:11 : 36/60  7.8%
23:32:03 : 37/61  6.2%
23:32:43 : 37/62  8.1%
23:34:02 : 38/63  6.5%
23:35:13 : 38/64  8.4%
23:36:07 : 39/65  6.8%
23:36:54 : 40/66  5.4%
23:37:16 : 41/67  4.3%
23:38:17 : 42/68  3.4%
23:41:16 : 42/69  4.6%
23:42:33 : 42/70  6.0%
23:42:52 : 43/71  4.8%
23:43:35 : 43/72  6.2%
23:44:12 : 43/73  8.0%
23:44:31 : 44/74  6.5%
23:44:57 : 44/75  8.3%
23:45:01 : Test finished.
 ---------- 
Total: 44/75 (8.3%)
 
Do you think I'm guessing?

 

[stv014]

Do you think I'm guessing?

 
Maybe you can hear something (it is not impossible when listening to some quiet part at high volume), but if you do, it is a very minimal difference with all those failed trials. If you continue the test, combining the results with this run, the p-value should keep going down on average if you do actually hear a difference, so you should eventually get well under 1% (keeping the 44/75 ratio constant, it would be 2%, 0.56%, and 0.16% for 88/150, 132/225, and 176/300, respectively). Also, do not discard any runs just because the score was bad. Selective inclusion guarantees "winning" over enough time, even with pure random guessing.

 

[Greenears]

This one the volume wasn't so pumped up - I would say I listened at a little higher than medium volume.  I could listen to the whole piece at that.  
 
It's very subtle and maybe just one note.  I thought maybe there was a slightly different texture on the first note but very difficult to place it with A or B so I gave up on that and focused on the 6th.
 
I seem to do better with short passages. I've also tried just to sit back and "feel" the whole experience without focusing on specific passages but not seen much success.
 
Are others posting more convincing logs for 16 vs 24 (down to 1%?)

 

[RRod]

Wow. Tough crowd. All I did was use minimal flags and default settings and dared to share interim results. Before I could shake my audiophile tail I'm jumping off skyscrapers into a fictional universe. Nice to see everyone has an open mind.

 
Oh the extended Kirk commentary wasn't about you, just the general problem with the ABX MMO. Do you have a link for that Linn track?