Setting Up An ABX Test + Simple Guide to Ripping, Tagging & Transcoding

[Brooko]

Sorry - I'm an absolute beginner when it comes to Android, and it's now been about 6-7 years since I converted back from being solely Linux based.

 

[glassmonkey]

I haven't done the test in a while, but did do the Tidal Test when they launched that and got all but the highly compressed Killers track.
 
That is something else to note, when doing the test you won't always be able to tell the difference because of poor sound quality of your master. Almost nobody will be able to tell the difference in modern metal, but put on some high dynamic range classical and people may find different. For me the difference is mostly in texture, soundstage depth, and amplification of the noise floor. You can look at a frequency chart for MP3s and see that they have a higher noise floor than flac. For me, the difference between mp3 and FLAC is almost always audible on well recorded tracks with a good set of headphones, but the difference on high bit-rate is much more likely to be a push. I doubt I could reliably tell the difference between CD quality and 24/192. I liked 24/192 better when I compared 24 bits of Christmas tracks (Linn Records, free tracks, starting soon. ) last year, and this was after reading everything that Archimago had to say about anything past CD quality being a waste of time and impossible to discern differences. 24 bits of Christmas allows you to compare a wide variety of well recorded tracks in multiple bitrates, including MP3. It is also likely to have tracks that have better transcoding than dBPoweramp can do.
 
Another monkey wrench in the A/B/X test, I find, is a lot of our experience of music is subjective, its about how we hear and feel the whole track, not a ten second snippet repeated until we make a conclusion. I recently did a test comparing SACD rips to DSF and SACD rips to FLAC88.2 and had my wife (sample size of 1!) listen to each of them with crudely attempted volume matching (DSF files can't have replay gain applied in JRiver), I then asked her to pick which of the tracks she liked better. On 5 out of 6 tracks, she picked the DSF file. I didn't fully adjust for confounding using an SPL meter, but I did try playing the DSF files quieter than the FLAC files, and the preferences remained the same. I didn't ask her to switch back and forth every few seconds or try to enguage anything cognitive. I think the cognitive engagement actually ruins the experiment for many people. In your normal listening, you might take notice when something sounds amazing, or you might 'get lost' in the music because it sounds so good, but you won't usually try to tie the music to a semi-abstract concept of format. I had the same experience as my wife on picking out DSD and FLAC, but I knew what the experiment was, and I had set the track list and then left it for several days. I didn't memorize the list, but I can't be sure that I didn't know what I was listening to on some level. There is also the possibility that the quality of recording set-ups varied, so I can't, in truth, eliminate all confounders. I'll be trying this on a friend or two at the end of the month. This time I've used an SPL app to try to less crudely match volume between the DSD and FLAC tracks.
 
The tracks were the following (convenience sample of tracks I know well from shared masters--shared masters confirmed by identical dynamic range levels)
 
The Rolling Stones - Sway
The Rolling Stones - Wild Horses
The Pixies - Velouria
The Rolling Stones - Mother's Little Helper
The Rolling Stones - Ruby Tuesday
The Rolling Stones - Angie
 
My wife preferred FLAC on Ruby Tuesday, and no other tracks. For me I could tell the difference between all of them except The Pixies - Velouria in this very non-scientific test. I'll see what i can do later with more samples.
 
I definitely concur on the master being the most important thing. If you want an example, just pull out just about any late 60s or 70s album that was remastered for the 90s. I don't really need to tell you what the trend on which sounds better will be. It is even true of 80s CDs versus later remasters. The best version of the Pixies - Doolittle is the original CD, the SACDs have less dynamic range, which can be heard in a reduction of crunch and less stage depth.

 

[landroni]

  Another monkey wrench in the A/B/X test, I find, is a lot of our experience of music is subjective, its about how we hear and feel the whole track, not a ten second snippet repeated until we make a conclusion.

Indeed so. In my view setting up ABX testing with 10s samples is for the most part futile. Just as it would be completely irrelevant using 1s samples. Using a whole track is the bare minimum, and 5-7 tracks in sequence with the same type of source would approach the optimum much more.
 
Mike Moffat of Schiit fame offers the following explanation:
http://www.head-fi.org/t/701900/schiit-happened-the-story-of-the-worlds-most-improbable-start-up/7725#post_11921090
"So I tried something new – I still did the A/B tests, matched levels, but allowed long-term listening to each; at least an hour or two with known recordings. Guess what! Suddenly I knew which was what. I tried it out on John B and Mike and Dave and all my other audio buddies. They called it too – tubes vs a bad solid state preamp. Every friggin' time. My enthusiasm had returned. This taught me that the human ear is an integral, NOT differential device."
 
Of course long-term AB testing requires considerably more time, patience, attention...

 

[Brooko]

Worthy of discussion gentlemen, but I wonder how you guys would fare actually doing the ABX between different formats of the same master - volume matched - using my method, but take whatever time you want.
 
Post your results.
 
And to be clear.
 

1. Take highest resolution track you have
2. Dither and resample to new resolution / bit-rate
3. ABX with replay gain enabled using Foobar 2000
4. Identify difference using the ABX tool.
5. For test to be valid - need at least 15 iterations in a single session to be statistically valid 

 
Mike was talking about difference in amplifiers - not music containers. And the discarded info (eg aac 256 vs FLAC) is supposed to be inaudible when played back.
 
I can't tell a difference - with decent gear.  Everyone who has actually tried it properly also fails.  The few who claim to be able to easily tell the difference seem to run a mile when I suggest introducing an independent observer to keep things above board.  Either of you want to step up and be guinea pigs?

 

[glassmonkey]

  Worthy of discussion gentlemen, but I wonder how you guys would fare actually doing the ABX between different formats of the same master - volume matched - using my method, but take whatever time you want.
 
Post your results.
 
And to be clear.
 

1. Take highest resolution track you have
2. Dither and resample to new resolution / bit-rate
3. ABX with replay gain enabled using Foobar 2000
4. Identify difference using the ABX tool.
5. For test to be valid - need at least 15 iterations in a single session to be statistically valid 

 
Mike was talking about difference in amplifiers - not music containers. And the discarded info (eg aac 256 vs FLAC) is supposed to be inaudible when played back.
 
I can't tell a difference - with decent gear.  Everyone who has actually tried it properly also fails.  The few who claim to be able to easily tell the difference seem to run a mile when I suggest introducing an independent observer to keep things above board.  Either of you want to step up and be guinea pigs?

I intend to do this later. Is there anything out-there that transcodes DSD64 to FLAC 88.2 for free? I'd like to do a scientific version of my casual experiment, also. I already have some tracks from 24 days of Christmas for testing PCM tracks. I still haven't deleted the 320 mp3s.

 

[landroni]

  I intend to do this later. Is there anything out-there that transcodes DSD64 to FLAC 88.2 for free? I'd like to do a scientific version of my casual experiment, also. I already have some tracks from 24 days of Christmas for testing PCM tracks. I still haven't deleted the 320 mp3s.

Could you also list the actual gear that you're using, from source to transducers?

 

[Brooko]

  I intend to do this later. Is there anything out-there that transcodes DSD64 to FLAC 88.2 for free? I'd like to do a scientific version of my casual experiment, also. I already have some tracks from 24 days of Christmas for testing PCM tracks. I still haven't deleted the 320 mp3s.

 
For all my transcoding I use dbpoweramp.  It's so good I bought it.  Cheap too - and well worth it.
 
https://www.dbpoweramp.com/
 
You'll need to manually add some of the transcoding options (DSD) - but very easy to do.

 

[waynes world]

Awesome thread @Brooko! I have been re-ripping my CD's to .wav format. But they are taking up a lot of space, so I wanted to know if I could tell the difference between flac (compression 0) and wav files. I finally got the abx comparator going and thought I'd start with the laughable comparison of 320 mp3's with the wav files before starting on the real test: flac versus wav. Hmm, not much point in doing the flac versus wav comparison, because I can't tell the difference between the 320 mp3 files and the wav files. Grrrrr!!!
 
Anyway, I do want to have lossless copies of my CD's regardless, so I suppose flac (compression 0) will be it instead of wav. But at least I know that it doesn't really matter lol. Although maybe it would as the gear improves?

 

[Brooko]

Hasn't been my experience (gear changes) - but that could be my hearing at work (high freq loss - age and tinnitus).  My advice - always keep an archive copy in the format you purchased (EG fro me I still have copies of my DSD purchases).  The use whatever format is convenient and meets your personal threshold. If/when you get more resolving gear - retest yourself.  You can then re-encode your higher res files if you need to.  My bet is that you won't need to though 

 

[WraithApe]

  Awesome thread @Brooko! I have been re-ripping my CD's to .wav format. But they are taking up a lot of space, so I wanted to know if I could tell the difference between flac (compression 0) and wav files. I finally got the abx comparator going and thought I'd start with the laughable comparison of 320 mp3's with the wav files before starting on the real test: flac versus wav. Hmm, not much point in doing the flac versus wav comparison, because I can't tell the difference between the 320 mp3 files and the wav files. Grrrrr!!!

 
Not much point? No point. Flac = lossless compression; de-compressed, the digital audio is a bit-perfect copy of the wav file, so attempting to hear a difference between the two would be futile to say the least! Wav files are simply a waste of space.

 

[pasi123567]

I tried this with my K702 an O2 amp and an 50$ dac from amazon (HiFimeDIY Sabre USB DAC ES9023)
Did 2 songs where I thought I can hear a difference. I compared 320kbps mp3 to flac:
-------------------------------------------
foo_abx 2.0.4 report
foobar2000 v1.3.15
2017-09-20 18:42:58

File A: Frozen Starfall - Ocean of Blossoms - 01 Sunlight Diamonds (FS Remix feat. Muffin).flac
SHA1: 58b54cbb0d418ddb3c75246e52d72eeca95ed205
File B: Frozen Starfall - Ocean of Blossoms - 01 Sunlight Diamonds (FS Remix feat. Muffin).mp3
SHA1: 624ca1738d63123c04770e963e628da1c1807bab

Output:
DS : SPDIF-Schnittstelle (3- Hifime Sabre Dac)
Crossfading: NO

18:42:58 : Test started.
18:43:53 : 01/01
18:44:11 : 01/02
18:45:01 : 02/03
18:46:05 : 02/04
18:47:13 : 03/05
18:47:43 : 04/06
18:47:58 : 04/07
18:48:29 : 05/08
18:50:04 : 06/09
18:50:26 : 07/10
18:52:10 : 08/11
18:52:39 : 09/12
18:54:08 : 10/13
18:54:32 : 11/14
18:55:01 : 12/15
18:55:51 : 12/16
18:55:51 : Test finished.

----------
Total: 12/16
Probability that you were guessing: 3.8%

-- signature --
ad2400de2c8b0783101b14ef85010f60a67200e4

-----------------------------------------------------------------------------

foo_abx 2.0.4 report
foobar2000 v1.3.15
2017-09-20 19:02:05

File A: Snail's House - Ordinary Songs 3 - 05 Lullaby.flac
SHA1: b5e77d369b3fa7c9f1f948d976fdf6d16f89c9da
File B: Snail's House - Ordinary Songs 3 - 05 Lullaby.mp3
SHA1: 1d982b09f122e5becbfcbdaa958063bfb581ece8

Output:
DS : SPDIF-Schnittstelle (3- Hifime Sabre Dac)
Crossfading: NO

19:02:05 : Test started.
19:03:54 : 01/01
19:04:16 : 02/02
19:04:58 : 02/03
19:05:17 : 02/04
19:06:10 : 03/05
19:06:49 : 04/06
19:07:54 : 05/07
19:08:11 : 06/08
19:08:32 : 07/09
19:09:19 : 08/10
19:09:52 : 09/11
19:10:19 : 09/12
19:11:55 : 09/13
19:12:17 : 10/14
19:13:02 : 11/15
19:13:27 : 12/16
19:13:27 : Test finished.

----------
Total: 12/16
Probability that you were guessing: 3.8%

-- signature --
85ac8c83fb78fcf50c19deb76c07db70567f56d5

I still feel like hearing a difference. I do think though that the difference is easier noticeable if you just enjoy the music. Hard to explain but even the tiniest difference can make the song worlds better.

 

[Brooko]

If it makes you happier, then use what makes you happy. The point of the test is for you to work out your own personal threshold. You used the same master, transcoded, and volume matched - correct? As long as you did all of that and you’re happy with the result - then use the results to steer your format choice.

 

[guisess93]

16/16 means you can completely differentiate between 2 samples right?

Since most of my flac files are only 16/44.1, and they don't even have high bit rate (just 300-1000 kbps), should I start from 320 kbps to lower and lower quality, or the other way round?

VBR vs CBR please?

Thanks

 

[Brooko]

Yes - correct. 16/16 means clear differentiation. You could tackle it one of two ways. Start at flac vs 320 and work back or flac vs 128 and work up. It’s up to you and your preference which way you go. It doesn’t matter if you use constant or variable. Suggest using what you use for your music normally.

 

[sonitus mirus]

I tried this with my K702 an O2 amp and an 50$ dac from amazon (HiFimeDIY Sabre USB DAC ES9023)
Did 2 songs where I thought I can hear a difference. I compared 320kbps mp3 to flac:
-------------------------------------------
foo_abx 2.0.4 report
foobar2000 v1.3.15
2017-09-20 18:42:58

File A: Frozen Starfall - Ocean of Blossoms - 01 Sunlight Diamonds (FS Remix feat. Muffin).flac
SHA1: 58b54cbb0d418ddb3c75246e52d72eeca95ed205
File B: Frozen Starfall - Ocean of Blossoms - 01 Sunlight Diamonds (FS Remix feat. Muffin).mp3
SHA1: 624ca1738d63123c04770e963e628da1c1807bab

Output:
DS : SPDIF-Schnittstelle (3- Hifime Sabre Dac)
Crossfading: NO

18:42:58 : Test started.
18:43:53 : 01/01
18:44:11 : 01/02
18:45:01 : 02/03
18:46:05 : 02/04
18:47:13 : 03/05
18:47:43 : 04/06
18:47:58 : 04/07
18:48:29 : 05/08
18:50:04 : 06/09
18:50:26 : 07/10
18:52:10 : 08/11
18:52:39 : 09/12
18:54:08 : 10/13
18:54:32 : 11/14
18:55:01 : 12/15
18:55:51 : 12/16
18:55:51 : Test finished.

----------
Total: 12/16
Probability that you were guessing: 3.8%

-- signature --
ad2400de2c8b0783101b14ef85010f60a67200e4

-----------------------------------------------------------------------------

foo_abx 2.0.4 report
foobar2000 v1.3.15
2017-09-20 19:02:05

File A: Snail's House - Ordinary Songs 3 - 05 Lullaby.flac
SHA1: b5e77d369b3fa7c9f1f948d976fdf6d16f89c9da
File B: Snail's House - Ordinary Songs 3 - 05 Lullaby.mp3
SHA1: 1d982b09f122e5becbfcbdaa958063bfb581ece8

Output:
DS : SPDIF-Schnittstelle (3- Hifime Sabre Dac)
Crossfading: NO

19:02:05 : Test started.
19:03:54 : 01/01
19:04:16 : 02/02
19:04:58 : 02/03
19:05:17 : 02/04
19:06:10 : 03/05
19:06:49 : 04/06
19:07:54 : 05/07
19:08:11 : 06/08
19:08:32 : 07/09
19:09:19 : 08/10
19:09:52 : 09/11
19:10:19 : 09/12
19:11:55 : 09/13
19:12:17 : 10/14
19:13:02 : 11/15
19:13:27 : 12/16
19:13:27 : Test finished.

----------
Total: 12/16
Probability that you were guessing: 3.8%

-- signature --
85ac8c83fb78fcf50c19deb76c07db70567f56d5

I still feel like hearing a difference. I do think though that the difference is easier noticeable if you just enjoy the music. Hard to explain but even the tiniest difference can make the song worlds better.

I'm assuming that you took a FLAC and converted it to mp3. What did you use to convert the files and what were the settings?