Some of you may remember the discussion I started months back about wanting to have a public ABX test for head-fiers. I found myself bored this saturday afternoon and decided to finally make it happen.
The purpose of this test is to prove that you can differentiate MP3 from Lossless. If you're new to the idea of ABX tests Wikipedia has some info here and here.
How to participate:
How To ABX (Step 12)
Excerpt from this page.
So if you think X is the same as A, click X is A and click "Next Trial", but if you think X is the same as B, click X is B and click "Next Trial". You'll want to do many trials. The more you do, the more significant your results will be. The minimum number of trials is 10. Any less and your results won't be very reliable.
When you're finished the ABX, don't close the window, just follow steps 13+.
Notes
Results
joze7205: 6/10 pval = 0.376
zachary80: 7/12 pval = 0.387
ailevin: 13/22, pval = 0.261
scompton: 5/10 pval = 0.623
agnostic: 3/10 pval = 0.945
Hardwired: 7/10 pval = 0.171
Haibane: 7/12 pval = 0.387
Over at the hydrogenaudio forums, a value less than or equal to 5% (0.05) is generally considered significant.
Should anyone want to participate in the future, they can feel free to email me their results and I will happily post scores in this thread. You can try the test multiple times and I will post your best score, although this does break the rules of a proper ABX test.
Thoughts
Obviously this is a very difficult test and I would think only a very small portion of people can pass it. If you can't distinguish between mp3 and lossless that doesn't mean you're wasting your time/money on expensive gear and it certainly doesn't mean that using lossless is wasteful.
I for one rip to lossless because I want the best sound possible. Even if I can't reliably distinguish from mp3, using lossless gives me peace of mind. This test wasn't intended to paint lossless advocates as delusional, but rather confirm that lame is a strong audio encoder even when using high end gear.
The purpose of this test is to prove that you can differentiate MP3 from Lossless. If you're new to the idea of ABX tests Wikipedia has some info here and here.
How to participate:
- Download Java ABC/HR here. It requires Java 1.5, it's available here.
- Extract Java ABC/HR
- Download the RARed samples. There are two samples to choose from, you can do one or both. A config file is included in the rar file.
- Extract the sample and config from the file you downloaded in step 3.
- Double Click abchr.jar from the Java ABC/HR file you extracted in step 2.
- Click Open ABC/HR Config
- Select the ECF file you extracted in step 4.
- Check "Show name in results file" and type in your head-fi name.
- Click ABX
- Change Sample B to Sample 1.
- Click Testing Mode.
- ABX (See Below)
- Click Finish
- Close the ABX window by clicking the X in the top right hand corner.
- Click File->Save Test Results and make sure your head-fi name and the song name are in the file name.
- Email the results file to me at hifire {at} gmail {dot} com
How To ABX (Step 12)
Excerpt from this page.
Quote:
| The ABX mode is meant to allow the listener to identify subtle differences between any two samples. Typically, one of the samples will be the original file. One of the samples is assigned to be 'A' and the other is assigned to be 'B'. Either sample A or Sample B is randomly assigned to be 'X'. The listener must determine whether X is the same as A or B. By performing this test multiple times, the listener can show with a specified level of confidence that his results are not by chance alone. |
When you're finished the ABX, don't close the window, just follow steps 13+.
Notes
- Results are encrypted so falsifying data is unlikely.
- The MP3 encoder is Lame 3.97 using preset v2.
- This is my first such test, so feedback is welcome!
- My instructions could probably do with some fleshing out, so feel free to ask questions and make suggestions.
Results
joze7205: 6/10 pval = 0.376
zachary80: 7/12 pval = 0.387
ailevin: 13/22, pval = 0.261
scompton: 5/10 pval = 0.623
agnostic: 3/10 pval = 0.945
Hardwired: 7/10 pval = 0.171
Haibane: 7/12 pval = 0.387
Over at the hydrogenaudio forums, a value less than or equal to 5% (0.05) is generally considered significant.
Should anyone want to participate in the future, they can feel free to email me their results and I will happily post scores in this thread. You can try the test multiple times and I will post your best score, although this does break the rules of a proper ABX test.
Thoughts
Obviously this is a very difficult test and I would think only a very small portion of people can pass it. If you can't distinguish between mp3 and lossless that doesn't mean you're wasting your time/money on expensive gear and it certainly doesn't mean that using lossless is wasteful.
I for one rip to lossless because I want the best sound possible. Even if I can't reliably distinguish from mp3, using lossless gives me peace of mind. This test wasn't intended to paint lossless advocates as delusional, but rather confirm that lame is a strong audio encoder even when using high end gear.
