[jaakkopasanen]

I present to you abxtests.com, a browser based app for double blind AB and ABX listening tests. This is effectively Listening Tests as a Service, meaning you'll be able to create your own listening tests easily and share it with other people.

The primary target user would be an audio scientist or an audio engineer but there are also many audiophiles who are interested in testing things blind instead of trusting their biases. I started building this app for myself because I need to conduct some listening tests without being able to have the participants in the same physical location. Since I realized others could use this too, the expectations went up quite fast from something scrappy just for my own use to this what I'm now very comfortable to share with the public and put place under a public scrutiny.

ABX app runs users through a test suite with any number of AB and ABX tests. Each test can be repeated arbitrary number of iterations for statistical significance. I put a fair amount of effort into ensuring the listener's user experience is very good, audio playback is smooth and that the app itself doesn't introduce any biases. The test suite starts with a welcome screen where the test author can give context and instructions to the participants. The welcome screen also has a survey form which is handy for collecting data about different demographics.

The last one in the test suite is results screen where the user is presented with all the choices the user made, simple statistics and a p-value. The p-value is calculated as multinomial PMF. The results also include statistics aggregated based on tags in the audio samples. This makes it possible to examine how different filters, codecs, etc stack up against different songs.

The tests are defined as YAML text files. YAML is a super simple syntax and requires no prior experience. Simply create the file, save it to a publicly hosted location (such as Dropbox) and share a test link with the participants. The test links look like this: https://abxtests.com/?test=https://www.dropbox.com/s/c00dylhi7ekqkfw/demo.yml?dl=0. The last part is the Dropbox share link of the YAML file. You would replace this with your own. See the project's Github page for full details.

Here's an example of a minimal YAML file

name: Minimal listening test
welcome:
  description: |-
    ### Minimal
    Minimal listening test
options:
  - name: lossless
    audioUrl: https://www.dropbox.com/s/9e92quf9tr7aj8s/Wintersun%20-%20Sons%20of%20Winter%20and%20Stars%20lossless.wav?dl=0
  - name: 64 kbps
    audioUrl: https://www.dropbox.com/s/epvit0keu9w7emp/Wintersun%20-%20Sons%20of%20Winter%20and%20Stars%2064%20kbps.mp3?dl=0
tests:
  - name: Lossless vs 64 kbps
    testType: ABX
    options:
      - lossless
      - 64 kbps
    repeat: 5

You obviously need to have the audio samples prepared too. Fortunately they too can be hosted in Dropbox.

I'd greatly appreciate any feedback about the app!

 

[71 dB]

Wow. Impressive.

ABX tests are not relevant in my life because:

- I am so old (50) that CDs are my format of choice (so no hassle with bitrates/lossy etc.)
- Absolute sound quality is not my priority. Quality (how well it suites my taste) of music is.

 

[bigshot]

With those criteria, just about any audio format would sound fine for you... lossy or lossless or high data rate or physical media.

 

[VNandor]

Hey,

Thank you for doing this, it looks like a great tool for blind testing. I looked at at the demo, and the way I'm allowed to interact with the samples could decrease my chances to pass the test. If the stakes were high to pass some blind test, I wouldn't pick this app, I would try to rely on foobar's abx comparator instead. It has two functions that makes the test easier the pass.

Foobar presents the tester with A/B - X/Y options. "A" and "B" are not randomized, they always represent the same samples throughout the trials which can help with passing the test.
Foobar also has a feature that lets the tester hone in on just a short part of the sample by letting them pick a starting positon and an ending position. In this case, remembering the playback position when making the switch isn't a helpful feature so foobar also lets the tester to disable it and replay the sample from the starting position when making the switch.

Maybe you designed the test the way you did on purpose (or I just missed something), I just wanted to say that in some cases, this design makes it harder to pass the tests compared to other ones.

 

[jaakkopasanen]

Hey,

Thank you for doing this, it looks like a great tool for blind testing. I looked at at the demo, and the way I'm allowed to interact with the samples could decrease my chances to pass the test. If the stakes were high to pass some blind test, I wouldn't pick this app, I would try to rely on foobar's abx comparator instead. It has two functions that makes the test easier the pass.

Foobar presents the tester with A/B - X/Y options. "A" and "B" are not randomized, they always represent the same samples throughout the trials which can help with passing the test.
Foobar also has a feature that lets the tester hone in on just a short part of the sample by letting them pick a starting positon and an ending position. In this case, remembering the playback position when making the switch isn't a helpful feature so foobar also lets the tester to disable it and replay the sample from the starting position when making the switch.

Maybe you designed the test the way you did on purpose (or I just missed something), I just wanted to say that in some cases, this design makes it harder to pass the tests compared to other ones.

Thanks for the heads up! I haven't actually ever used the Foobar's ABX comprarator but you're not the only one to bring it up so I guess I need to take some notes from it. The starting position (and maybe the end too) is something that I already have in mind but didn't have time to implement yet. It's certainly a good idea.

 

[Dynamo5561]

Nicely done mate! Beautiful interface and functionality. But the example test was too easy

Maybe one recommendation: It would be good to see how many songs will be testet. After the first song I thouhg I will be finished. Then the next came and I didn't know how many will follow, so I just clicked very fast to see if I will see a result at the end or not.

 

[Dynamo5561]

Also what I was wondering for some time now is what maximum bit / sample rate you can play in the browser? I mean there are limitations with amazon hd player or even the windows music player on a windows pc. WOuld be really interesting to know.

 

[bigshot]

In Mac, the browser sends payback through the Mac's own playback hardware, so it should be able to play anything the comp is able to play. However, it it's something it can't play, it might downsample it on the fly without telling you.

 

[vergesslich2]

This is a good service, very well done, had fun trying it, but I didn't wait for the drop in the hard core music so I missed the huge bitrate drop there.... I get frightened to easily ; )

https://abxtests.com/?results=ZlpFan92XFdmRhxDfGhtWm4wTxU6W3A6Q3poJHIkXikMQxR%2FETp4ET89NQxePGAaIm07ew%3D%3D&test=demo.yml

PS: Is the site mostly about file encoding? Judging by the length of the claims and myths thread and latest posts in there, cables are a common topic. Maybe you could use a headphone recording dummy head to let people abx different cables or amps. Ok, cables would require your service to use the cable that is transparent towards other cables, the cable of cables, but maybe the site could let people test headphones and amps also. I listened to such recordings (mostly ones that demo different headphones) on my speakers a quite often. It's just... bit like home shopping. Not that I think it's how things should work, but it does not hurt to do it either.

 

[TuneInSoul]

https://abxtests.com/?results=kzMXKz1vHkJCXjVGT1MfY1AjBWEFXVlXVGkSNDM%2BCoFgHx1EB3gyDmJLUT1jJiB6R0mDWg%3D%3D&test=demo.yml

I have a question: if I use philips x1s to take the test,I found it hard to get 100% correct result ,but I got a perfect result when I used sony mdr-z7,does it means z7 is better than x1s at some aspects?