KeithEmo
Member of the Trade: Emotiva
- Joined
- Aug 13, 2014
- Posts
- 1,698
- Likes
- 868
A VERY SIMPLE AND ALMOST FOOLPROOF TEST PROTOCOL
After spending a lot of time critiquing the flaws in other test protocols... I decided to offer a very simple and effective one everyone can use on their own.
(THhs can be a lot more difficult with hardware.... but it's not terribly complicated when it only involves software.)
This protocol can be adapted to a wide variety of situations.
It avoids most of the weaknesses of most others I've seen (for example it allows you to choose the test content and associated equipment).
It can be performed in a totally blind fashion.
It can be "scored" in a variety of different ways - and can be self scored.
It does require a small amount of help from an outside third party to set up (but it is comletely "self operated"; they don't have to stay around to switch wires or push buttons.)
Let's assume that, as an example, we wish to determine "If a 24/192k lossless audio file is audibly altered by being converted to a 16/44k lossless audio file."
1.
The first step is to select a test sample - in the highest quality format we wish to test.
So, in this case, choose a 24/192k file with which you are very familiar, and which you believe will be likely to lose quality by being converted.
We need to start with that file in WAV format (because a WAV file of a given bit depth, resolution, and time will always be the same size).
Name this file ORIGINAL.WAV
2.
Now, convert that file into 16/44k format, using any converter you choose, and any settings you choose (pick the ones you believe will be "audibly tarnsparent").
Now, convert that 16/44k file back into 24/192k format, again using whatever conversion software and settings you choose.
Save the new file as a 24/192k file named CONVERTED.WAV
You should now have two files of identical size and the same parameters.
If you were to do a bit compare they would NOT be the same.
JHowever, they will be the same size, resolution, bit depth, etc.
And, if the process of converting the original to 16/44k, and back again, was really audibly transparent, then they will be AUDIBLY identical.
3.
Now, put both files on a USB stick and give them to your helper.
Instruct your helper to create a set of sample files.... named SAMPLE01.WAV, SAMPLE02.WAV.... through SAMPLE10.WAV.
They are to copy ORIGINAL.WAV five times to make five of the samples.
They are to copy CONVERTED.WAV five times to make five of the samples.
They may decide which numbers to use for which any way they like (they don't have to be "really random" as long as you don't know what they are).
Your friend should also keep track of which are which - and list that information in a separate text file.
( A very simple program could be written to do this automatically.)
4.
You may now conduct the test any way you like.
You may listen to each sample as long as you like, as often as you like, and as many times as you like.
You may even listen to all of them on a varierty of different equipment if you like.
You may simply attempt to guess whether each sample file is a copy of the original or the converted version.
You may listen to both ORIGINAL.WAV and CONVERTED.WAV and then do a formal A/B/X test of the ten sample files.
Or you may simply listen to the files in various orders, and note when you believe you hear a difference between two of them when you play them in sequence.
(If they're "audibly identical" then you would NOT expect to hear differences between any two when played in any order.)
Whatever way you choose to conduct the test....record your results.
When you finally look at the "Key" text file....
You will either find that your results have no correlation whatsoever with what the files really are (in which case they really are "audibly identical").
Or you may find that you were able to group them with statistical significance...
Or that you were able to reliably note differences when you played files from different sources one after the other...
Or that you consistently noted that files from the same source sounded similar a larger percentage of the time.
And, of course, be careful not to fall into the "false significance trap".
(If you flip a coin enough times, odds are you WILL eventually throw five heads in a row, by pure random chance.)
(It is quite possible that patterns may appear by random chance... which is why tests like this should be run many times or with many variations.)
Obviously you can try this as often as you like, with as many files as you like, and YOU get to pick songs that you are familiar with.
(And it requires minimal assistance from a friend who merely needs to be computer literate.)
There are only two real requirements:
1. That you start with the highest measured quality.
(So, for example, if you want to compare CD quality and lossy compressed files, your ORIGINAL file should be the lossless one.)
2. That your sample files end up being the same format and size.
(So you can't tell which is which by looking at the sizes of the files, or the indicator in your player, or get a clue by which loads faster.)
After spending a lot of time critiquing the flaws in other test protocols... I decided to offer a very simple and effective one everyone can use on their own.
(THhs can be a lot more difficult with hardware.... but it's not terribly complicated when it only involves software.)
This protocol can be adapted to a wide variety of situations.
It avoids most of the weaknesses of most others I've seen (for example it allows you to choose the test content and associated equipment).
It can be performed in a totally blind fashion.
It can be "scored" in a variety of different ways - and can be self scored.
It does require a small amount of help from an outside third party to set up (but it is comletely "self operated"; they don't have to stay around to switch wires or push buttons.)
Let's assume that, as an example, we wish to determine "If a 24/192k lossless audio file is audibly altered by being converted to a 16/44k lossless audio file."
1.
The first step is to select a test sample - in the highest quality format we wish to test.
So, in this case, choose a 24/192k file with which you are very familiar, and which you believe will be likely to lose quality by being converted.
We need to start with that file in WAV format (because a WAV file of a given bit depth, resolution, and time will always be the same size).
Name this file ORIGINAL.WAV
2.
Now, convert that file into 16/44k format, using any converter you choose, and any settings you choose (pick the ones you believe will be "audibly tarnsparent").
Now, convert that 16/44k file back into 24/192k format, again using whatever conversion software and settings you choose.
Save the new file as a 24/192k file named CONVERTED.WAV
You should now have two files of identical size and the same parameters.
If you were to do a bit compare they would NOT be the same.
JHowever, they will be the same size, resolution, bit depth, etc.
And, if the process of converting the original to 16/44k, and back again, was really audibly transparent, then they will be AUDIBLY identical.
3.
Now, put both files on a USB stick and give them to your helper.
Instruct your helper to create a set of sample files.... named SAMPLE01.WAV, SAMPLE02.WAV.... through SAMPLE10.WAV.
They are to copy ORIGINAL.WAV five times to make five of the samples.
They are to copy CONVERTED.WAV five times to make five of the samples.
They may decide which numbers to use for which any way they like (they don't have to be "really random" as long as you don't know what they are).
Your friend should also keep track of which are which - and list that information in a separate text file.
( A very simple program could be written to do this automatically.)
4.
You may now conduct the test any way you like.
You may listen to each sample as long as you like, as often as you like, and as many times as you like.
You may even listen to all of them on a varierty of different equipment if you like.
You may simply attempt to guess whether each sample file is a copy of the original or the converted version.
You may listen to both ORIGINAL.WAV and CONVERTED.WAV and then do a formal A/B/X test of the ten sample files.
Or you may simply listen to the files in various orders, and note when you believe you hear a difference between two of them when you play them in sequence.
(If they're "audibly identical" then you would NOT expect to hear differences between any two when played in any order.)
Whatever way you choose to conduct the test....record your results.
When you finally look at the "Key" text file....
You will either find that your results have no correlation whatsoever with what the files really are (in which case they really are "audibly identical").
Or you may find that you were able to group them with statistical significance...
Or that you were able to reliably note differences when you played files from different sources one after the other...
Or that you consistently noted that files from the same source sounded similar a larger percentage of the time.
And, of course, be careful not to fall into the "false significance trap".
(If you flip a coin enough times, odds are you WILL eventually throw five heads in a row, by pure random chance.)
(It is quite possible that patterns may appear by random chance... which is why tests like this should be run many times or with many variations.)
Obviously you can try this as often as you like, with as many files as you like, and YOU get to pick songs that you are familiar with.
(And it requires minimal assistance from a friend who merely needs to be computer literate.)
There are only two real requirements:
1. That you start with the highest measured quality.
(So, for example, if you want to compare CD quality and lossy compressed files, your ORIGINAL file should be the lossless one.)
2. That your sample files end up being the same format and size.
(So you can't tell which is which by looking at the sizes of the files, or the indicator in your player, or get a clue by which loads faster.)