I’m really enjoying reading your posts so far, they’re insightful, open minded, and very interesting. Thank you for taking part in this discussion. Your one of the few people here that really add a lot to this discussion, and one of the people that I look forward to reading from.
Unfortunately, I don’t have any data relative to an audio test. However, I do have data that I personally believe can be used and is useful in demonstrating that at least the tests can be conducted in a better fashion. However, I’m not really allowed to share this publicly, but if you send me your email I can send it to you as a protected pdf file. I would really appreciate if you don’t post it on the net though. (As soon as I’m back to work)
It was a short term memory experiment conducted on children (ages 7 to 12), to try and see if they can find better ways in which children can retain information. The most relevant part in the experiment was when the children were shown a circle with slices shaped like well a pizza. The circle had 4 colors and 12 slices. One of each color has its position changed in the circle.
The children were shown first picture A with circle and colors arranged in certain way for 5 seconds, and then shown picture B which had a slight different color arrangement for 5 seconds. Finally the children were shown picture X which was either A or B and were asked to identify if picture X matched picture A or B. What I liked about this experiment was:
- At the end of experiment they showed the children all of the pictures together and asked them to see whether they can identify if picture X matched A or B. one of the tests was so hard that none of children could solve it even when pictures were right in front of them.
- Some of the children just couldn’t identify which picture matched X because well their either were color blind maybe, or didn’t understand the question, or didn’t want to cooperate. They were removed from the experiment.
- They started using more pictures later on and more colors, and they were able to measure when it was normal for the children to notice the differences and when it was extremely hard to impossible for the children to remember or be able to notice the differences. They were also able to tell you whether the children failed because there was too much information being presented so they can’t remember it all, or because the colors were blending together and it was too difficult to notice the difference.
- When they had 6 pictures up and asked the children to try to identify which picture matched picture X, none of children were able to identify which picture matched X. When they used 2 pictures the rate of success was around 76% with girls having a higher success rate. When they used 3 pictures, the success rate dropped to 47% if I recall correctly with girls having a higher rate of success than boys ( I have read the document it a month ago, so the numbers might not exactly be accurate).
- They added marks later on, arranged the colors in certain way, and used different shapes, from bright to dark to see if that would make it easier for the children to remember the difference or notice them, which resulted in a 23% increased success rate.
The difference between this test and most of abx tests that I’ve come across when it comes to sound. Is that when the children failed, they were able to accurately tell you why they failed and even readjust the test so they can accomplish their goal. They were able to identify and measure to a very accurate point, and tell you the success rate of children, in any point in experiment.
When the children failed they knew why they failed. Last but not least, they were able to remove the children who would have failed for other reasons, reasons that shouldn’t be taken into account when giving a test like this. For example a child who is blind shouldn’t even be participating in this experiment.
However, most of these audio abx tests pretty much don’t really take anything into account if people fail. They don’t really bother in researching why people might have failed. Most of these abx audio tests really only are interested in the results when people pass the test. That is why I’m claiming that when an audio abx tests fails, you learn absolutely nothing of importance. You have no idea why the test failed. Because no measure have been taken to see why the test has failed.
Thank you again for taking part in this discussion.
Originally Posted by liamstrain
While you have presented a theoretical flaw in abx testing. You need to present evidence that the memory issues you raise, actually would in practice, invalidate the tests or even have a measurable effect on the results. There is a lot of evidence to support these testing methods as useful within their known limitations. You need more than your theory and assertions to overcome their use, or to redefine their scope.
It is an interesting argument. But you have not demonstrated anything that would indicate we need to adjust our methodology. Do you have data, for instance about what threshold of audibility is that would make abx less reliable?
No, as has been stated before, our experience has been that abx makes the differences easier to identify (by providing immediate contrast in stereo under realistic listening conditions - short term memory not-withstanding). Not harder. Especially with large enough samples to provide a measure of error correction (as most good DBT/ABX data should be).
You have stated repeatedly otherwise, do you have experience or data to back that assertion up?
No, we can never be 100% sure of anything. But I'm pretty comfortable with statistical significance and high percentages of confidence. Especially when corroborated by different methodologies and data sets.