Having seen so many audiophiles, musicians, professional recording and mastering engineers fail these tests, I am well aware of difficulty of hearing such artifacts.
This raises a number of issues, largely covered by 1. What we are listening for (trying to identify) and 2. How we are accustomed to listening and making determinations:
1. For example: What musicians tend to listen for is quite different to recording and mastering engineers. Musicians listen for the aspects of performance over which they have direct control/responsibility, the technical performance choices consciously or unavoidably made by the musician/s and the artistic merit of those choices. In this respect they have extremely acute listening abilities, especially as far as their own particular instrument is concerned. Indeed, the difference between what musicians and engineers are listening for is demonstrated by vastly different vocabularies. For example, musicians think in terms of pitch, engineers in terms of frequency and classically trained musicians have a very wide vocabulary (including mainly Italian terminology) to describe aspects of performance/note production/musicality which are of little direct concern to engineers. This is why historically a Producer was required, to effectively to bridge the gap between engineers and musicians. Having trained as a classical musician from my early teenage years, studied in a renown music conservatoire and then been a pro orchestral musician for a number of years, I assumed my listening/analysis abilities would be better than most pro engineers but when I switched over and became a pro engineer myself I found that although my assumption was correct in some respects, there were many other respects/aspects which I was unaware even existed and in those respects my listening/analysis abilities were therefore no better than an average, untrained member of the public.
2. As an engineer, I am accustomed to working/analysing in a certain way. For example, focusing on some effect/artefact, locating the points in the audio file at which that effect/artefact is most obvious, repeating/looping those points, isolating (solo'ing) individual channel/s, making qualitative judgements and adjustments and then pre-rolling the full mix to judge those adjustments in context. That's not how most musicians tend to work/analyse, typically because they do not have the tools to do so.
I downloaded the Harman test, just to see what it was testing and how difficult it was. I only had limited time, did it on my rather old laptop with an old pair of stock Apple ear buds, listened to the first trial of each test and picked 3 of the tests to go further and judge difficulty. Up to and including level 5 I found it easy, picking the correct answer without fail within a couple of seconds or so. At level 6 (with the "Peaks/Dips" test) I did a couple of trials and got the correct answers but I needed more than a few secs, so I stopped at that point. I gave up during level 5 of the "Reverberant" and "Noise" tests, again for time reasons due to the increased number of items to compare.
My observations:
A. Given more time and running the tests in my studio, I think I would probably be struggling at about level 8 and not progress further than level 9, although I'm obviously only guessing at the magnitude of differences between the higher test levels with the noise and reverb tests. Nevertheless, I'm fairly certain I would be unable to reach the minimum Harman requirement of level 12!
B. I'm not at all surprised many musicians failed, going back to point 1 above, some/most of those tests are not really what musicians have trained their hearing to detect. I'm also not surprised that audio engineers failed, at least partially due to the format of the test being so different to how we are accustomed to working/analysing (as explained in point 2 above).
C. I'm sure I would benefit from practising and improving my Harman level but I'm not sure how much. How much time/effort would it take and how much of my improvement would only be in terms of improving my Harman test score, greater familiarity with the test format and songs (where in those songs the various artefacts are most noticeable)? Looking at it another way, if it were possible to create an identical level 12 test with completely different songs, would a Harman "trained listener" still be able to pass or would their level be reduced by their lack of familiarity?
D. Following on from the last point, I'd be interested in the results of some further tests. For example, how would Harman trained listeners' level be affected if they didn't know what they were listening for? IE. A test where the test level could simultaneously be altered for all of the 13 different band and attribute tests but the listener would not know which of those tests her/she was listening to. Without knowing what aspect of the song and which artefact to focus their concentration on, would that trained listener still be able to pass level 12? Another example: If instead of trying to identify which of 12 EQ bands one peak or dip belongs, I wonder how they would do in the same test but with say two or three simultaneous peaks and dips instead of one? Obviously that's a more difficult test but it's possible the gap between Harman trained listeners and audio engineers would be smaller as audio engineers are more accustomed to this sort of scenario.
The topic is what standard bar of fidelity do we set that when we talk to someone else, we are confident they can't hear any artifacts there.
I'd really like to answer this fully but don't have the time, except to mention one point: Again, there are two sides to this coin. I take it you are aware of the McGurk Effect and that we can easily hear a difference where in fact there is none. Extrapolating this, the "bar of fidelity" could be (erroneously) argued to be infinite, a fact exploited by many audiophile manufacturers. The other side of the coin, there are those who can hear artefacts which exceeding few others can but it requires an entire set of specific conditions. It's also been demonstrated that some people can hear up to about 25kHz but again, only certain people and only under specific (and potentially dangerous) conditions. In real life, how likely are we to find these people AND encounter the required entire set of specific conditions?
But even if I cannot consciously identify that there are distortion artefacts present in the recorded music or play-back system that I am listening to, yet alone what they might be, I suggest that unconsciously I perceive these distorts, at least at their higher levels, so that this distortion gives me a sense of conscious unease or dissatisfaction with what I am listening to. That is why I want to listen at least to Red Book standard recordings rather than MP3 and am willing to pay for a four figure DAC over a $29 one, although I never say no to a bargain!
1. To be aware of something only subconsciously, you still need to be able to hear it! Much of my work is directly in this area, I'm deliberately trying to affect the audience's perception without them being consciously aware of it and the levels we're talking about are very significantly above the thresholds of audibility.
2. As mentioned by others, there is the audiophile myth that more money gets you a better performing product. While that is typically true up to a point, in the audiophile world as the price increases the performance either effectively stays exactly the same or in a surprising number of cases actually gets worse once we get beyond that point and, that point is often significantly lower than audiophiles realise (or are willing to admit)! You yourself, earlier in this thread, posted an example of a $150 USB 2 audiophile cable which actually failed to meet USB 2 specifications.
G
