I absolutely agree.
Anecdotal evidence should never, in and of itself, be considered to be proof of anything.
But, the more anecdotal evidence there is that seems to agree about something, the more it indicates that there is something there worth testing.
You also need to remember that a lot of what we call "fact" is actually more accurately what science refers to as a model.
For example, for a long time, we were told that "matter is made of three basic types of particles, protons, neutrons, and electrons, and none of them can be broken apart".
This is what I was taught in school as "fact" - and that model was pretty much accurate enough to allow us to design nuclear reactors and atomic bombs.
Of course, it wasn't a fact. but merely a model... and now, after doing more research, we've discovered that those particles are actually made up of smaller bits and pieces after all.
(And we're back to devising more and better experiments to learn more about those pieces.)
The old model is still quite useful - but we are no longer able to ignore anecdotal evidence that conflicts with it.
We humans also have a nasty habit of assuming that OUR context applies to everyone.
For example, I would agree that I would not expect to hear audible differences between various audio interconects, placed between a modern preamp and power amp.
The reason is that most modern gear has a relatively low output impedance that won't interact very much with the range of capacitance we typically see in interconnects.
HOWEVER, the exact opposite is true of vintage consumer level tube gear, and modern tube gear based on many vintage designs.
(A lot of high-end vintage tube gear had a low impedance output, but the output impedance of most consumer level preamps was between 100 kOhms and 1 mOhm.
And, with that high output impedance, the differences in capacitance between interconnect cables would often produce audible and measurable interactions.
In fact, even modern magneitc phono cartridges are sensitive enough to load capacitance to interact differently with different interconnects - between the cartridge and preamp.)
Considering the fact that so many people anecdotally claim to hear differences between DACs, doesn't that suggest the need for extensive and thorough testing?
Does it really seem that tests of the limits of human hearing, conducted using static sine waves, are likely to be universally valid under all conditions?
And, even if a few people do turn out to be imagining differences that aren't really there, does that make it safe to conclude that
ALL of the reported differences are imaginary?
And, yes, if all of the existing tests are so flawed, and so easy to pick apart, doesn't that make you wonder if they
MIGHT have missed something important?
Likewise, an awful lot of people claim to hear differences between amplifiers...
How can you be sure, without a lot of testing, that "they
ALL must be imagining it"?
As for "absolute extremes"....
I believe it's up to the individual to decide for themself how important they are...
Assuming, for the sake or argument, I had a DAC that sounded different, but that difference was only audible with five songs, a certain amplifier, and a certain pair of headphones...
Wouldn't it be up to me whether I considered that to be important enough to influence my decision about which DAC to own?
How about whether I decide to pay extra for an amplifier and speakers because they can reproduce that Telarc recording with the cannons in it without clipping?
Isn't it my money - and my choice about how to spend it?
And, just as being any sort of collector, isn't being an audiophile sometimes about "excellence for it's own sake".
Just
knowing that, even if you can't hear a difference, the distortion on your new amplifier is much lower than on your old one.
One time I actually owned a MIL-SPEC signal generator that was rated to be able to operate at the North Pole - at minus fifty degrees.
I never went to the North Pole... but it was a cool piece of gear anyway (no pun intended).
To me it's also a matter of semantics...
If there is an exception to the rule, then it isn't a rule after all, and we should say "it's usually true " or "true most of the time".
However, we should never say "something is always true" if it isn't.
That would be misleading (even with the best of intentions).
To some people an anecdotal impression is an invitation to perform controlled tests to verify it. For others, anecdotes are all they need, especially if it validates their preconceived bias. Similarly, absolute extremes are only relevant at the North Pole, in an anechoic chamber or in a space station, not necessarily in a person's living room. Yet some people are more concerned with extreme examples than ones that hold true for the vast majority of circumstances. Again, it helps if the extreme example validates.