Originally Posted by aimlink
Again, great responses.
I'm pretty sure that I fully understand you and agree. Very nicely stated. This is an important post.
Thanks , after all I'm a Mathematician, I'm supposed to know and dominate at least this level of things.
The problem here isn't with a test that successfully demonstrates an audible difference. The statistical thresholds are that rigid that a positive result can be taken on face value.
It's the test that fails to demonstrate an audible difference that needs scrutiny. Can you seriously conclude that there's no audible difference. It's not all the time that increasing the numbers of test subjects will necessarily prove validity.
No, you conclude that there is no evidence to say that there is a difference, which is different than stating that there is none categorically.
As to analog vs digital hearing. I think what beeman is getting at here is:
If you send a complex stimulus to a robot with sensors, it will always register the stimulus in the same way. As a result, an detected stimulus will always be detected as such and in the same way, every time, once the stimulus remains the same. This is far from the truth for human perception and hearing. We listen to the same stimulus in different ways when it's presented to us. We will hear something in complex sound in one pass and not hear it in another. The more subtle the sound stimulus is, the more likely it is that we may not notice it on each pass, especially if it's embedded within a complex sound presentation. This is why I'm not surprised that those who pass DBT's often have to train themselves to do so.
Correct. In other word you are saying that there is non zero variance between human observations. That not only does not go against DBT (and a rigorous statistical hypothesis test), but is the very reason why statistics is used (will detail in next paragraph, which I decided to split).
Now, if someone fails a DBT, how does the DBT differentiate between someone who is just guessing and guesses correctly 25% of the times vs one who genuinely hears a real difference only 25% of the times because it's difficult to hear for whatever reason.
Correct, even if you hear differences there is a good chance you won't get it right 100% of the time, there are lot of variables that affect, even time of the day and mood of the participants.
Are we really saying here that you are truthfully seeing/hearing something only if you see/hear it all the time when the stimulus is presented, no matter how transient.
No, we are not saying that.
However, I do realize that by experimentation and statistical analysis one could not make a solid claim unless one is sure it could not be guessing that's underway. Unfortunately, we take this obvious requirement for conclusive evidence to erroneously assume that if a DBX does not prove an audible difference between two cables, it then means that there cannot be an audible difference. We cannot do this in a subjective/analog environment of hit and miss depending on a lot of factors peculiar to the sense of hearing and how we perceive sound.
One cannot conclude that. There are various reasons.
First this is statistics, all is chance, as I've stated before even the best tests can be wrong (although unprobable, but possible and it happens). Think about this way, the probability that one very good test is wrong is very small, but the probability that at least one test is wrong is actually bigger than what one believes, just as you need only 23 people so that the probability that 2 have the same birthday is more than half.
Second the conclusion that a negative results gives you is NOT to say that there is no difference. Is to say that there is no evidence that supports the alternative hypothesis, ie there is no evidence to say that cables make a difference, it is VERY important that distinction, since this conclusion is supported and is open to be proven wrong by further studies or results.
Third the results of a DBT of a single person are NOT statistically relevant for science, whether he gets 50% or 100% correct. The probability of error with a sample size of one is kind of huge, you need more than one subject to discard that the subject has super hearing, or that he is the luckiest guy on earth, or he was in a good mood, or that the stars suggested the result or whatever other factor there is, including the analogue nature of hearing as is being told by beeman. The selection of a really random sample is VERY important to reduce the probability of error due to all the factors, the more factors the bigger sample size you need. Note that this doesn't say anything about the validity of a single person's DBT for that person.
But as I stated before, the focus is that generally saying an innocent is guilty is worse than releasing free an assassin, so with the conclusion that there is no evidence to discard the Null Hypothesis science operates with the probably safer choice of assuming the Null Hypothesis.