The validity of ABX testing - Page 2

[QUOTE=mike1127;5845180]I don't get this nebulous concept that "data wasn't gathered in a proper manner."

I don't know how to make this any clearer for you. If you do not collect the data in a proper manner, any statistical analysis is useless.

If you do a sighted test, every subject tested will get 100% all of the time (even after 1,000 trials) unless they press the wrong button (highly unlikely). This will happen because if they know what the identity of A, B, X, and Y are, they'll be able to identify X as A or B in each trial without even listening to the music. You can't run statistical analysis and conclude anything meaningful when the subjects already know the answer to the questions you're giving them. You can do the math, but at the end of the day your hypotheses will either be meaningless or outright wrong.


Define P in this context, because you're not using a standard definition of a P-value here.

What would h1 be in this instance? You're begging the question that I'm asking you: would that test be sufficient evidence that cables have a causal effect on the audio system?

This is only true in some instances. It depends on what hypotheses you are testing.

If h0 is "cables a and b are audibly indistinguishable" and h1 is "cables a and b are audibly distinguishable," rejecting h1 is compelling evidence that h0 is likely true - this has nothing to do with statistics and has to do with logic - you cannot declare "not P" and declare "P" at the same time, assuming P is h1. In the case of statistics we're not dealing with logical truths, but you can substitute "not P is likely" and you still cannot conclude that "not P is unlikely".

Which is why I specified in my first post (in retrospect, rather redundantly, wrt quick-switching) that the samples would be representative.

Which is why I brought up the hypothetical in the first place: would you accept a DBT if it turned out that a positive result was ascertained, even if all of the factors that you claim reduce the sensitivity of the test were still there? To avoid being redundant see my response to PhilS's post (since this response is getting long enough as is).

Take mike's arguments (just because they're the most accessible, seeing as there are whole threads about them): all of his objections reduce to "something is messing with ABXs that reduce their sensitivity, and therefore they cannot be used to distinguish between cables." If, however, an ABX comes out with a positive result, despite the fact that ABX testing is insensitive (I'm granting arguments here), and mike (or whoever, I'm just using an example. This applies to many similar arguments) accepts these results as proof that cables make an audible difference, then in the real world (i.e. where ABX's have not in fact made a difference) mike's argument is circular: ABX's fail because they're insensitive, but if they showed that cables made a difference (in a hypothetical, i.e. a counterfactual) then they wouldn't be insensitive. At best this is just a heavily biased argument, but at worst this is circular. I'm not accusing him of anything, but I think that this logical structure either needs clarification or revision, which is why I made the thread.

Yes, I know, that big blob of text doesn't make much sense. I'll try to clarify it in a later post (one reason why I didn't reply for awhile).

That's a fair question, and worthy of discussion, it's just that I'm trying to get at something else in this thread.

Statistical hypothesis testing - Wikipedia, the free encyclopedia

Scroll down to "definition of symbols" and point out which p you're referring to (note that there are many, and always used in lowercase). Or, instead, find a website with a standard definition of the "p" you are using.

Re-read my first post: the fictional test uses a large sample size. I specifically put that there so that the discussion would not revolve around statistical power (Type II errors are non-issues when large sample sizes are involved).

I think the analogy is flawed because it's resting on a different assumption than with audio. In your analogy, you mention flying bullets. For one, we know that the flying bullets exist. Moreover, we know what the effects of flying bullets are outside of playing golf - for example, we know that flying bullets cause people to tense up and fear for their lives in almost any situation.

However, with the objections raised such as imagination contamination, it's a completely different story. We don't know if imagination contamination exists. We also don't have any conclusive proof that this condition manifests itself in anything outside of double blind audio testing, nor where we would find it, or even how to isolate it (unlike flying bullets, where we've witnessed them in tons of situations and can isolate their effects very easily). The only proof that imagination contamination even conceivably exists is that people are failing ABX tests where they "should" be passing them. Thus, the proof of their existence relies upon ABX tests yielding negative results. However, if ABX tests yielded positive results, and if one accepts those positive results as valid, then the only proof for imagination contamination's existence is gone. That means that, ultimately, the argument is circular:

If imagination contamination exists, ABX testing is flawed.
Imagination contamination exists only if ABX testing produces a negative result.
Therefore,
ABX testing is flawed because ABX testing produces a negative result.