Originally Posted by skamp
We're not scientists, but I believe that's what we're doing by debating the merits of DBT.
Do you agree that successful ABX tests are reliable, and that sighted tests are unreliable? And again, do you have anything better in mind?
Yes, I agree successful ABX tests are reliable at confirming differences. And poor at confirming preferences - e.g. the pepsi/coke results were skewed because the testers would be overtly influenced by sweetness in a quick swap test. Again, this limitation is more to do with the fast swap nature of typical ABX tests, rather than anything wrong with the ABX principle.
Yes, I agree that sighted tests are unreliable, because of expectation bias.
For what I call "Essentally Neutral Components", like cables, the most reliable test I know of is no test at all: Listen normally to your unchanged system for months, going through a range of circumstances (tired/bored/happy/sad/etc).
Then swap exactly one component and carry on listening to any of your recent heavy rotation albums - just enjoying the music.
If the new component is significantly better, the difference will hit you in the first few seconds. Any further listening is unnecessary, and will only confirm those first few seconds.
If you don't hear a difference in the first few seconds, then the new component is not for you.
If you then start swapping A with B, then you lose much of those months of "preparation" and quickly end up with all the problems of fast swapping tests.
For reasons I can't really explain, I don't appear to have an issue with expectation bias in the above test. On the basis that every single time I've been genuinely surprised with the results being different to what I had expected based on that component's price, sound signature reputation, etc. It's possible this is a sort of complex double delusion, but the results have been so clear and against expectation that I'm willing to live with that possibility.
The above test is obviously impractical and time consuming, So most of the time, I have to make do with a reduced version, which introduced unreliability - such is life.
It doesn't follow that what works best for me will work best for anyone else.
Edited by TheAttorney - 3/17/12 at 5:47am