I agree with this Prog Rock Man. It’s given me an idea of an analogy to show one of my concerns re DBT:
Test1. Sighted.
I’m watching TV with my wife (and lovely test assistant). The program is an outside broadcast, where the volume of both presenter and background noise is fluctuating. My digital AV amp’s volume control increment’s in 1db steps. In this situation, I can reliably tell the difference when I use the remote control to change the volume by 1db. It’s subtle, but completely obvious and repeatable time and time again. AFAIK, I can readily detect a 1db volume change – beyond reasonable doubt.
Test2. Secret Swap.
Unbeknown to me, my assistant now has the remote and secretly occasionally changes the volume by 1db (never cumulative, always 1db either side of start point). I’d bet money that I wouldn’t be able to spot the secret change, but if I did manage to do so, it would confirm what I already knew. And if I didn’t, then I would conclude that, based on all previous honest observations, Test2 is less reliable than Test1 for this particular application. If you accept this as reasonable, then it’s not such a big leap to...
Test3. DBT-like.
My assistant is given a modified remote, with a single button that randomly changes the volume by 0,-1db,+1db (never cumulative, always 1db either side of start point, and 0 is important in this test). It’s got an LED to let me know when a potential change is happening. I would probably not be able to repeatedly spot the difference in this test, although I may not bet as much money as I would for Test2.
Now, if the test moderator told me that had I failed Tests 2 and 3, and so the conclusion is that I can’t detect a 1db volume change, and also I must be deluding myself on Test1, then I would not accept that conclusion. Even though Test1 has it’s own flaws, I can reasonably reason (!) that Tests 2 and 3 have greater flaws for this particular application. For different applications, my conclusions may change. For example, if the TV was to output a continuous tone.