Would it be a good idea to do the following, or can there be any problems with this from a scientific perspective:
First do extensive sighted listening to the things you want to compare, take as much time as you like. Get "familiar" with the sound, and try to find "suspected differences" and specific audio fragments with wich you seem to hear these "suspected differences". Then do a well set up double blind ABX test using those specific audio fragments. Only if you can confirm the "suspected differences" in this test they can become "accepted differences".
In fact, I am wondering if this approach should not even be considered mandatory before concluding that there is no difference (or rather: I think this approach would raise the level of confidence that there is no difference). (Certain differences are maybe only audible with specific kinds of audio fragments, and not by all people. If you just do an double blind ABX test with a "random group of people" and a "randomly chosen set of audio/music fragments" and just look at the "statistical relevance" of the deviation from 50% correct score you could get to a false conclusion if for example none of the audio/music fragments was suitable to hear the difference.)
There are a few problems here and in fact there's a better idea, one that is commonly (but not always) employed. Firstly the problems:
A. It's not such a good idea to use say an ABX test for "suspected" differences. First find out if there is an actual difference, which is generally easier, quicker and more reliable anyway. This way, the ABX test is hopefully going to answer a specific question: "Is this specific difference audible". As I said in my post to AudioThief, "Commonly we have two sides to the scientific/rational approach", and "commonly" indicates "often but not always", and that's because we sometimes don't need the "observed evidence" side. For example, if there is no actual difference then there's no point in any sort of listening test because either it will just confirm what we already objectively know or it won't, in which case it must be a faulty test. "Observed evidence" might also be unnecessary because it's irrelevant for another reason or has already effectively been done. For example, if the difference is say just random noise at -140dBFS, then we can't even reproduce it in the first place, or if the difference is above 30kHz there's zero chance we can hear it because we've already extensively tested frequency hearing thresholds over many decades, with countless subjects, and 30kHz significantly above that threshold.
B. The goal of an ABX or other double blind test should always be to pass it. I can't think of a single published scientific audio double blind study where they used a "randomly chosen set of audio/music fragments", they always use a set of carefully chosen of audio fragments which maximise the chances of the differences being audible. In fact, a test signal will often be used, a signal which doesn't necessarily ever exist in music/audio recordings but is specifically designed to make the particular difference being tested more audibly detectable. This enables us to apply the results of the test to a wider group of people than just the test subjects, with a higher degree of confidence. A good example of this is the testing that has been carried out for random jitter audibility. Using a specifically designed test signal, some subjects have been able to discern random jitter down to just two or three nano-seconds. However, this signal never exists in nature or in any audio/music recordings and when doing random jitter tests using music recordings, the lowest reported results have been 200 nano-secs, although most subjects struggle to discern lower than 500 nano-secs. From this we can draw some conclusions with a high degree of confidence, for example, while different people have different hearing acuity and different levels of listening skills, we can say with very high confidence that no one one can hear random jitter below two or three nano-secs with musical material.
C. It's not really a problem. If those test subjects could not discern a difference then you have your answer, it makes no difference. The problem only occurs if you try to apply that answer to other people/everyone else.
And now the better idea, which also has several points (but can depend on exactly what it is that we're testing):
A. We don't just have a "random group of people" but a selected "random group". I know this sounds contradictory but let me give an example: If we randomly pick 20 people off the street, the probability is high that they'll all just be members of the public, it's unlikely there will be any audiophiles or any experienced, professional musicians or sound/music engineers in that group. So depending on what we're testing, we can (and often do) select a group of random test subjects that deliberately includes representation of all 4 of these groups.
B. Essentially the same as "B" above, not use a "randomly chosen set of audio fragments" but a very carefully chosen set of audio fragments or a specifically designed test signal.
C. Quite commonly the test subjects are "trained" for 20 or 30 minutes before the actual DB test, especially if the difference being tested is not one we're accustomed to identifying. This is typically done using either test signals or a section of modified music that's had the specific difference added at an artificially very high level, to make it easily identifiable and then over the course of the training the amount of this added difference is reduced. This acclimatises/sensitises the subjects to the difference being tested, optimises their chances of detecting the difference during the test and makes the results applicable to others with a higher degree of confidence.
[1] I'm seeing a major flaw here in terms of how the idea of "subjective vs objective" is being conflated with the difference between blind and sighted tests. Doing a double blind test does NOT ensure you of getting objective results...
[1a] For one thing, if you have an expectation bias NOT to hear a difference between two devices, a simple blind or abx test will do nothing to eliminate it.
[2] For another thing, a blind test doesn't eliminate purely subjective responses; for example, even in a blind test, a subject might PREFER the sound of a device with more distortion.
[2a] (And, even if you're supposedly testing for "whether they hear a difference or not" they may still apply a subjective standard to reporting "tiny differences" or "significant differences".)
[3] There is a classic example of "visual bias" that is often described in psychology textbooks...
It is in fact a demonstration that, due the complex way in which our rbains are wired, input from one sense may affect how we interpret input from a different sense.
We actually, OBJECTIVELY, experience the taste of the steak differently when we see a green color associated with it - and an MRI will show a different sensory response in our brain.
1. Yes it does, unless it's a flawed double blind test, in which case one may still have some confidence in the result but obviously not as much.
1a. Yes it will, unless it's a flawed blind or ABX test. Firstly, an expectation bias not to hear a difference doesn't necessarily result in a false negative. A number of times I've done ABX tests not expecting to detect a difference but have, or even thought during the test that I wasn't detecting a difference but the results demonstrated otherwise. This isn't always the case of course, in which case "a simple blind or ABX test" is flawed and something other than a "simple blind or ABX test" is called for (in order to arrive at a higher confidence level result), for instance a more complex ABX test. A bigger/wider sample size for example and/or some control iterations (EG. One or more of the samples has an artificially high difference that's easily discernable).
2. I don't understand this assertion, a blind test is not designed to "eliminate purely subjective responses" it's designed to test them! What a blind test is designed to eliminate is biases which affect our "purely subjective response" (hearing). Most obviously, a blind test is designed to eliminate the biases introduced by sight and leave us with just the "purely subjective response" of hearing. If you want to eliminate subjective responses then you can't use ABX or any listening test, you have to use an objective measurement.
2a. If you're testing for "whether they hear a difference or not", it doesn't matter if the subjects detect a tiny difference or a massive one, just that they detected a difference.
3. Sorry, that's completely illogical. If two pieces of steak objectively have the exact same flavour, texture, etc., but we experience a difference between them, then by definition that experience cannot be objective, it must be subjective! And, if our subjective response (of taste) is influenced by a different sense, say sight, then of course there will be some difference in sensory response in our brain, it will obviously include more activity in those areas of the brain responsible for processing vision and probably some differences in those areas of the brain combining all the information to create the overall/final perception.
G
