is this really a problem with blind tests?
Jul 9, 2016 at 8:36 AM Post #121 of 126
  wow. you guys were active.
@johncarm.  you're demanding a all lot of things at a degree of precision that you should very well know can't be demonstrated while testing audibility on subjects. even MRI and other advanced scanning of the brain are questioned when it comes to making correlation between brain activity and what we can tell from them. not long ago there were some concerns about how such evaluations could lead to exaggerated conclusions. so it's very much a work in progress.
 
also you had your idea from the start and instead of looking into how to test the relevance of that idea yourself, you've been going on rampage to question and argue about pretty much anything but your own hypothesis. so of course people are starting to feel like instead of being curious, you're just here to play games. you can feel in in the air, read allusions in a few posts and I also got PMed on the matter. so I'm slowly starting on the path of statistical evidence against your good will.
 
 
about the null test, the 2 things you need to look up are obviously masking and hearing thresholds(and you could probably spend the next year just on those 2 points and still not have certainty like the one your demand from us). we can expect something at -80db below music to go unnoticed because of hearing threshold, ambient noise, and of course a very high chance of masking by the loudest music. but also because we simply don't have unlimited dynamic range when listening. our ear can trigger a protection mechanism that will move our sensitivity down by (IDK about 20db I guess?) and help reach higher total dynamic range, but it cannot be in both states at once, so when we can perceive the loudest sounds right before hurting, we can't notice the quiet sounds we would be able to detect in a quiet room without the loud noises. so the listening level will affect what we can hear(and counter to intuition, we tend to notice things better a little before the protection mechanism kicks in, so that's usually in the 60/70db loud.
and we can't tell if something at -40db will or won't be heard under music because while of course we can hear something 40db below music at normal listening level, it may or may not be masked by the louder music content.
 
 
 I suggest you spend some time doing a few blind tests(foobar's abx plugin is one of the easiest way to start), and a few null tests(audio diffmaker). and think about ways to test your idea about long samples vs short samples. maybe the results would convince you of what we apparently cannot.

 
I'm not demanding a precise answer. I was just trying to get any answer at all other than a flip one. Look, mapping out a region in which you don't know something is a way to advance knowledge. Looking for ways to approximate an answer advances knowledge. 
 
Yes, I finally got an answer a few hours ago about the application of masking theory to the difference test. As I have written in several places now, I'm getting the textbook.
 
I'm interested in comparing a lot of things that can't be done via Foobar ABX, such as DACs and amplifiers. I HAVE done lots of blind tests in both short and long form. As I have stated repeatedly my intuition is that I can tell more precisely in long form with DACs and amps, but I have been unable to check that against rapid switching because I don't have a way to do it. (This is a problem a lot of audiophiles have... it takes resources to conduct blind tests on bulky components that have to be switched out.)
 
Apparently your intuition is that rapid switching is better. So can you give me your evidence for that? What is the evidence that rapid switching is better?
 
Jul 9, 2016 at 8:46 AM Post #122 of 126
You can't possibly get any more than special case answers, because there is no "general case". To give another example, if B differed from A only in terms of a phase shift, you could get a difference signal as loud as A (because you just can't null a phase-mangled copy of a signal on the original) yet one would still not be able to tell A and B apart except in cases of extreme phase distortion.
 
HiBy Stay updated on HiBy at their facebook, website or email (icons below). Stay updated on HiBy at their sponsor profile on Head-Fi.
 
https://www.facebook.com/hibycom https://store.hiby.com/ service@hiby.com
Jul 9, 2016 at 9:10 AM Post #123 of 126
You can't possibly get any more than special case answers, because there is no "general case". To give another example, if B differed from A only in terms of a phase shift, you could get a difference signal as loud as A (because you just can't null a phase-mangled copy of a signal on the original) yet one would still not be able to tell A and B apart except in cases of extreme phase distortion.

 
Maybe the term "general case" is not the right one.
 
The point is to examine what we know about the question in broad contexts. 
 
You are getting hung up on (A-B) alone. I never asked if you could tell the difference from that except when Spruce and Gregorio started to talk about it. The primary question is: can you answer the question given A and B. 
 
Masking theory would apply in a pretty broad context I think. If A is some kind of music or natural sound that covers a wide spectrum, and B is the same thing with another musical or natural sound added, that would be a large category to which it might be useful to apply masking theory, and the context could include a large range of amplitudes of the injected sound.
 
So that's an example of an answer that is not a special case. 
 
Jul 9, 2016 at 9:48 AM Post #124 of 126
I'm not demanding a precise answer. I was just trying to get any answer at all other than a flip one. Look, mapping out a region in which you don't know something is a way to advance knowledge. Looking for ways to approximate an answer advances knowledge. 

Yes, I finally got an answer a few hours ago about the application of masking theory to the difference test. As I have written in several places now, I'm getting the textbook.

I'm interested in comparing a lot of things that can't be done via Foobar ABX, such as DACs and amplifiers. I HAVE done lots of blind tests in both short and long form. As I have stated repeatedly my intuition is that I can tell more precisely in long form with DACs and amps, but I have been unable to check that against rapid switching because I don't have a way to do it. (This is a problem a lot of audiophiles have... it takes resources to conduct blind tests on bulky components that have to be switched out.)

Apparently your intuition is that rapid switching is better. So can you give me your evidence for that? What is the evidence that rapid switching is better?


Though you can't ABX DACs and amps as easily as digital files, you could plausibly just as easily test your hypothesis with the latter--take a set of A and B audio samples for which your ability to tell apart is marginal, run two sets of ABX tests, one for which you time each desired switch between A, B and X to be at least, say, two minutes apart, and one for which you time each switch to be at most, say, 10 seconds apart. Compare your accuracy score after a set number of runs in each case. Repeat with a few different sets of A and B.
 
HiBy Stay updated on HiBy at their facebook, website or email (icons below). Stay updated on HiBy at their sponsor profile on Head-Fi.
 
https://www.facebook.com/hibycom https://store.hiby.com/ service@hiby.com
Jul 9, 2016 at 10:53 AM Post #125 of 126
  [1] My frustration is that I was getting absolutely nothing but flip answers or special cases. ... [2] We can map out some cases where we do have an answer. [3] We can ask what additional tools we might need in other cases.
[4] We can create some theories that help us understand what's going on "under the hood" and test those theories. We can ask some questions for further development.
 
[5] You have given the best answer so far, which is that masking theory applies.

 
1. No, you were getting accurate proven answers to special cases and as JB has stated, in effect they are all special cases!
 
2. We've done that but you refuse to accept those answers even though they are provable!
 
3. I assume you're talking about tools to measure brain function/perception because we already have the all the tools to measure acoustic sound waves.
 
4. That is exactly what psychoacoustics is! A fundamental problem here is that your arguments are based on what you think you know, on your intuition. This presents a serious problem because while psychoacoustics has advanced our knowledge significantly, in some respects we actually "know" less than we did before psychoacoustics! For example, volume/loudness is such a simple, fundamental concept, we hardly have to consciously think about it and even very young children understand and can do it. However, psychoacoustics has demonstrated that volume is actually a complex perception rather than a "reality" (property of sound), as was previously assumed. It's only in the last few years that we've been able to model loudness and even now, the model is only an approximation and only under limited circumstances because it doesn't account for all the variables which are thought to be involved in the task of evaluating loudness.
 
5. No disrespect to castle but that's not "the best answer so far", it's just one answer which depends on the specific case.
 
G
 
Jul 9, 2016 at 11:30 AM Post #126 of 126
  I'm not demanding a precise answer. I was just trying to get any answer at all other than a flip one. Look, mapping out a region in which you don't know something is a way to advance knowledge. Looking for ways to approximate an answer advances knowledge. 
 
Yes, I finally got an answer a few hours ago about the application of masking theory to the difference test. As I have written in several places now, I'm getting the textbook.
 
I'm interested in comparing a lot of things that can't be done via Foobar ABX, such as DACs and amplifiers. I HAVE done lots of blind tests in both short and long form. As I have stated repeatedly my intuition is that I can tell more precisely in long form with DACs and amps, but I have been unable to check that against rapid switching because I don't have a way to do it. (This is a problem a lot of audiophiles have... it takes resources to conduct blind tests on bulky components that have to be switched out.)
 
Apparently your intuition is that rapid switching is better. So can you give me your evidence for that? What is the evidence that rapid switching is better?

I told you, my first assumption comes from tests about the reliability of memory that revealed those crucial first seconds(and going up to .. I believe it was around 5 or 6 seconds on one paper, and maybe 10seconds on the other one), with everything longer getting significantly less accurate when asked about. what anybody would expect really, even remembering a cellphone number will be hard but possible for me to repeat just after it's be told to me. but if I wait a minute, I'm sure I wouldn't have more than 3 or 4 numbers, and maybe less if I kept listening to other stuff in that minute.
now if the person tells me one number per minute and nothing else, I will learn those number within the time I've been given and I expect not to be too bad in the end. but what if the guy keeps talking while the minute passes? I'm totally screwed.
 
my second reason is that detecting a rapid change is the most obvious thing needed for any animal to survive, so I assume that living animals are indeed good at it for Darwin reasons.  when looking for a difference between 2 tracks, I imagine an efficient way is to switch almost instantly and "see" if it triggers a reaction that something is going on or not. and why a short sample? well because I'll have identified the loudest difference by measurement, or by listening to the 2 samples a few times, and will want to concentrate on what I found to be the revealing passage instead of wasting time on the parts where I don't think I found anything obvious.
 
my next reason is a practical one based on experience in actual blind test.
let's say we have only a small volume difference and everything else is identical. do you think you need 20mn samples to notice a volume difference? how would that improve anything? same with pretty much any correlated change/distortion. the louder the sound, the louder the difference. it comes down to about the same problem as finding a volume difference and if it's audible, it will be on a well picked short sample.
now let's say it's some treble thingy I'm trying to get in a mp3 vs wave test. some bells or castanet or whatever. I will listen to the track(a long one), try to find the moment when such instruments are played. I pretty much don't care about the rest when those instruments are not used. something feels wrong(or maybe I'll look at the the null in audacity to find where the most change occurs), then I will stick to that passage and switch back and forth to try and confirm it. if I feel like I got it, I'll just start an abx test to make sure.  if I pass, I'm done, there is an audible difference, I've demonstrated it, the end.  I couldn't care less if there are more as that was never the purpose of the blind test. 
now if I fail, then I will listen again to the all song and switch from time to time trying again to find that one passage that I hopefully can identify repeatedly. the demarche itself pushes me to go from long to short, and I doubt that many people would bother listening to the entire song every time when they have found a short sample that works.
 
while on the other hand the arguments for longer listening seems to consistently come from people who don't do much if any controlled testing. so of course my skepticism is always strong for those feedbacks.
 

Users who are viewing this thread

Back
Top