Let's Prove The Null Hypothesis

[ILikeMusic]

Quote:

Originally Posted by bobsmith /img/forum/go_quote.gif Telling (or failing to tell) the difference in an "objective" test is meaningless if the test methodology itself is flawed.

Agreed. So, employ a proper test methodology.

 

[bobsmith]

Quote:

Originally Posted by ILikeMusic /img/forum/go_quote.gif That has nothing to do with the validity of the test, just set it up to be less time-dependent if you like. You can set up a test to distinguish a difference over short periods of time, or long periods, or whatever makes it easiest for the test subject. But in the end you will still be able to identify a difference, or you won't.

Yes, it could affect the validity. Many people believe that it may take days or weeks to fully appreciate certain sonic differences. Realistically, there is no practical way for a test to be structured to permit those lengths of time, and I am not aware of any that have attempted to do so. If, in fact, very long listening times are required for an accurate judgment, then the methodology of the tests using shorter times (pretty much all tests) is flawed and basically invalid.

 

[myinitialsaredac]

Quote:

Originally Posted by bobsmith /img/forum/go_quote.gif Yes, it could affect the validity. Many people believe that it may take days or weeks to fully appreciate certain sonic differences. Realistically, there is no practical way for a test to be structured to permit those lengths of time, and I am not aware of any that have attempted to do so. If, in fact, very long listening times are required for an accurate judgment, then the methodology of the tests using shorter times (pretty much all tests) is flawed and basically invalid.

This is my point exaclty, it may take days to find the difference in those two pictures, but no one is going to sit there for that long to discern them. Anything as complex as our sensory perceptions requires time and ability.

Dave

 

[bobsmith]

Quote:

Originally Posted by ILikeMusic /img/forum/go_quote.gif Agreed. So, employ a proper test methodology.

See my post above. You are assuming that it is possible and/or practical (e.g. commercially reasonable) to use a proper methodology.

 

[terriblepaulz]

Quote:

Originally Posted by PhilS /img/forum/go_quote.gif I've been a lawyer for almost thirty years, and I can tell you the evidence that consists of the perception (i.e., observations) of individuals is not considered bad evidence in a court of law. In fact, it is often considered some of the best evidence. And I would submit it is also pretty good evidence in terms of making decisions in our daily lives if you don't want to refuse to make a decision until a Nobel Peace prize winning scientist has weighed in on the issue. Of course, one should weight "good" evidence more than "bad" evidence. I didn't say anything to the contrary. What I said was than an open-minded person trying to find the truth about something should probably evaluate all the evidence before he comes to a conclusion -- not just the evidence he decides in advance -- based on his prejudices -- that he or she will accept.

No offense counselor, but I'm shocked that a member of the bar would make such a statement. Lawyers memorialize everything in writing so as not to rely on people's memories of what was said or done. What's a better case - a cop's testimony that the driver was drunk, or a videotape showing the driver falling down. A case that rests solely on witness testimony should not (outside of traffic court) survive the cross examination of any competent attorney. I could go on. There is a mountain of research on the unreliability of eyewitness testimony, and there has been a load of Frye and Daubert related litigation on the topic. (See, e.g., http://www.fclr.org/docs/2006fedctslrev3.pdf). And yes I know people are convicted on the basis of eyewitness testimony everyday. I procured a few hundred such convictions myself (I only have about half of your thirty years before the bar). That's a bug, not a feature.

There's a limit to the value of an open mind in certain contexts. In court, you call it "relevancy". In the context of comparing and measuring the performance of audio products, the perceptions of individual listeners is simply bad evidence.

 

[ILikeMusic]

Quote:

You are assuming that it is possible and/or practical (e.g. commercially reasonable) to use a proper methodology.

Any you seem to be hoping that it isn't.

'It's impossible to test' is a basic tenet of charlatanism and one should be extremely circumspect whenever anyone claims that situation to exist.

 

[myinitialsaredac]

Quote:

Originally Posted by ILikeMusic /img/forum/go_quote.gif Any you seem to be hoping that it isn't. 'It's impossible to test' is a basic tenet of charlatanism and one should be extremely circumspect whenever anyone claims that situation to exist.

Mm, it is not impossible to test; simply the tests that have occurred thus far are fatally flawed and should not be used as evidence.

Dave

 

[bobsmith]

Quote:

Originally Posted by ILikeMusic /img/forum/go_quote.gif Any you seem to be hoping that it isn't.

Not quite sure what you are trying to say. I am not a "believer" one way or the other (and I really do not care much about the outcome). You basically stated as fact that everything other than "objective" tests was irrelevant. I disagreed and pointed out (1) that all such "objective" tests to date are not be meaningful IF they all have the same/similar flaws in their methodology, and (2) that IF such flaws exist, they may be very difficult or practically impossible to correct.

Quote:

Originally Posted by ILikeMusic /img/forum/go_quote.gif 'It's impossible to test' is a basic tenet of charlatanism and one should be extremely circumspect whenever anyone claims that situation to exist.

That is a well worn mantra, but not really applicable in this case. That statement is directed to "theories" that are truly impossible to prove (e.g. prove there is no God etc.). There is no question it is technically "possible" to run a DBT that permits days or weeks for listening. In fact, it would be very easy to theoretically design a test to do this. Its just that no one (yet) has had the time, money or inclination to actually run such a test.

 

[PhilS]

Quote:

Originally Posted by ILikeMusic /img/forum/go_quote.gif All that matters is whether you can objectively identify a difference a not, the rest is side commentary.

That's all that matters to you, because you have decided in advance that's all you will accept. You're reasoning is circular, and you are not really contributing anything to this discussion except the reiteration of your dogma over and over. I won't waste any more time discussing this with you, since it's not a productive dialogue. Take care.

 

[PhilS]

Quote:

Originally Posted by terriblepaulz /img/forum/go_quote.gif No offense counselor, but I'm shocked that a member of the bar would make such a statement. Lawyers memorialize everything in writing so as not to rely on people's memories of what was said or done. What's a better case - a cop's testimony that the driver was drunk, or a videotape showing the driver falling down. A case that rests solely on witness testimony should not (outside of traffic court) survive the cross examination of any competent attorney. I could go on. There is a mountain of research on the unreliability of eyewitness testimony, and there has been a load of Frye and Daubert related litigation on the topic. (See, e.g., http://www.fclr.org/docs/2006fedctslrev3.pdf). And yes I know people are convicted on the basis of eyewitness testimony everyday. I procured a few hundred such convictions myself (I only have about half of your thirty years before the bar). That's a bug, not a feature.

I don't disagree with most of what you said (except for the ridiculous hyperbole in the first sentence and the part about a case based on eye witness testimony not surviving cross), but it is not directly responsive to what I said, which directly responded to your point that evidence that is based on perception or observation is "bad evidence" in a court of law. It's not.

Quote:

Originally Posted by terriblepaulz /img/forum/go_quote.gif There's a limit to the value of an open mind in certain contexts. In court, you call it "relevancy". In the context of comparing and measuring the performance of audio products, the perceptions of individual listeners is simply bad evidence.

That's a matter of opinion. Another opinion is that DBT's are "bad evidence" because they do not mirror the conditions under which people normally listen to music, and they involve test parameters designed to obscure differences that can be perceived or can be perceived over time.

Again, it's fascinating to the extent to which the "scientific viewpoint" will only consider one side of the argument. It seems very "unscientific" to me. And as a lawyer, who presumably has dealt with experts, you know that they often are just plain, flat wrong, or as unreliable as the eyewitness testimony upon which you so readily cast aspersions.

 

[jonathanjong]

Erm, I still don't understand the objections. Some people seem to think that it's really difficult to set up a "proper" test. Environment, moods, etc. need to be controlled for. OK, I get that, but two points need to be made:

1. With a sufficiently large sample and random allocation, group differences average out. You'll have the about same number of golden-eared people, the same number of depressed people, same number of people tested during each time of day etc. Sure, matching for every conceivable confound is better, but unnecessary. Very, very few scientists do that. If you reject a DBT on the grounds that some variables weren't matched, you have to reject lots and lots of scientific research.

2. If the differences between two systems are really THAT difficult to discern...then who cares if there "really is" a difference? I mean, then there'd be "practically" no difference, even if there is technically a difference...one that can only be discerned in a dark, silent room, on a cool day, by a person in a perfectly calm mood wearing a felt hat. I care about the truth of the matter as much as anyone else, but our primary concern is whether or not it's worth it to pay for cables etc. isn't it? It's a practical consideration.

 

[PhilS]

Quote:

Originally Posted by jonathanjong /img/forum/go_quote.gif Erm, I still don't understand the objections. Some people seem to think that it's really difficult to set up a "proper" test.

There was a thread started about a year or so ago on Head-Fi about what a proper test would look like. It went on for pages and pages, before ultimately everybody gave up. I'm just pointing this out to convey the possibility that it might not be as easy as you think. Anyway, I don't want to divert this thread with arguments about testing methodology, but if you do a search, you can probably find the other thread. Some of it is interesting reading; some of it is the usual krap hurled back and forth by both sides.

 

[bobsmith]

Quote:

Originally Posted by jonathanjong /img/forum/go_quote.gif 2. If the differences between two systems are really THAT difficult to discern...then who cares if there "really is" a difference? I mean, then there'd be "practically" no difference, even if there is technically a difference...one that can only be discerned in a dark, silent room, on a cool day, by a person in a perfectly calm mood wearing a felt hat. I care about the truth of the matter as much as anyone else, but our primary concern is whether or not it's worth it to pay for cables etc. isn't it? It's a practical consideration.

Just a personal example unrelated to audio. I bought a large LCD monitor a few years back (when they were still expensive), after comparing it side by side with others and deciding it looked the best. After a full day or so of use I started to notice what I _thought_ was a crosshatch pattern that I could notice only in my peripheral vision, and only when my eyes actually moved. I didn't worry about it much though. Over the course of the next month or so, I started to grow increasingly irritated with it (and started to wonder if it was just my mind playing tricks on me). I also found I was getting headaches. I also found the same issue on another identical monitor of the same brand, but NOT on another of a different brand. After some research, I had a theory about what was causing it and decided to test it by taking an extremely high shutter speed photo of the screen. Sure enough, the pattern showed up as clear as day (I still have the photo if you want proof). I changed to a different monitor forthwith, taking a financial hit in the process, and have not had the same issue since.

The point of the story is that if I had DBTed the monitor against another, I never would have noticed that difference. It took me almost a month of constant use to become fully sensitized to the issue, but once I did notice it, it became a 100% dealbreaker. Our senses often take a long time to spot even obvious differences, as anyone who has played Photo Hunt at a bar well knows. When those differences are subtle, it can take an extremely long time, but that does not mean the differences are unimportant.

 

[ILikeMusic]

Quote:

Originally Posted by bobsmith There is no question it is technically "possible" to run a DBT that permits days or weeks for listening. In fact, it would be very easy to theoretically design a test to do this. Its just that no one (yet) has had the time, money or inclination to actually run such a test.

Agreed, and no offense meant. But what you are saying is fundamentally different than many of the others in these types of threads in that you seem to agree that the fundamental philosophy of DBT is sound, but that a long-term (to the extent that it's credible that long-term listening is somehow more acute than short-term) listening test hasn't been run yet, and I have no argument there. But you do realize that even if such a test were to be run the hard-core subjectivists would doubt it and claim some other flaw, and the bar would just be set further and further out so that testing becomes impossible no matter what you do. That's the kind of thing I was referring to...

Quote:

Originally Posted by Phils That's all that matters to you, because you have decided in advance that's all you will accept. You're reasoning is circular, and you are not really contributing anything to this discussion except the reiteration of your dogma over and over.

If you're concerned about productive comments perhaps you might spend less time characterizing my position as 'dogmatic' and more time factually refuting it?

.

 

[jonathanjong]

Interesting anecdote, bobsmith. I agree with ILikeMusic that all this means is that we have to build in such considerations when we set up a DBT. Make it a 6-month-long DBT, even. If the differences take more than 6 months to emerge, they might not be worth talking about anyway.

@ILikeMusic: I really cannot be bothered reading through the debate. Were the methodological criticisms damning, or just mere squibbles? Like I said, if sonic differences are not discernible in a simple DBT (one that lasts for months, to placate bobsmit, say), then they're too small to be bothered with.