[SoundAndMotion]

However, for me the null hypothesis for speakers is that they sound different. Congruence does not equal logic. <Steve999 will have to ponder why he thinks the null hypothesis for speakers is that they sound different. It is an intuition with which he is struggling at the moment.><A half hour later Steve999 continues to puzzle over this,>

LOL! Well technically you don't define the null hypothesis however you want. But we can easily avoid arguing definitions by avoiding specific terms. The specific terms are not really relevant to the discussion.

Let's say your default hypothesis is that speakers sound different, but I'm skeptical. You bring me 2 active speakers that look identical, but you've changed the DSP inside, quite a bit, and you know they measure dramatically different and you've ABX-ed them with clearly significant results. But I don't know that. They look the same and I'm skeptical.

My buddy and I use the same protocol I mentioned above and we conclude: both are "transparent". We reject your default hypothesis. Is this useful? Again, no. And the definition doesn't matter.

 

[Steve999]

LOL! Well technically you don't define the null hypothesis however you want. But we can easily avoid arguing definitions by avoiding specific terms. The specific terms are not really relevant to the discussion.

Let's say your default hypothesis is that speakers sound different, but I'm skeptical. You bring me 2 active speakers that look identical, but you've changed the DSP inside, quite a bit, and you know they measure dramatically different and you've ABX-ed them with clearly significant results. But I don't know that. They look the same and I'm skeptical.

My buddy and I use the same protocol I mentioned above and we conclude: both are "transparent". We reject your default hypothesis. Is this useful? Again, no. And the definition doesn't matter.

I want to know why the fricking null hypothesis cannot be that speakers sound different, even as I agree that for DACs it is that they sound the same, I don’t want to know about any dang default hypothesis. I don need to known whose default it is. Blame him, blame her, I really don’t care whose default it was. What I want to know is what’s the conceptual framework behind arriving at the fricking null hypothesis. Because I am not understanding that.

 

[SoundAndMotion]

I want to know why the fricking null hypothesis cannot be that speakers sound different, even as I agree that for DACs it is that they sound the same, I don’t want to know about any dang default hypothesis. I don need to known whose default it is. Blame him, blame her, I really don’t care whose default it was. What I want to know is what’s the conceptual framework behind arriving at the fricking null hypothesis. Because I am not understanding that.

It's just terminology. Like I said, just avoid the word and say whatever you want.
Or if you really want to understand it better, click here (link).

 

[Steve999]

It's just terminology. Like I said, just avoid the word and say whatever you want.

You’re just falling back on the null hypothesis that null hypotheses are just terminology. I think it’s more than just terminology. To prove it I posit the following. That it is he who seeks to disprove the null hypothesis who is expected to lay out his alternate hypothesis and his factual evidence. The null hypotheses guy apparently just gets to take medicine and freak out, if I am hearing you right. That’s a big difference. Do hypothesizers, e.g., reserchers, argue over whose is the null hypothesis? Do they refine it? Is the null hypothesis that there will be no statistical correlation between a population of people or things and certain characteristics? Because I want understand the null hypothesis. I’m afraid Webster’s doesn’t cut it here!

 

[SoundAndMotion]

You’re just falling back on the null hypothesis that null hypotheses are just terminology. I think it’s more than just terminology. To prove it I posit the following. That it is he who seeks to disprove the null hypothesis is he who is expected to lay out his alternate hypothesis and his factual evidence. The null hypotheses guy apparently just gets to take medicine and freak out, if I am hearing you right. That’s a big difference. Do hypothesizers, e.g., reserchers, argue over whose is the null hypothesis? Do they refine it? Is the null hypothesis that there will be no statistical correlation between a population of people or things and certain characteristics? Because I want understand the null hypothesis. I’m afraid Webster’s doesn’t cut it here!

I just put this link above, but here it is again.
There is no terminology police or statististics police. If you want to post in an Internet forum, you can say whatever you want! If you want to publish an article on neuroscience or perception, no one is going to get caught up by the definition of null hypothesis. If a reviewer or editor doesn't like your usage, you should just edit it out and rewrite another way.

 

[Steve999]

I just put this link above, but here it is again.
There is no terminology police or statististics police. If you want to post in an Internet forum, you can say whatever you want! If you want to publish an article on neuroscience or perception, no one is going to get caught up by the definition of null hypothesis. If a reviewer or editor doesn't like your usage, you should just edit it out and rewrite another way.

Thanks, I read it, but did not get out of it what you did. Probably, and I am not joking, because you know a lot more than I do about this.

But i did not intend to go nowhere with this. Here is my concern: if we go telling people they are the ones who have to show DACs sound different, but the eggheads are going to essentially decimate the validity of the null hypothesis and the resulting implications of who needs to prove what, then what’s to stop a back and forth argument about, say, whether x has to show the DACs do make a difference, or y has to show that DACs don’t make a difference. Obviously, this leaves y with a challenge of epic metaphysical proportions. If we just kick the terminology can down the road, then we have people asked to prove nothing could ever happen, rather than coming up with useful tools that have predictive value.

 

[SoundAndMotion]

Thanks, I read it, but did not get out of it what you did. Probably, and I am not joking, because you know a lot more than I do about this.

But i did not intend to go nowhere with this. Here is my concern: if we go telling people they are the ones who have to show DACs sound different, but the eggheads are going to essentially decimate the validity of the null hypothesis and the resulting implications of who needs to prove what, then what’s to stop a back and forth argument about, say, whether x has to show the DACs do make a difference, or y has to show that DACs don’t make a difference. Obviously, this leaves y with a challenge of epic metaphysical proportions. If we just kick the terminology can down the road, then we have people asked to prove nothing could ever happen, rather than coming up with useful tools that have predictive value.

This is an internet forum, not a high-impact journal. So, I would offer this:
1. It doesn't matter who knows more. You are not convinced.
2. Avoiding unreasonable demands of validation and/or ignoring reasonable demands of validation, which leads to unending back-and-forth arguments, is pretty ubiquitous in forums. You won't impact its occurrence at all just by using "null hypothesis" correctly or incorrectly.
3. What I find relevant both in forums and journals is convincing someone of your conclusions, as I spelled out a bit to @castleofargh here:

... "burden of proof", may be relevant for debating how to debate, but doesn't have much relevance wrt discussing listening tests in a forum. First, "proof" is too strong a word. You may show probabilities, and they may be quite convincing to many people, but proving anything in studies of perception is generally not possible. Second, the "burden" anyone feels is self-imposed and relates to the goal of convincing a person or a group.

If I state my goal is finding an audible difference between 2 DACs, and I state that I listened sighted and there is a huge difference, and I state nothing else, I have no burden of anything. I may only want to convince myself and no one else!

But if I say I used foobar2k with the abx plugin and I got 30/30 correct and here's the output with authentication hash, you may tell me you are not convinced. If my goal was to convince you, my burden is to figure out what you need for convincing and produce it.

It is worth noting that another member of this forum told me something years ago that I didn't believe at first: that no matter what you do and how you do it, you will never convince 100% of the people on a forum. I have since seen his wisdom and agree he is correct.

Also, you are right that if you fail to reject the null hypothesis, you really don't have any new information. But many, many people believe that if you do fail to reject, that you have supported the null hypothesis: I couldn't show I heard a difference, so there is no difference; absence of proof is proof of absence. As you point out, this is not true.

4. If I have not convinced you about the importance, or lack thereof, of the definition of "null hypothesis", I'm at a dead end.

 

[Phronesis]

I look at the null hypothesis as being a sort of practical matter which depends on the context. Typically, the null hypothesis reflects believing less rather than believing more, but it may also reflect the prevailing belief based on theory, research, experience, etc.

With drug trials, the burden is on the drug maker to demonstrate efficacy, and a lot of drugs that go to trial may not work, so the null is that a drug doesn’t work.

With DACs, we have justification based on theory and measurements to expect them to sound the same, and we can dismiss anecdotal reports of their sounding different as being due to misperception, so IMO it's reasonable for the null to be that they sound the same.

With speakers, the physical differences are very obvious, and we know enough about acoustics to expect those physical differences to result in sound differences, so IMO it’s reasonable for the null to be that speakers sound different (though the differences need not be large).

 

[Steve999]

I look at the null hypothesis as being a sort of practical matter which depends on the context. Typically, the null hypothesis reflects believing less rather than believing more, but it may also reflect the prevailing belief based on theory, research, experience, etc.

With drug trials, the burden is on the drug maker to demonstrate efficacy, and a lot of drugs that go to trial may not work, so the null is that a drug doesn’t work.

With DACs, we have justification based on theory and measurements to expect them to sound the same, and we can dismiss anecdotal reports of their sounding different as being due to misperception, so IMO it's reasonable for the null to be that they sound the same.

With speakers, the physical differences are very obvious, and we know enough about acoustics to expect those physical differences to result in sound differences, so IMO it’s reasonable for the null to be that speakers sound different (though the differences need not be large).

Thanks. Thanks really interesting. That's in accord with my intuition but with much better reasoning. I am somewhat surprised that how to choose the null hypothesis not more cut-and-dried and is not in some senses more "objective" or formulaic. This seems to leave room for argument in areas where I was convinced there was none. I suppose there are situations where you could flip a coin as to which of two views should be the null hypothesis--and this affects profoundly the conclusions reached and the types of testing done.

Even interesting more is that I got four substantially different answers from four very intelligent people (don't get too smug, I might change my mind tomorrow).

Well, I imagine that most of us would agree that the hypothesis that all properly designed DACs used as intended under the conditions intended would sound the same. And I imagine there are some readers who would find this choice of null hypothesis undesirable. And based on what I have read, it is more a matter of judgment and purpose rather than "hey buddy, you made the claim, the burden of proof is on you." I would also perceive what @bigshot is doing is testing what most of us would consider to be the null hypothesis. It's fair game in a sense to inquire of his methods, but we should also keep in mind that in a sense he is going above and beyond in doing so. If someone wants to question his methods in testing the null hypothesis rather than an alternative hypothesis, it would seem to me to be fair game to suggest that someone criticizing him either do a better job themselves of testing the null hypothesis, or come up with an alternative hypothesis and assemble the tests and data necessary in order to do so. Otherwise it comes down to @bigshot or others like him did at least something (since we have ruled out in the course of this conversation the idea that he is not telling the truth) and that's more than anyone else in this conversation did (including yours truly).

So here is what I propose is a good null hypothesis of this thread, by the consensus of @castleofargh, @SoundAndMotion, @gregorio, and @Phronesis: all properly designed DACs used as intended under the conditions intended would sound the same.

Does anyone disagree that this should be the null hypothesis? Remember, technically speaking, as I understand things now, it is a guess--having it be the null hypothesis does not make it a truism or an undebatable point. All is not lost for anyone who has another view. So. . .

Does anyone have a proposed alternative hypothesis that we can test against this null hypothesis in some kind of decent way?

 

[Phronesis]

Thanks. Thanks really interesting. That's in accord with my intuition but with much better reasoning. I am somewhat surprised that how to choose the null hypothesis not more cut-and-dried and is not in some senses more "objective" or formulaic. This seems to leave room for argument in areas where I was convinced there was none. I suppose there are situations where you could flip a coin as to which of two views should be the null hypothesis--and this affects profoundly the conclusions reached and the types of testing done.

Even interesting more is that I got four substantially different answers from four very intelligent people (don't get too smug, I might change my mind tomorrow).

Well, I imagine that most of us would agree that the hypothesis that all properly designed DACs used as intended under the conditions intended would sound the same. And I imagine there are some readers who would find this choice of null hypothesis undesirable. And based on what I have read, it is more a matter of judgment and purpose rather than "hey buddy, you made the claim, the burden of proof is on you." I would also perceive what @bigshot is doing is testing what most of us would consider to be the null hypothesis. It's fair game in a sense to inquire of his methods, but we should also keep in mind that in a sense he is going above and beyond in doing so. If someone wants to question his methods in testing the null hypothesis rather than an alternative hypothesis, it would seem to me to be fair game to suggest that someone criticizing him either do a better job themselves of testing the null hypothesis, or come up with an alternative hypothesis and assemble the tests and data necessary in order to do so. Otherwise it comes down to @bigshot or others like him did at least something (since we have ruled out in the course of this conversation the idea that he is not telling the truth) and that's more than anyone else in this conversation did (including yours truly).

So here is what I propose is a good null hypothesis of this thread, by the consensus of @castleofargh, @SoundAndMotion, @gregorio, and @Phronesis: all properly designed DACs used as intended under the conditions intended would sound the same.

Does anyone disagree that this should be the null hypothesis? Remember, technically speaking, as I understand things now, it is a guess--having it be the null hypothesis does not make it a truism or an undebatable point. All is not lost for anyone who has another view. So. . .

Does anyone have a proposed alternative hypothesis that we can test against this null hypothesis in some kind of decent way?

I can see how the concept of a null hypothesis is needed, or at least useful, as a sort of formality for the purpose of doing research and statistical analysis, but IMO the question you raise about how to select a null hypothesis illustrates that the concept can also be counterproductive in a way. In general, if our goal is to pursue truth by systematically gathering data and carefully interpreting it, we're not obligated to pick a null hypothesis and a single alternative hypothesis - we can work with multiple hypotheses and decide which one of them (if any) best 'fits' the data based on various kinds of statistical analyses and our judgment, and there isn't necessarily one 'correct' way to go about this. The judgment aspect always leaves room for uncertainty and disagreement, but for me that's part of what makes reality and science interesting.

To illustrate further, if we say that our null is that DACs sound the same, and we repeatedly test and find in the tests that they don't seem to sound different, that doesn't 'prove' that they sound the same. It just shows that we've been unable to demonstrate that they sound different, and that could be due to things like the experiment being poorly designed or conducted, listeners who lack perceptual acuity, listeners who are lazy or biased to not hear differences, test conditions being very different from normal listening conditions, etc. OTOH, if you do those tests and it does appear that some DACs sound different for some listeners, you've shown something significant. So the situation is good for finding differences, but it's not so good for conclusively ruling out differences. I think that people sometimes don't make this distinction.

Now if we invert this and say that our null is that DACs sound different, do our tests, and find that the alternate hypothesis is supported (they don't sound different - sound the same), does that really show that they sound the same? I would say not really, because it's easier to 'fail' the test and not perceive differences due to bias, etc. than it is to consistently perceive differences. So we wind up having sort of confused ourselves because of the null we chose, compounded with potentially misinterpreting the test results.

IMO, the bottom line is that we have to have a good scientific and statistical understanding of what we're doing when we design, conduct, and interpret these tests, otherwise we're quite prone to reaching incorrect conclusions. The scientific research process can't be reduced to a cookbook procedural or mathematical process where a person can reach accurate conclusions without a good understanding of the relevant science. My impression is that too many people in the audio world have done testing in an amateurish way and reached unreliable conclusions as a result. We need to be skeptical not just about claims based on anecdotal reports from uncontrolled listening, but also claims based on tests which haven't been conducted and interpreted to scientific standards.

 

[bigshot]

There's no reason to apply complicated scientific principles to know that this guy is full of it when he says the human brain can detect sound down to -300dB. Just look up what 300dB represents. I'm sure he's very knowledgeable and can make a nice DAC that measures a little better than the competition. But there is no reason to pay that much money for abstract numbers on a page that don't relate to sound you can hear. Trying to make customers think they might be able to hear the unbearable is the dividing line between doing honest business and snake oil.

 

[Laura80]

There's no reason to apply complicated scientific principles to know that this guy is full of it when he says the human brain can detect sound down to -300dB. Just look up what 300dB represents. I'm sure he's very knowledgeable and can make a nice DAC that measures a little better than the competition. But there is no reason to pay that much money for abstract numbers on a page that don't relate to sound you can hear. Trying to make customers think they might be able to hear the unbearable is the dividing line between doing honest business and snake oil.

From what I’ve read on the two popular Chord threads on the high end audio section, the general consensus appears to be that Chord products (tt 2 & Dave) are value for money due to their respective additional features to the DAC, as also a headphone amplifier and preamplifier. Owners also seem to appreciate build quality and trust in Chord as a brand.

So I’m not sure owners committed solely on the DAC feature alone.

 

[Steve999]

From what I’ve read on the two popular Chord threads on the high end audio section, the general consensus appears to be that Chord products (tt 2 & Dave) are value for money due to their respective additional features to the DAC, as also a headphone amplifier and preamplifier. Owners also seem to appreciate build quality and trust in Chord as a brand.

So I’m not sure owners committed solely on the DAC feature alone.

Give me an XM5 any day of the week. Bass boost switch, treble boost switch, variable crossfeed, made of sturdy metal, tiny, impedance switch so you can match low or high impedance headphones, +10 db gain if you need it, good measurements if you take them at their word, it's a headphone amp and a DAC, portable, rechargeable, powerful, and honest guys at the other end of the phone at least when I talked to them some number of years ago. An XM4 sounds the same as an XM5 sounds the same as an XM6, but you'll get better measurements and more features as you move up. That's what they told me, straight-up. What's not to like?

http://www.practicaldevices.com/

I am just curious, do these Chord things have all of that? I honestly don't know.

 

[bigshot]

Do you happen to know what the features people are choosing the Chord for are? I assume those features aren’t available with other DACs.

I have a funny story about the Oppo players. On another forum a guy was making a big deal about how the 205 is much better than the 203 because of the advanced DAC chip that adds a grand to the price tag. Then he mentioned his player was connected via HDMI and it sounded better than any of his other players.

When you patch with HDMI, it bypasses the Oppo’s internal DAC and converts in your AVR’s DAC. The whole time he owned it, he had never even heard the DAC that he claimed sounded so much better! About ten posts from three different people patiently explaining to him just got him all angry. He was convinced people were insulting him. He finally disappeared from the thread without admitting his mistake, so I assume he realized it and slunk off.

That’s how people work. They get an idea in their head about a conclusion without going through the steps to reach a conclusion. I see that here every day.

 

[Laura80]

Do you happen to know what the features people are choosing the Chord for are? I assume those features aren’t available with other DACs.

I have a funny story about the Oppo players. On another forum a guy was making a big deal about how the 205 is much better than the 203 because of the advanced DAC chip that adds a grand to the price tag. Then he mentioned his player was connected via HDMI and it sounded better than any of his other players.

When you patch with HDMI, it bypasses the Oppo’s internal DAC and converts in your AVR’s DAC. The whole time he owned it, he had never even heard the DAC that he claimed sounded so much better! About ten posts from three different people patiently explaining to him just got him all angry. He was convinced people were insulting him. He finally disappeared from the thread without admitting his mistake, so I assume he realized it and slunk off.

That’s how people work. They get an idea in their head about a conclusion without going through the steps to reach a conclusion. I see that here every day.

I’ve heard it said that people primarily buy a headphone on how it looks cosmetically, which initially sounded counterintuitive to me, as I would have expected people would choose a headphone primarily on its sound reproduction appeal. Perhaps Chords outer designs with multi coloured lights etc are a major attraction.