[StanD]

When was this demonstrated under controlled listening conditions?

se

Perhaps that was an anecdotal post. It happens.

 

[KeithEmo]

Maybe. But ABX testing still eliminates other biases, which is why it's important. So that doesn't discount the value of DBT over sighted listening at all. Plus, you obviously wouldn't have that problem, so seems a moot point.

So why not do some testing instead of all this speculation that the ODAC will sound different from chip X or that such and such filters produce an audible response?

 
I agree entirely - that ABX testing is very effective at what it does.
 
I guess I'm confused.... I thought this thread was about "audiophile claims and myths" - and perhaps about discussing them. If it's really only about "publishing the results of tests" then we should all just go join the AES. The way real science works (at least in other places) is that people come up with theories and ideas, discuss them, decide which ones seem worth testing, design tests, validate those tests to decide if they'll really test what wants testing, do the tests, and then analyze and discuss the results...
 
Back when I went to school, we were taught that matter was made up of protons, neutrons, and electrons. As I recall, a few people even ran various tests to "prove" this theory. And, in fact, a few trivial devices - like atomic bombs and nuclear reactors - were designed based on this idea. Now, as it turns out, that was wrong after all. (Matter is either made up of little bundles of energy, or of little string things vibrating in eleven dimensions, or perhaps those are wrong too.) It seems like, in real life, it's not all that easy to figure out where the line between myth, theory, and reality falls (and it has this nasty habit of moving when we look into it a bit).
 
However, there seems to be a serious misconception about what ABX testing actually gives you in most situations. In actual point of fact, all it gives you is a result showing that a statistically significant number of people could tell Product X from Product Y, and which one they "liked better", under very specific conditions. This makes it a great way for an individual to decide what product to buy, and a great way for the marketing department to figure out how many of their potential customers can hear a difference between product X and Product Y and, if they do, which one they like better. However, as a scientific tool, it's not very useful because the information it yields is not properly controlled. (Let's say it's a good way to apply scientific principles to the study of product preferences.)
 
In a "proper scientific experiment", you first theorize what results you hope to find. You then devise a test that meets certain basic requirements:
1) it allows you to vary ONLY THE THING YOU WANT TO TEST while keeping everything else constant
2) it offers some way to quantify the results
3) it does both of these things over the range of interest
 
To put this in the context of audio equipment....
1) if you're testing the threshold of audibility of THD, then you need to be able to vary THD WHILE EVERYTHING ELSE REMAINS UNCHANGED
2) since we're testing audibility, some sort of test comparing the audibility of different amounts of THD would be appropriate (an ABX test between different levels of THD would be appropriate)
3) we need to verify that our test equipment allows the results to be observed
 
To put this in the context of where I came into this discussion.... if we want to actually determine the audibility of different digital filters in DACs....
1) we need a way to change digital filters in a DAC while everything else remains equal (this can be tricky because different filters often have different frequency response)
2) this is easy (we can ask a sample of people whether they can hear a difference; although, of course, we can then only declare the results valid for our particular sample group)
3) this may be the tricky part (we need audio samples with less ringing that that which we are hoping to test, and we need speakers that can deliver those samples accurately)
 
Obviously, if the speaker we use has 10 mS of ringing, then that will quite probably prevent us from hearing the difference between 1 mS of ringing and 2 mS of ringing due to different filters. This will render the test results invalid or meaningless. So, in order to actually test this, we need to find a speaker or other transducer that has little enough ringing to avoid obscuring the results - and we need to document that we've tested that to be the case and provide the specifics of what speaker we used and how we tested it. (If we find out that the speaker we used displays the same amount of ringing with all our sample filters, because it has worse ringing on its own, then we can't consider the test to be valid, right? So, at a very minimum, we need measurements showing that our speaker was faithfully delivering the test signal.... or we may find that we can't even do a valid test because there is no speaker on the market that meets our requirements. If that happens, we may have to design a special transducer to use for our experiment, or we may have to conclude that we can't test it. This happens in "real science" like physics all the time.)  
 
Now, an IMPORTANT distinction: If the marketing department is doing this study, then it is quite sufficient to demonstrate that "using a reasonable sample of people chosen at random, using typical home audio equipment, a significant majority was or was not able to hear a difference". However, if we want to state the result as a scientific fact, then we must either ensure that the test equipment is indeed able to perform the test effectively, or, failing that, that we have made the best possible attempt to ensure that it has (and then we need to state that limitation).
 
Again, back to my example, if you can find a transducer that actually has less ringing than the filter on a DAC, then it would be ideal. If not, then using the best available transducer (in terms of ringing) will have to do - but you had better state its limitations in your test results.
 
Now that we've reached this point, we will, of course, come back to the claim that what we're talking about is "beyond the limits of human hearing". Personally, I place most of the various claims about "the limits of audibility" of various things in the same category as the other claims we're discussing here.... and I'm not willing to accept that one claim is invalidated simply because it conflicts with another claim. (And, yes, there have been AES papers "proving" that nothing above 20 kHz is audible; but there has also been at least one other AES paper "proving" that there were audible differences between a 20 kHz band-limited signal recorded at a 44k sample rate and the same signal recorded at a 96k sample rate. Therefore, at least to me, that all falls into "reasonable but unproven claims".)

 

[Steve Eddy]

However, there seems to be a serious misconception about what ABX testing actually gives you in most situations. In actual point of fact, all it gives you is a result showing that a statistically significant number of people could tell Product X from Product Y, and which one they "liked better", under very specific conditions. This makes it a great way for an individual to decide what product to buy, and a great way for the marketing department to figure out how many of their potential customers can hear a difference between product X and Product Y and, if they do, which one they like better. However, as a scientific tool, it's not very useful because the information it yields is not properly controlled. (Let's say it's a good way to apply scientific principles to the study of product preferences.)

You really don't have a clue what you're talking about here and it is abundantly clear that you have no real understanding of ABX testing. So until you do, I would suggest you just make no further mention of it (hint: it has nothing to do with preference testing).

*sigh*

I shouldn't have to feel embarrassed that I'm involved in this industry. But I do.

se

 

[KeithEmo]

Maybe. But ABX testing still eliminates other biases, which is why it's important. So that doesn't discount the value of DBT over sighted listening at all. Plus, you obviously wouldn't have that problem, so seems a moot point.

So why not do some testing instead of all this speculation that the ODAC will sound different from chip X or that such and such filters produce an audible response?

 
Hmmmm... that sounds like what I said right at the beginning...
 
... That the measurements show that there is an actual difference between the response of various digital filters
... And that many people (including myself and the companies who make the DAC chips) claim that the differences are audible
... And that it therefore makes sense to do some serious tests (before dismissing all such claims because they disagree with other claims)
 
I also find it interesting how so many people (especially here) are so selective in their willingness to believe....
 
For example, ESS, the company who makes the Sabre DAC used in the 0DAC states all sorts of specifications.
They also state (or at least they did state in their original published marketing literature) that their DACs sound different than everybody else's.
I find it very... entertaining... how many people here seem eager to use one of their claims (the specs) as proof that their other claim (that their DACs sound different) is a lie.
 
You might find this white paper from ESS interesting....
 
It not only describes their design philosophy, but mentions in several places how they adjusted design characteristics because they believed doing so achieved "better sound quality" - even though these characteristics produce results that are far below "the limits of audibility" that some people on this thread seem quite convinced of.  
 
http://www.esstech.com/PDF/sabrewp.pdf

 

[cel4145]

I agree entirely - that ABX testing is very effective at what it does.

I guess I'm confused.... I thought this thread was about "audiophile claims and myths" - and perhaps about discussing them. If it's really only about "publishing the results of tests" then we should all just go join the AES. . . .

It's quite simple. This is not about other people. You have repeatedly claimed that some DACs sound different from others to YOU when audio science would suggest that might be not true. For validating your OWN opinion, you could ABX test.

 

[Steve Eddy]

It's quite simple. This is not about other people. You have repeatedly claimed that some DACs sound different from others to YOU when audio science would suggest that might be not true. For validating your OWN opinion, you could ABX test.

When someone says there is a serious misconception about ABX testing and then goes on to demonstrate that the serious misconception is their own, what more can you expect?

Too bad he doesn't spend as much time trying to actually learn something as he puts into these lengthy posts of his. But I sense that he's only here to preach and has no particular interest in learning anything.

se

 

[cel4145]

When someone says there is a serious misconception about ABX testing and then goes on to demonstrate that the serious misconception is their own, what more can you expect?

Too bad he doesn't spend as much time trying to actually learn something as he puts into these lengthy posts of his. But I sense that he's only here to preach and has no particular interest in learning anything.

se

Exactly.

If had access to an ABX comparator and the means to do an accurate volume match, I'd be testing all kinds of stuff that I have because I'd like to learn.

I'm willing to hear discussions that criticize DBT when it comes to comparing things like speakers which DO sound different, because I think there is more flaw to the process than some objectivists are sometimes willing to admit. But comparing to see if something is different or not with ABX? It works.

 

[StanD]

@KeithEmo I'm one for a good analogy but I don't think that trying to equate the validity of simple audio engineering and far out string theory is in the same ballpark. Electrical Engineering is applied/practical physics and string theory is just that, theory.

 

[Steve Eddy]

If had access to an ABX comparator and the means to do an accurate volume match, I'd be testing all kinds of stuff that I have because I'd like to learn.

Great!

I'm willing to hear discussions that criticize DBT when it comes to comparing things like speakers which DO sound different, because I think there is more flaw to the process than some objectivists are sometimes willing to admit. But comparing to see if something is different or not with ABX? It works.

What flaws do you mean?

But yeah, with things like loudspeakers where there's no question with regard to audible differences, ABX is pointless because ABX is all about identification. But blind testing in general still has value. Floyd Toole (now retired from Harman) for example, used blind preference testing to try and correlate preference to measured performance. Such tests needed to be blind so that preference was based on sound alone, instead of other influences like looks and price.

se

 

[Steve Eddy]

@KeithEmo
 I'm one for a good analogy but I don't think that trying to equate the validity of simple audio engineering and far out string theory is in the same ballpark. Electrical Engineering is applied/practical physics and string theory is just that, theory.

Methinks those strings are actually straws.

se

 

[StanD]

  @KeithEmo I'm one for a good analogy but I don't think that trying to equate the validity of simple audio engineering and far out string theory is in the same ballpark. Electrical Engineering is applied/practical physics and string theory is just that, theory.

 
 

Methinks those strings are actually straws.

se

So now he's a straw man? 

 

[Steve Eddy]

So now he's a straw man? 

No no, think "grasping."

se

 

[StanD]

No no, think "grasping."

se

That happens. Too Often

 

[KeithEmo]

   
Another (in my opinion not unlikely) possibility is that the "clear" differences you heard are mainly the result of expectation bias and simple factors by level differences.
 
 
That is why the test should ideally be performed by people who do expect to hear differences. You could try some ABX tests yourself, although I guess you most probably will not.
 
 
Actually, there is a new revision of the ODAC that no longer uses an ESS DAC.

 
You could be right, although I rather doubt it (based on the fact that I also find the differences between different filter choices and DAC chips to be clearly audible - to me and to others who were present at the same time - with other products as well). However, since I'm only human, and so are they, I'm certainly willing to consider that I might be wrong.
 
(I'll also drop at this point that Emotiva - who I work for - makes several DACs. Both of our current models - the DC-1 and XDA-2 - only have a single digital filter choice - which is quite accurate. However, we will be introducing a few new models very shortly which do have switchable filters. They offer three different filters, which do indeed sound slightly different, to me and to everyone else who's auditioned them, although, since they also have slightly different frequency responses, I won't make any specific claim as to what the audible differences consist of. Personally I don't find any particular choice especially better.... and, since we don't charge extra for the feature, we encourage anybody who doesn't notice or care about the difference to NOT push the button.)
 
Honestly, since I already own quite a few DACs, many of which I find to sound better than the 0DAC, and several of which have other features which I also consider important (like full sized balanced outputs instead of the 1/8" jack on the 0DAC, and multiple digital inputs, which I do use), I really don't have much reason to set up a full test at this point. However, I wasn't really trying to "prove" that the differences exist; I was simply trying to point out that the logic of the person who insists that such differences couldn't possibly exist is flawed, and that it is indeed worthy of being tested. (There seems to be at least one person who insists that there's no possible point in doing any tests at all because it's just "fact" that any differences, if they exist at all, are "below the threshold of audibility").
 
I personally think that there is a major disconnect between "important differences" and "just differences". I'll also say up front that I don't consider most of the differences that I'm hearing to be important. My Wyred4Sound DAC2 has six filter choices, several of which are very flat - and are claimed to differ only in impulse response to transients. You can switch between them using a button on the remote control, which makes doing so very easy. And, when you switch, the sound does change. To be totally honest, I wouldn't bet that I could tell you which filter was selected at a given moment - and certainly not if I were to leave the room and come back - yet that doesn't invalidate the fact that they are different - and this is the weakness of ABX testing as a scientific tool (outside of as a way to select an audio product).
 
The closest analogy I can come up with is how we perceive color. If you consider two square tiles painted similar colors, experiments have shown that we humans can distinguish amazingly small differences in color when you place two tiles side by side. However, if shown each tile individually, or one after the other, we are much less able to distinguish differences. So, if I were to take several tiles of very similar colors, and hold them up one after the other - in the same format as an ABX test - most humans would probably not be able to distinguish them. However, if I were to lay those same tiles on a table, overlapping each other, most of us would be able to see where two dissimilar colors were next to each other. Likewise, we could probably say quite accurately which was brighter or darker if they were displayed next to each other, but would fail miserably if they were displayed one at a time.
 
Clearly, how important this is depends on interpretation. If you run out of paint in the middle of a wall, then you'd better match it exactly for the remaining half. However, if you're painting two different rooms to match, precisely matching the color is a lot less critical, and there might be times where putting a contrasting strip of molding between two slightly dissimilar walls would be enough to prevent us from noticing the difference.
 
I personally suspect that differences much less that what we normally accept as "the limits of audibility" are audible - but only under special circumstances. For example, most of us would totally unable to hear 0.1% THD if it were added to a typical music signal. However, if I were to play a perfect clean 440 Hz sine wave, and then add to it 0.1% as much of a 1 kHz sine wave, I suspect most of us would find THAT to be easily audible. Therefore, simplistic claims like "xx% THD is inaudible" are just that, simplistic. In reality, in some circumstances, 1% THD may be inaudible while, in others, 0.05% may be clearly audible. (If you don't believe that, take a 20 Hz tone, and add 0.05% as much of a 1 kHz tone..... )
 
Whenever we design real scientific tests, we must take into account the test conditions, the accuracy and resolution of the equipment, and the "validity" of the test procedure. In most cases, we end up having to "test the test" before we can test the unit under test.
 
In my color tile example, a "regular ABX test" where each tile was held up in sequence would yield one result (no difference seen), while placing the tiles next to each other would produce a different result (difference reported). Neither result is, in and of itself, wrong; each is "correct" under the conditions in which it was performed. However, if we were to conclude from the first test that "human beings cannot see the differences between the colors of the tiles", and make that as a blanket statement, then that conclusion would indeed be wrong.
 
In this thread, we have someone who, based on the facts that:
1) he cannot hear differences between three specific devices under specific circumstances
2) "established fact" "proves" that certain differences will be inaudible UNDER ANY CONDITIONS
There is no reason to test my claims (or those of anyone else who claims to hear a difference).
I say his logic and "facts" don't rise to the level of certitude to justify his claim.
 
 
As for the new 0DAC, if you're referring to the new PCB version by JDS labs, yes that is indeed interesting.
From the picture, it seems to use the TI/BB PCM5102a DAC chip - which includes a choice of two switchable digital filters
(since they're supposed to be "audibly indistinguishable" I wonder which one they decided to use; I don't see any mention of the testing they used to decide).
If you Google the DataSheet, you will also see all sorts of nice graphs and scope traces of the (quite different) impulse responses of the choices.
 

 

[cel4145]

What flaws do you mean?

But yeah, with things like loudspeakers where there's no question with regard to audible differences, ABX is pointless because ABX is all about identification. But blind testing in general still has value. Floyd Toole (now retired from Harman) for example, used blind preference testing to try and correlate preference to measured performance. Such tests needed to be blind so that preference was based on sound alone, instead of other influences like looks and price.

se

I do agree. DBT does have value in speaker testing. But I think the flaw is that some objectivists over generalize what you learn from DBT. DBT typically tests how speaker A sounds compared to speaker B at volume level X, which is based on some kind of weighted averaging, not that speaker A or speaker B always compare the same at different listening volumes. What I have found is that some headphones I really like at moderate listening volumes but not at loud listening volumes. And vice versa. And my "theory" is (and you are welcome to correct me) that due to equal loudness contours, a headphone with certain types of emphasis can be good at lower volumes, and then I don't need that emphasis at higher volumes and it tends to sound too aggressive. Grados for example. Moderate volumes, I love them. Higher volumes, the treble is too aggressive for me, fatiguing, where as headphone with more demure treble is not and I enjoy them more at higher volumes.

And then 2nd, some music seems to benefit more for headphone A over headphone B, and vice versa. Or if I am listening more for vocals vs. drums where the unique signature of a headphone tends to bring out one or the other more.

I know that some people recognize the limits of what DBT can tell you. But others seem to over attribute a DBT test as proof that speaker/headphone A is always better than speaker/headphone B.