[KeithEmo]

Actually you missed my point entirely about the VW scandal.
My point there was simply that sometimes the results of tests turn out to be wrong - for various reasons (in that particular case it was because of deliberate trickery).
Even, as in that case, when the test results are "certified" by the government, and published as being "reliable".
(I wonder how many people used those published results, of that single test, as "proof" of something.)
However, errors in test results are often simply due to error, or to a badly designed or implemented test procedure.
This is why, in real science, we virtually never rely on the results of a single test as proof of anything.
We publish the details, and invite other folks to run the same test, or to devise different tests, approaching the question from slightly different directions, to confirm our data.
The computers in those VW diesel vehicles were programmed to fudge the data for a very specific test.
A quick run around the track, instead of on a dynamometer, with a full tank of fuel, would have identified the problem with the test results... or, at the least, provided conflicting data.
But, instead, for years, everybody CHOSE to accept the results of one specific test, instead of actually taking the time to confirm it with a different test.
(We know why VW fudged the tests; we must assume that everyone else was either too lazy, or too eager to believe those results, to bother to confirm them.)

So far, as far as I know, NO independent reliable source has done a proper scientific test to show whether different DACs sound different or not.
Several reviewers have done tests that seem to show the opposite - many of which have been linked to on this forum.
(Every review that shows that, in a blind test, one or another DAC was preferred, is clearly inferring that an audible difference did exist... )
Likewise, many vendors have done their own listening tests, but we can safely assume them to be biased (they wouldn't have published results that made their product look bad).
Obviously, we have lots of anecdotal evidence, in both directions.
And we have a lot tangentially related data showing that, under certain other circumstances, many of the differences measured with DACs have been found NOT to be audible.
And BigShot has done some credible double-blind testing, but with only two or three DACs, one or two sets of headphones, and one set of ears.
I'm STILL waiting to see a single properly run test that shows SIGNIFICANT results either way.
(And I'm sure nto seeing several different tests whose results support or agree with each other.)

I would not expect anyone to "take anybody's word for anything".
Therefore, I'm still simply waiting to see actual test data either way.

WIth any luck, eventually, someone will run such a test (or several folks will run different ones).
Or, until then, we can continue to discuss the results we expect, and what we each believe based on the current incomplete and inconclusive information.
This doesn't seem unreasonable to me...
But, personally, I would prefer to avoid prtending that we have data that we do not.

Great Keith. The VW scandal is about corruption, not science. Same thing with this discussion about audio myths, in my opinion. Now, you claim to hear differences sighted without any confirmation. The null hypothesis for that assertion would be that no audible difference exists that can be identified. We can't prove a negative, so the onus is on you to show that a difference can be heard. Thus far, you haven't provided any proof other than what would be obvious differences if the measurements were made available from the huge amounts of advertisements you frequently post to support your company and what you do for that company. I get it. With the amp off, the sound appeared to be softer, as if it were silent. Amps do sound different.

 

[gregorio]

I think we need to get away from the idea that there's such a thing as a "perfect" test in this domain.

Why? We're talking about DACs here, how is measurement with say a null test not a perfect test?

[1] I'm pretty convinced that I heard a difference - in a sighted, and not especially scientifically conducted, test.
[2] You, who haven't heard either product, and weren't there at all, seem quite convinced that you KNOW no such difference existed.

1. Then why didn't you do an objective (Null) test, it would have cost you next to nothing and taken all of 10 minutes or so?

2. I'm quite convinced that no such audible difference existed, unless the fidelity of the DACs in question had been deliberately or incompetently compromised. I base this on a generalisation from years of my own (and the pro audio industry's) objective (inc. Null) tests between DACs and I'll continue to be so justifiably convinced until some objective, reliable and compelling evidence is presented to the contrary.

[1] What I said was that, in order to prove it, reliably and scientifically, using a test procedure that lacks obvious flaws, would be complex.
[2] (For example, I can't speak for you, but I have no source of "scientifically proven transparent headphone amps" or "headphones guaranteed not to obscure small differences".... so I would have to test and qualify all the equipment first. )

1. And, what I've said is that this repeated assertion is INCORRECT. A null test is not complex and it would help your cause if you could show a difference file containing something that's at least potentially audible.

2. And again, what's that got to do with anything? We're not testing headphones, we're testing DACs.

[1] Neither one of us has bothered to actually perform a proper test.... And neither of us has presented the results of one....
[2] I am unwilling to make assumptions based on incomplete data ...

1. So why didn't you? This is the science forum, you can't simply assert theories and facts on the basis that you couldn't be bothered to perform a proper test!

2. I on the other hand am. For example, I'm willing to make the assumption that no adult can hear filter artefacts above 20kHz at say -40dBFS, even though not every adult on the planet has been tested for hearing response above 20kHz at quiet levels (and therefore the data is incomplete). Do you not make this assumption too? And if not, why not, as all the reliable scientific evidence overwhelmingly indicates such an artefact would be inaudible?

G

 

[bigshot]

OK..... so where ARE the test results you wanted to provide that show that no audible differences exist?

THE FIRST POST OF THE THREAD YOU ARE POSTING IN!!!!! HAVE YOU READ IT?

And for the hundredth time... I"M NOT SAYING DIFFERENCES DON'T EXIST. I SAY I HAVE NEVER ENCOUNTERED ONE AMONG DOZENS OF SAMPLES AND I WOULD LIKE AN EXAMPLE OF A DAC OR PLAYER THAT SOUNDS CLEARLY DIFFERENT SO I CAN CHECK IT FOR MYSELF.

All caps might help getting the blatantly obvious point across. There is a test in the first post where DACs are put in a shootout to find the best one- random results because they were all the same. Most of us here in Sound Science do controlled tests and measurements on a regular basis and we have found that our DACs and players are audibly transparent. You haven't gone to that trouble. DACs are measured and specs are published... all of them well within the range of audible transparency. There is an overwhelming body of evidence to support the argument that DACs are audibly transparent. I have yet to see any evidence that they aren't. I am simply asking for some evidence if you are going to claim that.

All opinions aren't equal. Some opinions are supported by evidence. Some are smoke and mirrors. When I run across those sorts of opinions, it's easier to determine that they are flawed than it is to figure out why they are flawed. It can be unconscious bias, it can be conscious cherry picking, it can be cognitive or communication problems, it can be lack of a grasp on elementary logic, it can be to protect their ego, it can be to gain a commercial advantage, or it can be outright intent to deceive. I've seen all kinds of those things coming through our doors here over the past 14 years that I've been posting here. I am still learning why people insist on trying to defend ignorance. I'm a lot better at spotting the errors than I am knowing why. But I take it all in and process it. Maybe someday I will know.

 

[kukkurovaca]

THE FIRST POST OF THE THREAD YOU ARE POSTING IN!!!!! HAVE YOU READ IT?

So, I took a look at the first post, and I found one link to a blind test comparing DACs, and casually browsing that post, it seems likesome of those comparisons produced a consensus preference among the listeners, particularly at the first level, before the lower-performing DACs were eliminated.

Wouldn't that seem to fit with what I think is a relatively common and uncontrosversial stance/assumption that there are sometimes audible differences between DACs (and between SS and tube implementations), but the differences between well-implemented DACs are not significant?

 

[bigshot]

They did a runoff test which is going to eliminate contenders, even through random chance. They also identified the two contenders and allowed the judges to discuss them among themselves. That makes for "group think". The important thing is the result- cheap vs fancy- which showed that there was probably no clear consensus on any of the ones that were selected or eliminated. Just agreement on randomness. There used to be more published DAC tests in the post, but the links expired. Audio magazines have been eliminating non-advertiser friendly content lately.

Here are some more if you want to dig further...

https://www.diyaudio.com/forums/dig...blind-test-audible-difference-whatsoever.html

http://nwavguy.blogspot.com/2011/03/dac-listening-challenge-results.html

This one involves CD players, not DACs but it is the same kind of thing.

http://matrixhifi.com/ENG_contenedor_ppec.htm

More useful links

https://hydrogenaud.io/index.php/topic,82777.0.html

I don't have time right now to dig them all up, but there are lots more in forums. A simple google search will pull them up. But you'll have to sort through the threads to find the results and see if their testing methodology might have caused error. If you can find a DAC or player that appears to be clearly different than others (not due to amping) in real world use, I would be interested in hearing about them.

 

[Jaywalk3r]

Aren't those lights the same ones they're selling at the P.O.S. price point?

Sort of.

Manufacturers bin emitters by performance and color temperature, within the same model. The POS/budget flashlights tend to get the poorer performing emitters, those that are efficient only at high color temperatures (which, counter-intuitively, produce very cool light). Better lights use emitters from better performing bins, with visually pleasing color temperatures and, often, high Color Rendering Index (CRI) values. The increased cost of the better-binned emitters is trivial for a light that will sell direct to customer for well over US$100, but would be a significant increase in cost of goods sold for budget lights that utilize more traditional supply chains.

Most custom makers tend to use their own electronic controls for the emitter. These are frequently more efficient than with which budget lights are fitted. It's sometimes programmable. The spacing between levels is usually better thought out. UIs tend to be more user-friendly.

Budget lights tend to use lower quality reflectors, with noticeable visual artifacts.

For most people, most times, only the UI is going to be the only real difference. On occasion, we might need to see colors fairly accurately (Is that dark puddle under the car oil or transmission fluid? Is this steak medium-well, as ordered?), a task low-end bins tend to do very poorly, except for blues. When colors are important, high-CRI LEDs are the only good alternative to incandescent flashlights. But most times, two beams of roughly the same brightness, and of the roughly same shape, are going to be pretty equivalent. Beam artifacts are lost in the textures of the world. The higher price pays for better regulation, better host enclosure, better UI, and better quality control, not superior light.

I've never handled a Cool Fall light. However, I daily-carry a high-CRI McGizmo Sapphire and a custom-configured HDS Systems EDC Rotary. The McGizmo is titanium keychain-jewelry (also the most useful keychain light I've owned), while the HDS is a technological marvel of utility and reliability. It has 24 equally spaced levels, such that the switch from one to the next is barely perceptible, ranging from a low-low that will run for weeks, all the way to maximum. The interface is programmable. Neither the McGizmo nor the HDS is designed as a dive light, but both are built to tolerances sufficiently tight that they can be used for that purpose. The HDS Systems EDC was originally designed for caving, The light that gets you home.™

 

[Jaywalk3r]

Neither one of us has bothered to actually perform a proper test....

Incorrect. You haven't performed a proper test, given the differences you claim to have been able to hear. Bigshot's tests have been amply sufficient, given what he is claiming.

I am unwilling to make assumptions based on incomplete data;

Wait, what??? Were you able to type that with a straight face? That's all you've done this entire thread, made assumptions on incomplete data. As an example, you've claimed there exists an audible difference between two DACs with exactly zero supporting evidence, only your assumption that you are somehow magically immune from the cognitive biases that affect the rest of the species.

 

[TheSonicTruth]

Sort of.

Manufacturers bin emitters by performance and color temperature, within the same model. The POS/budget flashlights tend to get the poorer performing emitters, those that are efficient only at high color temperatures (which, counter-intuitively, produce very cool light). Better lights use emitters from better performing bins, with visually pleasing color temperatures and, often, high Color Rendering Index (CRI) values. The increased cost of the better-binned emitters is trivial for a light that will sell direct to customer for well over US$100, but would be a significant increase in cost of goods sold for budget lights that utilize more traditional supply chains.

Most custom makers tend to use their own electronic controls for the emitter. These are frequently more efficient than with which budget lights are fitted. It's sometimes programmable. The spacing between levels is usually better thought out. UIs tend to be more user-friendly.

Budget lights tend to use lower quality reflectors, with noticeable visual artifacts.

For most people, most times, only the UI is going to be the only real difference. On occasion, we might need to see colors fairly accurately (Is that dark puddle under the car oil or transmission fluid? Is this steak medium-well, as ordered?), a task low-end bins tend to do very poorly, except for blues. When colors are important, high-CRI LEDs are the only good alternative to incandescent flashlights. But most times, two beams of roughly the same brightness, and of the roughly same shape, are going to be pretty equivalent. Beam artifacts are lost in the textures of the world. The higher price pays for better regulation, better host enclosure, better UI, and better quality control, not superior light.

I've never handled a Cool Fall light. However, I daily-carry a high-CRI McGizmo Sapphire and a custom-configured HDS Systems EDC Rotary. The McGizmo is titanium keychain-jewelry (also the most useful keychain light I've owned), while the HDS is a technological marvel of utility and reliability. It has 24 equally spaced levels, such that the switch from one to the next is barely perceptible, ranging from a low-low that will run for weeks, all the way to maximum. The interface is programmable. Neither the McGizmo nor the HDS is designed as a dive light, but both are built to tolerances sufficiently tight that they can be used for that purpose. The HDS Systems EDC was originally designed for caving, The light that gets you home.™

BTW do you know how much is the 'POS price point'?

 

[Jaywalk3r]

BTW do you know how much is the 'POS price point'?

These days I do not.

Here are a few of the more important things about lights:
It takes a ~40% increase in light for the increase to be visually perceptible. You're unlikely to notice any difference switching from 200 lumens to 250 lumens, for example.
If you'll use the light frequently, it's worth getting an emitter with a color temperature you find pleasant. For myself, chasing a blue-white beam for long periods isn't very pleasant.
Seeing colors correctly is important enough, often enough, that high-CRI emitters should always be preferred, all else close to equal.
Indoors, at night, how low a light goes is more important than how bright it goes. Low low modes are frequently useful.
It's better that a multimode light can access low from off than high from off.

If it can't fail, I'm always reaching for my HDS or one of my Surefires. Surefire is probably still the baseline for premium quality, but that quality is overkill for most people's needs. Flashlights also have their own equivalent of Chi-fi, imported models that are sometimes surprisingly good for the price point. You could probably treat sub-$50 flashlights similarly to sub-$50 IEMs. You'll likely find a lot of good options, a lot of poor options, but very few models that compare favorably (except on price) with models available at the $350 price point.

 

[TheSonicTruth]

These days I do not.

Here are a few of the more important things about lights:
It takes a ~40% increase in light for the increase to be visually perceptible. You're unlikely to notice any difference switching from 200 lumens to 250 lumens, for example.
If you'll use the light frequently, it's worth getting an emitter with a color temperature you find pleasant. For myself, chasing a blue-white beam for long periods isn't very pleasant.
Seeing colors correctly is important enough, often enough, that high-CRI emitters should always be preferred, all else close to equal.
Indoors, at night, how low a light goes is more important than how bright it goes. Low low modes are frequently useful.
It's better that a multimode light can access low from off than high from off.

If it can't fail, I'm always reaching for my HDS or one of my Surefires. Surefire is probably still the baseline for premium quality, but that quality is overkill for most people's needs. Flashlights also have their own equivalent of Chi-fi, imported models that are sometimes surprisingly good for the price point. You could probably treat sub-$50 flashlights similarly to sub-$50 IEMs. You'll likely find a lot of good options, a lot of poor options, but very few models that compare favorably (except on price) with models available at the $350 price point.

FYI: If it sounds too good to be true, it's probably $19.95 (or 19.99!) In other words, a (P)iece (O)f (S)h*t! lol

 

[KeithEmo]

Really?

BigShot is claiming that there is NO audible difference, between ANY two well designed DACs, that can be heard by ANY human being, using ANY combination of test equipment, and ANY content sample. It seems to me that such a broad calim would require a significant amount of proof before I would accept it as a generality.

If he were claiming that "there is very little audible difference between most well designed DACs" I wouldn't disagree at all.

Incorrect. You haven't performed a proper test, given the differences you claim to have been able to hear. Bigshot's tests have been amply sufficient, given what he is claiming.



Wait, what??? Were you able to type that with a straight face? That's all you've done this entire thread, made assumptions on incomplete data. As an example, you've claimed there exists an audible difference between two DACs with exactly zero supporting evidence, only your assumption that you are somehow magically immune from the cognitive biases that affect the rest of the species.

 

[Jaywalk3r]

Really?

BigShot is claiming that there is NO audible difference, between ANY two well designed DACs, that can be heard by ANY human being, using ANY combination of test equipment, and ANY content sample. It seems to me that such a broad calim would require a significant amount of proof before I would accept it as a generality.

If he were claiming that "there is very little audible difference between most well designed DACs" I wouldn't disagree at all.

No. Bigshot is claiming that he's never seen (heard) any reason to doubt the null hypothesis.

You, on the other hand, would have us reject the null hypothesis wholly without any data to suggest unlikely.

You are making an extraordinary claim, and the onus is on you to back up that claim with solid data, lest your claims be justifiably dismissed out of hand as BS. Bigshot is making no such extraordinary claims. He has no onus to provide extraordinary evidence. If you don't get that, you're in the wrong forum.

 

[KeithEmo]

OK... I guess I owe BigShot an apology.
Apparently, I misunderstood, and he is NOT saying that "differences don't exist" or trying to claim that "all well designed DACs audibly sound exactly the same" .
In that case, we seem to be in agreement after all.
I too have found many instances where well designed DACs sounded so much the same that I couldn't tell them apart.

However, everything I've said so far still stands.
He asked what DACs I suggest testing for audible differences... and I responded with a suggestion.
I responded.
I will eagerly await the results of a properly conducted test on those two DACs if someone performs one.
(And we obviously both have our own personal predictions about what we believe those results will prove.)

Yes, DACs are measured, and the test results are published.
And, yes, most of them test very similar in some ways, and different in others.
And we're back to either actually testing for audible differences...
Or making assumptions about what the tests we haven't performed will show...

And, no, I for one am not comfortable writing off every result of every test that showed "some sort of preference" as "they must all have been due to bias".
Because I have yet to see a test that provided results in either direction that DIDN'T suffer from serious flaws or limitations.

THE FIRST POST OF THE THREAD YOU ARE POSTING IN!!!!! HAVE YOU READ IT?

And for the hundredth time... I"M NOT SAYING DIFFERENCES DON'T EXIST. I SAY I HAVE NEVER ENCOUNTERED ONE AMONG DOZENS OF SAMPLES AND I WOULD LIKE AN EXAMPLE OF A DAC OR PLAYER THAT SOUNDS CLEARLY DIFFERENT SO I CAN CHECK IT FOR MYSELF.

All caps might help getting the blatantly obvious point across. There is a test in the first post where DACs are put in a shootout to find the best one- random results because they were all the same. Most of us here in Sound Science do controlled tests and measurements on a regular basis and we have found that our DACs and players are audibly transparent. You haven't gone to that trouble. DACs are measured and specs are published... all of them well within the range of audible transparency. There is an overwhelming body of evidence to support the argument that DACs are audibly transparent. I have yet to see any evidence that they aren't. I am simply asking for some evidence if you are going to claim that.

All opinions aren't equal. Some opinions are supported by evidence. Some are smoke and mirrors. When I run across those sorts of opinions, it's easier to determine that they are flawed than it is to figure out why they are flawed. It can be unconscious bias, it can be conscious cherry picking, it can be cognitive or communication problems, it can be lack of a grasp on elementary logic, it can be to protect their ego, it can be to gain a commercial advantage, or it can be outright intent to deceive. I've seen all kinds of those things coming through our doors here over the past 14 years that I've been posting here. I am still learning why people insist on trying to defend ignorance. I'm a lot better at spotting the errors than I am knowing why. But I take it all in and process it. Maybe someday I will know.

 

[KeithEmo]

Not at all... I'm not asking anyone to accept or reject anything.
In science, a hypothesis generally isn't "accepted " or "rejected" without evidence and experimentation.
In science, over time, a hypothesis accumulates data that either supports or contradicts it.
All I've ever said is that there are lots of assumptions and very little actual relevant test data

And the same holds true for the ever-popular debate about "whether high-res audio is audible".
A bunch of tests HAVE been conducted on that question.
But every single one I've seen described in detail was deeply flawed.
(That doesn't mean that the results were necessarily wrong; just that they are far from truly conclusive.)

No. Bigshot is claiming that he's never seen (heard) any reason to doubt the null hypothesis.

You, on the other hand, would have us reject the null hypothesis wholly without any data to suggest unlikely.

You are making an extraordinary claim, and the onus is on you to back up that claim with solid data, lest your claims be justifiably dismissed out of hand as BS. Bigshot is making no such extraordinary claims. He has no onus to provide extraordinary evidence. If you don't get that, you're in the wrong forum.

 

[KeithEmo]

Honestly... I don't care what, or who, you believe... and I "wouldn't have you do" anything whatsoever.

I've simply pointed out that "the default hypothesis" is based on incomplete, out of date, and largely only tangentially related information.
I've also noted, in various instances, where the tests a lot of people seem to treat as gospel were badly designed, poorly executed, and often just downright sloppy.
Of course, this doesn't necessarily mean that their results were wrong, but they are far from authoritative, or worthy of being treated as definitive in the general case.
(You might as well test a half dozen snake oil cancer cures, note that none of them work, and conclude that "all cancer drugs are snake oil and should be ignored".)

I should also point out that I didn't actually make a claim.
BigShot asked if I had noted any DACs where I believed there was an audible difference; I responded... quite accurately, and in absolute detail, BASED ON MY PERSONAL EXPERIENCE.
I would still suggest that, IF ANYBODY WANTS TO RUN AN ACTUAL TEST, those two DACs would be pretty good candidates.
(They are both considered to be "high end products", both measure very well, and both seem quite characteristic of typical implementations for their respective DAC chips.)
Obviously, in one sense, it would make sense to choose current production samples (unless you're planning to purchase samples on the used market or find volunteers to donate them).

I should also point out that performing a proper null test on DACs is actually quite difficult.
(It is relatively simple on analog devices like amplifiers because they usually don't introduce a significant time delay - which allows the signals to be subtracted directly.)
However, unlike analog devices, most modern DACs introduce a delay of somewhere between a fraction of a sample and multiple samples, and the drivers on the computer involved often introduce several more milliseconds of delay to each..
Therefore, in the analog domain, there tends to be a slight time offset between the two samples, which makes simply performing an "accurate analog subtraction" impossible.

And, while performing a direct comparison of digital samples is relatively easy, and there is at least one piece of software that allows "fractional sample time adjustments", you would still require difficult-to-achieve precision to do so.
You would need to sample your sources at extremely high precision, both in terms of time and amplitude, to rule out artifacts, or missing something due to simple lack of detail.
(In most cases, you're likely to get a very poor null, but there will be a very good chance that most of the differences you find will be due to testing limitations... which makes the entire test largely invalid.)

No. Bigshot is claiming that he's never seen (heard) any reason to doubt the null hypothesis.

You, on the other hand, would have us reject the null hypothesis wholly without any data to suggest unlikely.

You are making an extraordinary claim, and the onus is on you to back up that claim with solid data, lest your claims be justifiably dismissed out of hand as BS. Bigshot is making no such extraordinary claims. He has no onus to provide extraordinary evidence. If you don't get that, you're in the wrong forum.