I attended the talk. It was mostly about knowing which FFT parameters to use and making sure you are using bit-exact path to test products. My team wrote the Windows audio stack while I was at Microsoft, and one of my professional specialities is signal processing (I managed that team at Microsoft too) so, neither was news to me. His testing and mine both use bit-exact and I have taken his measurements and replicated them on my Audio Precision analyzer showing the same problems. In almost all cases Bob's data actually matches mine. It is just that his words are far more positive and people go by that, instead of data that represents otherwise.
I have shown this numerous times on ASR Forum. There is no question that Schiit DACs don't measure well no matter who measures them. Buy them for other reasons than engineering excellence.
Yeah but give me credit for having more Audio Precision gear than anyone!
There are no less than three of them in this picture (with one just peeking from the corner):
First off, my opinions and responses to your assertions are meant as a reflection back to you, as such it is not my intention to demean you nor your efforts, but to provide you a form of feedback which can be used as a means for you to assess what myself and others see as discrepancies.
Take them as you will, but these are my observations and assessments based upon your contributions, and if viewed as such they could be quite helpful for you to help refine your processes.
And it appears evident to me that your processes need more refinement in order to be accepted as ‘real’, at least by me and those who agree with me.
And to be clear, I wasn’t there for that AES meeting, but I did get reports back from those who were, as to what was presented and based upon that, “knowing which FFT parameters to use” wasn’t what it was “mostly about”, and the “making sure you are using bit-exact path to test products” is only one of the more important topics that were presented.
Indeed the other unmentioned topics were of far more importance, not just in what was presented, but also in their implementation, and what their effects would be if they were applied in all area’s of properly taking measurements and subsequently validating them.
And your claim that “There is no question that Schiit DACs don't measure well no matter who measures them” is simply an inaccurate assessment.
It is true that they tend to be more difficult to measure accurately and will measure differently that many other dacs, which is to be expected, but your bias (based upon your measurements, which are still in question), is the focus of my opinion on these statements.
Proudly showing your investment in gear is an indication, to me, of misdirection, an attempt to bolster your credibility, which actually works against you.
IOW bragging rights don’t lend credence nor viability in and of themselves, real, accurate results do.
Um I disagree with this statement as well, “In almost all cases Bob's data actually matches mine”.
Since I see anomalous and spurious artifacts interspersed in your results, and this is key.
This says to me your setup and situational conditions are not equivalent between your setup and his.
Additionally this tells me that your test conditions allow spurious artifacts to be included into the measurements.
As a result these additional artifacts are allowed to contaminate your results.
As for “his words are far more positive…”
Again more misdirection away from the basic issues of measurement accuracy, reliability, and credibility.
And data is ALWAYS subject to scrutiny, because it’s ALWAYS easier to generate bad data than accurate and ‘real’ data.
This is a known and given aspect when using measurement gear, where the resolution and dynamic range of all of this gear (both measurement and DUT’s) is capable of measuring such small signals.
Which makes the job of obtaining ‘accurate and ‘real’ data’ all the more difficult and problematic.
Seriously, you are right that it is easy to make mistakes and knowledge level it takes to test mixed-signal products like DACs can be pretty large. I like to think that as an owner of such measurement gear for 20+ years, and professional experience related to everything here from analog to digital and signal processing, I know what I am doing. But if I am not, I am open to others showing otherwise. So far, that has not happened.
So while there can be doubt in everything, I suggest by default you should accept the data as presented unless it is shown otherwise. Doing it in reverse means closing one's eyes to information that is purely created to make consumers more informed.
“I am open to others showing otherwise” is laudable and should be de rigueur for anyone who aspires to improve their processes, especially with gear that pushes the envelope into these extreme voltage ranges of -100dB to -160dB down, where even the smallest voltages (one ten thousandths of a volt to as low as one hundredth millionths of a volt) are being measured.
These voltages are not trivial nor easy to insure that extraneous outside sources of such small voltages are excluded from your measurement setup.
And this is but one lone aspect, one that is rather difficult and sometimes complex to adequately deal with, for a variety of reasons.
As for your claim that “So far, that has not happened” it is again, in my opinion, simply an inaccurate assessment, as there are published results and indeed mounting evidence that at a minimum your results are at variance with other more well established measurements.
Not to mention those ‘other’ aspects mentioned above that were covered in the AES presentation.
As for, “there can be doubt in everything”, because in this case there is doubt, but it is of your methodology in obtaining results, based upon your published results.
And that we should just blindly “by default”, “accept the data as presented unless it is shown otherwise”, is self evident to me and others, to simply not be applicable.
And “closing one's eyes to information that is purely created to make consumers more informed” is yet another distraction, one that is unrealistic in this day and age where anyone can obtain the necessary measurement gear and create results.
It’s much more a matter of, do the results reflect the reality of the DUT, do they corroborate with others results and do they actually benefit the entire audio community as a whole.
It is my hope that these comments and details can help you improve your processes which in turn should help you obtain more meaningful results.
JJ
