...You assume that he does apply bandwidth limiting to achieve the optimized results. To me it looks like he's applying the inverse of a smooth version of the deviation of the analyzer's DAC. Does the APx555 have such a feature? Where you can define a mathematical formula to apply to the measurement result? The deviations from perfect linearity in his graph seem to correspond inversely to deviations from an idealized curve (regression, I suppose) approximating the sample points below -100 dB in his unoptimized result.
Maybe I'm completely wrong about this, but if so, he would apply the inverse of the analyzer's DAC imperfection to every measurement of other DACs (because he attributed them to the analyzer's measuring components, not its generating components).
[Edit: I don't know likely that is, given that he is clearly aware of that possibility when he says "What this means is that the AP has a positive error in "linearity" (It is more than that but let's go with it) at levels less than 100 dB. As such, you can not, let me repeat, NOT measure any DACs with it as you will be showing the sum total error of both the DAC and ADC measurement errors in the Audio Precision." That would be wonderfully ironic.]
But, again, just spitting out hunches.
@Alcophone, regarding the assumption that he's bandwidth-limiting, here's something
@amirm posted on 2018-06-28, confirming that (a) his linearity measuring method does indeed involve the removal of noise, and (b) trying to suggest that doing that is what he's been saying to do all along (when anyone following this knows that is most certainly not the case):
amirm at audiosciencereview said:
I want to emphasize again that substantial amount of noise reduction exists in my measurements. Levels are attenuated by a whopping 50 dB on either side of the frequency of the generator. Without it, you just measure the noise, not the signal...
Again, in response to his 2018-06-08 criticism of our method which measured anything (noise and THD included) in the audioband, that's what I did in
my second set of linearity measurements on 2018-06-15 (after having discussed the topic at length with the team at Audio Precision). Of course,
@amirm criticized even those measurements roundly. And after someone pointed out to him that I had contacted AP to discuss this, he said:
amirm at audiosciencreview said:
Not quite. I explained how my measurements were made to AP folks after pointing out to Jude that he was doing it wrong. They then worked with Jude on an implementation even though I had told Jude that the right solution on APx555 requires much more research. I have explained all of this to AP folks this morning again and issues in what Jude has published.
In a nutshell, they have set up a test where all the distortions and aberrations of the DAC are wiped clean. And they then declare: "oh look, it is linear down to -130 db" or whatever. Well duh. Of course if you remove all the noise and distortion from a DAC, it then looks accurate. Why bother running such a test when in real usage of the DAC no such filtering exists. Talk about running off with measurements with no thought of what they really mean and what benefit and correlation there is with audibility.
On top of that Jude continues to block all of my posts on head-fi. So no way for me to convey this information there. This is not the way we converge to a consensus.
So my advice remains: please wait to draw any conclusions until I remeasure the device with my APx555. Until then, my data remains 100% valid in pointing out serious issues in the performance of Yggdrasil.
And let's not forget this gem:
amirm at audiosciencereview said:
...I can pull rank on Jude and even AP folks on my understanding of such topic. But I am not. So let's move on such tactics.
He then blamed me for asking AP the wrong questions:
amirm at audiosciencereview said:
No, they [Audio Precision] do know how to "set up the machine." Question is, what are they being asked to set up?
What they were asked to help setup was detection of level while eliminating all distortion and noise. This is NOT what we want. If the device creates X amount of noise and distortion on top of Y signal, we want to measure both.
What they needed to ask instead was how to replicate this measurement I made on AP2522 but on APx555
There he goes with his insistence that the 25-year-old SYS2522 is better suited to measuring this than the current flagship APx555, and that the only right way is to replicate that particular analyzer.
amirm at audiosciencereview said:
...The test I ran [on the Yggdrasil 2] takes into account distortion and noise and hence is able to differentiate between DACs easily. Theirs does not. I know because I replicated their method and it would no longer do anything useful.
Remember that statement, because it won't be long before I get back to that.
When asked if he's suggesting if he's the only one who knows how to do this properly, he responded (again, in that same post):
amirm at audiosciencereview said:
I am currently the only one who:
1. Owns both the 2522 and APx555 analyzers
2. Have a large body of results and many DACs on my bench to evaluate using both analyzers
It has taken me good bit of effort to replicate the way the 2522 measured linearity on APx555 with the above tools.
The 2522 has a cascade of analog analyzer and digital analyzer. The APx555 is only digital. So there is no 1:1 relationship between how the two run.
Are you having as hard a time keeping up with his statements as I am? I think that's his intent, because he knows most who read it there will take him at his word and not check it out for themselves. For those who might check it out for themselves, I think he intends to keep moving the shells around so that it's harder to follow. Anyway, here's a bit of summarizing:
- He said the linearity test he ran and posted from his SYS2522 (the older analyzer) "takes into account distortion and noise and hence is able to differentiate between DACs easily. Theirs does not." This would suggest (along with his many other statements on the matter) that his SYS2522 was not bandpass filtering like my second set of linearity measurements.
- He said that my second set of linearity measurements (and the subsequent supporting nested FFT's) -- regardless of who helped me, none of whom he'd have you believe are as qualified as him (which I think is a reasonable interpretation of his "pull rank" comment) -- were useless because of that bandpass filtering.
- On 2018-06-08, he posted a loopback measurement from the APx555 that he says he had "taken me good bit of effort to replicate the way the 2522 measured linearity on APx555 with the above tools." Again, that loopback graph indicated likely bandpass filtering (which we now know to be true).
More quotes from
@amirm:
When asked by one of the forum members at his site to explain what we were doing to "wipe out distortion and aberrations of the DAC" in response to
@amirm's comments, he said:
amirm at audiosciencereview said:
Sure.
The problem is challenging. They are attempting to measure linearity down to -140 dB. As you and I both know, there is no DAC in the world that produces meaningful signal at anything close to those levels. But importantly, there is no ADC in any analyzer that can do the same. Yet they tried anyway based ironically on advice I gave to AP
. That if you use aggressive filtering of noise and distortion, you can indeed eliminate a lot of variability.
So they did that and took that the N'th degree...
This suggests we're doing something wrong (of course) or deceiving (of course). He continued:
...We see that the filter has completely removed all traces of distortion and noise. So of course if you then measure this, it shows that the DAC is doing well.
But that is NOT what we hear out of the DAC. Nor what it electrically produced. We are cleaning up the output of the DAC and then measure it, then declare it a winner.
The "trick" here is to use only the filtering necessarily for the ADC to not have its noise and distortion profile be below that of the DAC under test. This can only be done through a bunch of trial and error which I went through on APx555 analyzer. My older 2522 "happened" to do this well out of box. I tried many things including changing the excitation signal, settling parameters for measurements, custom filtering, etc. I finally found something that while may not be identical to 2522, is very comparable.
Summary
Any filtering in the analyzer cleans both the DAC and ADC output. It is tempting to select an exceptionally narrow filter to get rid of all noise and distortion as to even show accurate values to -140 dB. But we know such data is fictitious as we don't know how to build such DACs. By carefully selecting the filtering and analyzer setting however, we can get reasonable results to about -120 dB. Any attempt to go beyond that in my testing will lead one into a ditch.
P.S. The FFT method is even a more extreme case of such filtering as there, you get to look at one individual spike and ignore all other noise and distortion characteristics.
It seems to me he's strongly suggesting we're attempting to deceive. He calls what we're doing a "trick." He calls it "fictitious." He says by posting my second set of linearity measurements and my later FFT plots (also made with the help of another measurement engineer) we "lead one into a ditch."
Note what he said about the FFT's in criticizing our methods as filtering out noise and distortion:
amirm at audiosciencereview said:
The FFT method is even a more extreme case of such filtering as there, you get to look at one individual spike and ignore all other noise and distortion characteristics.
Yes,
@amirm, that was exactly the point. A point you later use yourself in this graph, in discussing the linearity of the Benchmark DAC3:
So, let me get this straight:
When I used the FFT spectrum view to drill down on the test signal with the Yggdrasil 2 as another measure of linearity I'm being deceptive. But when you do
the exact same thing (only three days later) to illustrate another measure of linearity with the Benchmark DAC 3 it's perfectly legit. I guess this is you pulling rank?
By the way, of that technique,
@amirm said:
amirm at audiosciencereview with emphasis by me said:
We want to use time domain analysis here because AP runs in automated mode there. So we can make the 60+ measurements in linearity mode or whatever we like. Trying to do that in FFT manually gets very tedious. Hence the reason we are interested in proper filtering for measurements in time domain.
It's not that tedious,
@amirm, and the switching of levels in FFT view can be automated. Try APx's Nesting feature.
Continuing...
When asked on his forums why his claims may stretch credibility, he responded:
amirm at audiosciencereview with emphasis by me said:
Why are you not skeptical of the motivation on behalf of Jude? You think he is going into this with just a search for the truth? If so, why ban me from responding to him? Or refusing to send me his AP project files?
There is only one truth here: he wants to help Schiit by invaliding my test results. Which is fine. But he needs to replicate my tests, not invent his own and say it still represents the same thing.
As others have pointed out, he was asked on his forum if he would post his project file (which he has repeatedly all but demanded I should share). Among the key reasons given for not producing project files was file size, stating that they can exceed 50MB. Even if it was 50MB in size...really? The project file I uploaded that includes the settings and one linearity measurement of the Yggdrasil 2 is 79.9K. If you do have a project file that contains large measurements (like 1.2M-point FFT's), then, yes, the project file can be quite large. If sharing your settings is the primary reason, then delete the measurements, and just keep the settings intact. The file should be quite small then.
But, again, even if it is 50MB...seriously? If you don't want to share it, that's fine. That's completely up to you. But don't throw up a silly excuse like file size as one of the main reasons.
And if you're going to insist I need to replicate your tests, wouldn't it help to have some idea how, especially with all of your suggestions that there's a lot of special sauce needed to to do a linearity measurement with the APx555? Yes, it was straight-away assumed you were bandwidth-limiting when you posted your optimized loopback. You're only finally now admitting that (a) it does bandwidth-limit, despite your vehement protestations when I did that, and (b) the results, as you've stated yourself, are the same (in your Benchmark DAC3 linearity example, which I also posted).
He continued:
amirm at audiosciencereview with emphasis by me said:
AP has done nothing wrong to apologize for. They have spent time and effort helping Jude do something he should have known at the start, and had the benefit of my explanation here. If Jude had asked AP to help him replicate my 2522, they would have given him different advice (although not clear they would have been able to give him ultimately what he wants).
Yes, Amir, we all have by now experienced the benefit of your explanation.
He's correct that I did not ask AP to replicate his 2522. I have an APx555 here, so I didn't see any reason to replicate the 2522. Even though I did not replicate the 2522 with the APx555 (and he apparently did),
even he admitted we arrived at the same result. If you're having a hard time following all of this, again, I think that's exactly his objective -- because I think by now he realizes he's talked himself into a pickle. Bandwidth-limiting is deceptive when Jude does it. It's okay when Amir does it using a secret method imbued with decades of signal analysis experience. Oh, by the way -- same dang result.
When a few of his forum mates suggested that bandpass limiting does perhaps make sense for linearity measurements, he stuck to his guns (despite the fact that his loopback showing how it should be done on an APx555 does show bandpass limiting):
amirm at audiosciencereview said:
Two issues:
1. You are measuring what we don't hear. We hear the total signal.
2. There are devices that nail the response to the level we are measuring.
Linearity is the ultimate test of a DAC: that it has a straight line transfer function between input digital samples and output analog. That output analog must be definition include all contributions including noise and distortion.
Checking just the level after removing all noise and distortion is an academic exercise devoid of real world value.
Looking at that bolded part in the quote: In case you haven't figured it out after all this, that's exactly what he's doing, too.
When we're showing other measurements (be they FFT spectrum or THD, THD+N, and/or noise figures) then we have an idea where the DUT's floor and distortion is. We can show linearity plots that take that into account (which I did in the first ones), and we can
also try to get to the very limit of the DAC's ability to linearly decode the signal at the lowest levels (which requires bandpass filtering if the DUT's ability to linearly decode reaches signal levels below the across-the-audioband noise levels of DUT itself and/or the analyzer).
What I'm getting at here is that I think
@amirm is quite urgently trying to convince folks that
only he is qualified to make these measurements, and anyone else's is merely a product of devious motives and/or should be dismissed immediately as unqualified, useless plebeian scribbles. So he'll discredit my measurement for taking noise and distortion through the audioband into account. And then when I post measurements that do not do that, he'll discredit those, too.
To me, he seems so set on establishing himself as the only guy qualified to makes these measurements that he'll "pull rank" on not just novices like me, but even very qualified measurement engineers (some of whom developed and built the insanely precise tools tools he and I both use). And he'll also create criteria as arbitrary as his +/- 0.1 dB error that must be met to pass muster -- like having to come to the table with no fewer than his years of (no doubt impressive) experience, or even a proviso as outlandish and madcap as stating that one must have experience with -- or be in possession of -- his particular model of 25-year-old Audio Precision analyzer (which is admittedly a fine tool to this day) in addition to the one we've got (which is perhaps the finest audio analyzer created to date).
Never mind the fact that after all this hullabaloo
those fictitious, useless plebeian scribbles of mine -- even according to him now -- mirror his own results with the same analyzer.
