2021, Chapter 6:
Measurements II, The Sequel
This is the long-awaited follow-up to
Measurements (with a Side Order of Sanity), a chapter published back in the dim dark ages of 2015, only shortly after the earth cooled.
When I wrote
Measurements (with a Side Order of Sanity), most people thought we were some kind of neanderthal/swamp ooze/mouthbreather crossbreed with extremely limited engineering skills, and absolutely zero concept of measurements whatsoever. “
Sanity” was our way of explaining what tools we use, what we measure, how useful the measurements were, and how easy or difficult it was to get good results.
Looking back, a lot has changed, at least in methodology:
- The chapter was written before we owned a single APx audio analyzer—we now have 3
- It was also written before we had standardized on Avermetrics for production test—we now have 6 (but Avermetrics has thrown in the towel, more on that later)
- Back then, we didn’t put up APx tests of all new products on release
- We didn’t have good EMC prequalification, USB probing capability, or ESD testing capability
- Heck, we didn’t even really use FLIR on the line, which are great for catching stuff and ensuring consistency
- We had a lot less people using sophisticated measurements—heck, it was basically just me—now, there are people throughout the company using the tools
On the other hand, what we believe, and the way we use measurements in development, hasn’t changed very much:
- We believe that measurements are absolutely necessary to develop quality products, and to help ensure no “gotchas” on the production line—on the first run, the 10th, or the 100th
- We also believe that there are differences between components that cannot be quantified by a single number or small suite of steady-state measurements, and these differences may be significant to some listeners
Yeah. I know. Some people think we’re crazy. That’s fine. There are plenty of good products out there, many developed without the “magical thinking” of the latter point above.
But even with our soupcon of magical thinking, we’ll be the first to say that many audiophiles tend to blow small differences wayyy out of proportion, and/or make earthshattering pronouncements after 30 seconds of listening at a show (see “How We Fool Ourselves.”)
But we also like to let people make their own decisions. And pursue our own beliefs. Which is why we do things like True Multibit and discrete designs—even though it doesn’t measure as well as delta-sigma and op-amps, even though it’s more difficult to do, even though it’s more complex to produce. Because we believe it is better.
“Yeah yeah, blah blah, whatever whatever, where’s the chapter?” you might be asking. “What kinda cool gear you got, and what do you measure?”
Fine. I hear you. On to the chapter. For real.
Metrology Players, 2021 Team Edition
What is metrology? Well, if you read the old chapter before taking this one on—seriously, stop laughing—you’d know that “metrology” is just a fancy word for the science of measurement. Last time, we took a look at what was on my desk. This time we’ll be looking at stuff throughout both of our production facilities—Santa Clarita and Corpus Christi.
Audio Precision APx555, with upgraded analog generator (x1). In 2021, Audio Precision has taken over from Stanford as our lab analyzer of choice. We have one APx555, which was just upgraded to the Series B analog generator so we could get some more resolution out of it. It’s the one I use the most when I’m in the Santa Clarita office, but it’s shared between me, Cameron, Naomi…and even Tyler has used it for wow and flutter measurements of the Sol.
- Why do we have this? Because we need the high resolution analog generator for op-amp designs (and for playing with the limits of discrete), and because it’s in a convenient place for a lot of people to use it. Cameron and Naomi frequently use it for repairs. When I am in Santa Clarita, I use it pretty much every day in development.
- How much will this set me back if I want one? Ouch. About $30K to start.
- What alternatives are there? Audio Precision is the leader in this space. If you don’t need a super precise analog generator, you can get by for about half the price with a APx525. An APx515 is smaller and fine for most digital measurements, but its analog generator is a big step down. You can also buy the APx software for use with a number of pro audio interfaces for measurement, but you may take the chance of blowing up an input (unlikely you’ll be able to put 120V p-p into an audio interface, typical of speaker amp measurement). Stanford and Prism are still around, but Avermetrics has thrown in the towel. Too bad, because the Averlab was pretty close to the APx525 in terms of performance, and $3K. Prism has the DScope Series III and M1, which might be interesting in that they probably use the same test files and the M1 is affordable, but we unfortunately have no experience with these products.
- ’m just starting, how do I get decent measurement capability for something that isn’t priced like a car? Look into the QuantAsylum line—they have a QA402 coming for (I believe) $449. It won’t do everything the APx line does, and you’ll need to watch the input levels as with a pro audio interface. Beyond that, it’s pro audio interfaces and open-source software—which you can get surprisingly decent results from!
Audio Precision APx525 Series B (x2). In addition to the APx555, we picked up a couple of APx525s, because (a) they are pretty much as good as the APx555 for digital work, and (b) because the first one we bought was taken by Dave—he works offsite most of the time.
- Why do we have these? Because we needed an analyzer that has many of the capabilities of the 555 for the Corpus Christi office, and because Dave worked out a ton of stuff super fast when he borrowed it for a while. So we ended up letting Dave keep his and picked up another one for Corpus Christi. The CC model sits on the main tech bench there. I can use the same APx files generated by the 555 to do tests in Texas, which is a huge benefit, and our techs there can coordinate with California.
- How much will this set me back if I want one? Ouch. About $14K to start. I know, I know. But being able to use the same test files is a big deal. And increasing Dave’s productivity is an equally big deal. YMMV.
- Alternatives? See above.
Avermetrics Averlab (x6). This is the workhorse of the line, used for 100% testing of equalizers, phono preamps, and parallel-input True Multibit products (Modi Multibit and True Multibit cards). In summary, this was a damn nice analyzer for a really appealing price, hamstrung only by the software. It never got the capability to do automated USB measurement, for example, like the APx line.
- Why do we have these? Because they were the best way to do production line test for the lowest investment (the 6 units here cost less than one APx555), and they were able to share their own automated test routines. Much like the APx line, the Averlab could be programmed to do sequence and pass/fail testing, and those files could be shared amongst the Averlabs through the shop. They were also relatively easy to use and small.
- How much will this set me back if I want one? They aren’t available new anymore, sorry to say. When new they were $3000.
- Alternatives? This is something we have to look at, as these products will eventually fail on the line, and will need to be replaced. The easiest option is the APx515, which could use the same test files as our other gear. Those are about $7K to start, though. Using APx software with pro audio interfaces is another option. In short, we’ll see. So far, all the Avermetrics are still working.
Custom Cables and Loads for Analyzers. Sounds kinda silly to call out cables? Well, use a crappy BNC-RCA adapter that screws up your test results 3 times out of 10 and suddenly cables are a whole lot more important. We get custom cables made exactly to our specs (like BNC to RCA, or male 4-pin XLR to dual 3-pin for headphone output testing), make our own custom headphone load boxes (with switchable loads, or the provision to use a headphone as a load), and have a big variety of 250W power resistors to use in power amp testing.
- Why do we invest in custom cables, etc? Because bad test results can send you down all kinds of bad roads, and end up costing tons of time and money. Before we went to high-quality, custom cables, I would regularly think a new design was bad, and waste a bunch of time trying to fix it. Heck, even today, I have to pay attention to cables—leaving the single-ended inputs connected when testing balanced, and/or vice-versa, can cause spurious results. If you’re going to get an analyzer, it’s worth making sure your results aren’t being borked by bad cables, or bad cable management.
- How much will this set me back? Not that much on the cable side, and you can do it yourself. We’re not talking hand-braided yak-hair insulation and unobtanium conductors here, we’re talking Canare and Neutrik or similar. If you’re talking custom load boxes, it doesn’t cost much to do a short run of boards, or wire up one point-to-point. And if you’re talking big 250-500W speaker loads….erm, hit up eBay to get some used stuff for 1/10 the cost of new.
- What alternatives are there? Not knowing your measurements are good, wasting a bunch of time, tearing your hair out…but those are some crappy alternatives!
Agilent/Keysight MSO-X 2024A Mixed Signal Oscilloscope (x6) and friends. “Why a scope if you have all these damn analyzers?” seems like a relevant question given what’s come before. But if you’re looking for quick “does this work?” or signal-tracing troubleshooting, a scope is hard to beat. We’ve standardized on the Agilent/Keysight MSO-X series, due to its moderate cost, high performance, and surfeit of features (if you shop intelligently, you can usually get a refurb with 4 channels of analog, 8 channels of digital, and function generator turned on at a good price—Keysight sells refurbs on eBay, hint hint. But we also have some wacky “friends,” including a GHz scope (for Dave) and some other non-Agilent/Keysight stuff for quick production checking. There are a ton of scopes in both offices.
- Why do we use these? For quick “Does it work? Is it oscillating? How’s the clipping look?” type measurements, a scope and function generator a fast way to sanity-check a design. Plus, the bandwidth of an audio analyzer is quite limited—compare the 1MHz bandwidth of the APx555 to the 100MHz to 1GHz bandwidth of an oscilloscope. You probably won’t even see a design oscillating on an AP (though you will probably see some oddball distortion), but you’ll see the fat trace on a good oscilloscope immediately.
- How much will it set me back? New, about $3500 for a 100MHz/4 channel version like we use above. Starting at $500 or so for the Keysight 1000-series models. Up to $35-50K for super high bandwidth/highly capable models. Check eBay for refurbs to save a ton of money.
- What alternatives are there? Tons. Agilent/Keysight is just what we use. If you’re a Tek aficionado, don’t get up in arms. There are plenty of good inexpensive options as well—but note that on the lowest end of the market, the waveforms displayed may not look so “analog” as the higher-end products, making troubleshooting more difficult. And seriously, with Keysight and Tektronix starting around $500, why be so cheap?
Agilent or Tektronix AFG (x3) and friends. “AFG” is short for Arbitrary Function Generator. These are devices that output sine waves, or square waves, or ramps, or noise, or a whole lot of weird functions (as in, arbitrary—hence the name.) This may sound like a kinda bizarre piece of test gear, but when paired with an oscilloscope, it’ll let you see some things you can’t with an audio analyzer. And yeah, the “and friends” is because we probably have 4 more non-Agilent or non-Tektronix versions around the shop, but the lifespan of those devices seems to be, er, a bit short. So we’ll talk about the ones that survive.
- Why do we use these? Because if you want to look at square wave performance, or open-loop frequency response out to 10MHz, you won’t be doing it on any APx. And this is a very important thing when it comes to analog electronics, especially discrete analog electronics—this tells you a lot about the stability of a design.
- How much will it set me back? About $1200-2000 for a good one.
- Alternatives? Yeah, they exist. They are cheap. They seem to last about a year. Your call.
Fluke 8846 6.5 Digit Multimeter (x6) and friends. These are desk multimeters that allow you to measure things like AC and DC voltage, 2-wire and 4-wire resistance. We’ve standardized on the newer 6.5 digit model, which has more than enough accuracy to chase down things like shorts on a board. There are also lots of friends—most every desk has a Fluke of some kind on it, from 8840 to 8846.
- Why do we use these? First reason: for checking if your power supplies are working, or if you put the wrong resistor in somewhere, or if you have DC offset, or a board short…you need a good DMM, period. Second reason: battery-powered multimeters are always out of battery when you need them most…and you usually forgot to stock up on the 9V batteries.
- What will it set me back? A new fancy 6.5 digit Fluke will be about $1000-1200. Used is wayyy less expensive. Make sure the display is still bright.
- Alternatives? Come on, get a Fluke. You know you want one. If for the name alone. (A test and measurement company being called “Fluke” must rank up there with Schiit.)
FLIR C5x, C3x, C2 (x2,x1,x2). These widgets show you graphically how hot something is getting. Sounds boring? Oh no. These are amazing tools, which is why we have a boatload of them. I use one in development, Cameron uses one in repair, there’s two more on the line and one in Corpus.
- Why do we use these? Because they are the fastest way to see if something is working, and working right. Everything is dark? Then the circuit isn’t biased. Blazing like the sun? Something’s too hot. And it beats the hell out of losing thumbprints on hot components with the “finger test.” Seriously, it’s hard to figure out sometimes if a SOT23-sized component is running at 50 degrees C or 150 degrees C—and if it’s the latter, it won’t last long. It’s also easy to compare new runs against old runs to make sure something isn’t amiss in the bias network, or elsewhere. Flirs are one of our most used tools.
- What will one set me back? Not too horrible, $500 or so for a good one. Of course, like everything else, if you want super-fancy, bring a big wallet.
- Alternatives? You can get a Flir or Seek Thermal camera that connects to your iPhone or Android phone—both are less expensive.
Handheld Bug Zapper, aka ESD Gun. This is how you check your stuff to make sure it can take an ESD hit. Like, when one of our techs who lives up in Lancaster (very very dry) shuffles his gum boots over the shag carpeting and gives a component a 1.5” lightning bolt from the tip of his finger that makes the air smell like ozone (some slight hyperbole here, but you get the picture). We usually send out for complex ESD problems, but for in-house testing, we have a gun.
- Why do we use this? Because ESD damage is a real thing, especially on something that has, say, USB inputs, or uses fragile op-amps. We need to ensure our products don’t die or lock up when faced with static discharge.
- What will one set me back? On the low end, Bartek has a non-certified version that is about $250, but $10K is the price for a good, programmable hand-held device.
- Alternatives? Get some cats, balloons, fake sheepskin jacket and a slip-n-slide…no, just kidding. Get the Bartek.
Agilent Spectrum Analyzer. I’m afraid I don’t know much of the specifics on this one, but I do know that Dave picked up a spectrum analyzer to pre-qualify our stuff before it goes off for EMC testing (FCC). I know we haven’t had much trouble with the FCC since getting it.
USB Protocol Analyzer. Again, I don’t know much about this, but I know we have at least one of these for debugging stuff like Unison USB. It just shows how much more test gear we have to have these days.
HP Interval Analyzer. Another one I don’t know much about, but I know Dave and Mike use this to check jitter where it matters most—at the word clock.
Of course, there’s a ton of other stuff on and around my desk that is used as ancillary to measurement, like:
LCR Meter. One thing that most multimeters don’t do is measure inductance and capacitance. Hence, I have a cheap LCR meter. Cheap because I don’t use it much.
Beta tester. As in transistor beta. We don’t need to measure this often, but when we need it, it’s very handy.
Custom JFET matcher. Something we built to match SOT-23 JFETs, a small version of the one they use to match and re-reel parts for Nexus products. Because sometimes you need matched JFETs.
Starrett calipers. Because you can only get so far with 3D CAD.
Powerstat Variac. This is an ancient device that allows you to bring the power up on a new design slowly (variable-AC, get it?) Powerstats last forever, so you only need to buy one. Per tech, that is. Very useful if you don’t want things to instantly smoke when you botched something on the board. Hook up with an AC ammeter in series, slowly turn up…if it spikes, well…you messed something up.
And, of course, there is some gear we’ve retired (from the last chapter in 2015):
Stanford Research SR1a Audio Analyzer. Yep, we still have one of these guys kicking around. It’s a really neat audio analyzer in many ways, starting with the fact that you don’t need to hook it up to a PC. It no longer does anything that the APx series doesn’t do, so it’s really just a back-up now. We had another SR1, but we gave that to a guy who wanted to do audio measurements.
Tektronix 7603 Oscilloscope with 7A22 and 7A18 Modules. We used to use these for ground optimization, but it’s easier to do it on the APx now.
Fluke 179 Multimeter. We kinda stopped using battery-powered Flukes. Nothing terribly wrong with them, except the fact they always seem to be out of battery when you need it the most.
Seek Thermal Camera. Replaced by the Flir C2/3/5s. Better to have something that doesn’t plug into a phone.
3-Channel Power Supply. I usually go straight to board these days, no real need for lab supplies. Sometimes useful for finding PSRR problems, so it still sits around.
Weller WES51 Soldering Station, Yihua 898D Rework Station. Replaced by Metcal or Edsyn on the soldering side and whatever-is-in-stock-and-cheap on the hot air blowy side of things. Also not a measuring thing.
Now, that’s a hell of a lot of gear. I’m actually surprised we haven’t retired more of it.
But enough of the gear. How about the measurements?
The Measurements, APx Edition
Go to pretty much any of our products, and you’ll find:
- A summary of the most common measurements, plus dimensions, power consumption, etc.
- An APx report that gives you a lot more detailed info, including graphic plots of FFTs, frequency response, IMD, power output, linearity, and more. Some of these go into the many dozens of pages.
So why do we do both? It’s simple. The summary is for a quick overview of how the product performs, and the APx report is for people who want to go deeper.
But why do we measure what we measure? Let’s break down the measurements you expect to see on a piece of gear, discuss what meaning they have for us—and if there are correlations to audible differences.
So, here we go.
If you take a look at one of those APx reports I mentioned, you’ll see loads and loads of data.
First, there’ll be a summary of the Signal Path setup, which shows which connectors we are using (unbalanced RCA, balanced XLR, digital ASIO, digital optical, etc), output reference levels, input levels, etc. This lets you know how we’re testing.
From there, you’ll find a whole lot of data:
- Level and Gain
- DC Level
- Signal Analyzer (FFT Plot)
- Frequency Response
- Signal to Noise Ratio
- THD+N and Harmonic Breakdown
- IMD Level Sweep
- IMD Frequency Sweep
- Crosstalk
- Stepped Level Sweep (for amps, THD vs Output)
- Bandpass Level Sweep, AKA Linearity (DAC)
- Jitter Level Sweep (DAC)
Plus, this whole raft of tests are typically repeated at:
- Different gain levels
- Different outputs (balanced and SE)
- Different loads
Now you see why some of these reports are 70+ pages long!
Heck, the setup of these tests—even now that we have standard templates that we’ve established for typical product types—usually consumes an entire day per product. All the different levels need to be reset for the product’s specific gain, for the anticipated loads, for the type of product, for any special cases, and the prompts need to be changed so the instrument can pause and wait while you change a gain setting or load.
Then you have the tests. This also eats lots of time. Bad cables can screw up an entire measurement series. Delays caused by USB interfaces and ASIO or ASIO/WASAPI mean you may need to tweak settling time and and averaging to get good results. For extremely low distortion and noise products, just running a power cord too close to the test cables can bork the measurement. And then there’s operator error—maybe you didn’t put it in high gain at the right time, and the measurement series is shot. Doing APx test suites isn’t like pressing the “EASY” button—there are plenty of things that will trip you up.
Oh, and…you gotta do it several times, because you don’t want to take the chance you got a “golden sample.” There’s always some variation in product performance (usually very tiny, except in the case of tubes, and then with tubes about 99% of the variation is the tubes themselves), so when you’re doing a standard measurement suite, you should ensure you’re using a standard product, not a special hand-of-god one-off, or a half-working prototype that once had smoke come out of it, but “still works-ish.”
“Well, that’s cool and all, but what are all these measurements, and what do they mean?” you say. “You promised to tell us that, and to talk about if they correlate to audible differences.
Okay. Cool. I hear you. Here you go:
- Level and Gain. This is the level at which we take the measurement, like 1V RMS. A measurement of 1-2V RMS into 32 ohms is pretty common for headphone measurements (and pretty darn loud, actually!) Gain is simply the difference between input and output. 0dB gain = no gain = buffer.
- Why is level important? It’s important to specify the level, because most products have THD+N that looks like a J-curve, sloping down to a minimum level, then ramping back up as it nears clipping. If you measure one product at 1V and one at 4V, the 1V product may appear to have lower signal-to-noise ratio than the 4V product (where the 4V product has an inherent 12dB (!) advantage), and the 1V product may appear to have an advantage in THD over the 4V product, because it’s not working as hard into the load.
- Why is gain important? Because sometimes you need gain, and sometimes you don’t. If you have too much gain in a headphone system, you can end up with no volume control play—twitch the knob and it blasts you out of the chair. Headphone amps have -10 to +16dB gain (!) and speaker amps can be much higher (26-36dB).
- Fun fact: the higher the gain, the higher the noise. TANSTAAFL.
- Fun fact 2: the higher the gain, the lower the feedback, so the higher the distortion.
- DC Level. This is the level of DC at the output of the device being measured. Music, the stuff you’re listening to, is AC. DC is the stuff that, when it gets too large, becomes problematic—causing clicks and pops, and, in extreme cases, damaging drivers.
- Why is DC important? Because you should see a consistently low level of DC output from any device—yes, even those that use output coupling capacitors. Leaky capacitors can result in significant DC offset at the output. That’s why we include this sanity check in the AP test.
- Signal Analyzer (FFT Plot). If you know how to read it, the FFT plot is probably one of the most telling of the conventional measurements. It can help you understand how quiet a piece of gear is, and it can show you visually the distortion harmonics, which may correlate to some subjective effects.
- Why is an FFT important? Because it’ll help you tell how quiet a component is, and may give you some clues as to how it sounds.
- How do you tell how quiet a component is? Two ways.
- First way: Look at the FFT plot. See all the crazy spikes? Ignore them. Look at the level of the “grass” along the bottom. That’s indicative of thermal noise. At the FFT lengths we use, if the “grass” is at -140dB or below, that’s really, really quiet. At -120dB, you may hear some hiss on very sensitive IEMs. Please note that at different FFT lengths your results may differ.
- Second way: look at the FFT plot again. There’ll be a big spike near the middle, usually centered on 1 kHz. On the left side of the spike, you may see some smaller spikes. If the product is from a US manufacturer, and if it uses a linear power supply, the smaller spikes appear at 60Hz and its harmonics: 120, 180, 240, 300, etc…). These smaller spikes are power supply noise caused by a typical linear power supply.
- If those spikes start getting above -100dB, especially if they are 120Hz and above, and you’re not talking about a speaker power amp, you may hear some buzzzzzz. (Speaker power amps have more leeway…you are farther away from the speaker, and they have much larger power supply transients to contend with—a -90dB speaker amp will only have audible noise with your ear pressed to the speaker cone.)
- Fun fact: -75 dB is about the limit for 60Hz—it’s a lot harder to hear! That’s pure hummmmmm.
- What’s this about distortion harmonics and sound? Remember that big spike in the middle? Now look to the right. You’ll probably see some spikes out there, too. In the case of the 1 kHz sine, you may see something at 2 kHz, 3kHz, 4 kHz, and so on. Hopefully you don’t see much more than that. And hopefully the distortion is either (a) confined to second and third harmonics primarily), or (b) decreasing rapidly with increasing harmonics.
- Fun fact: tube amps have the (b) characteristic—they all show a strong, exponential decrease in distortion with increasing harmonics, which is due to their square law nature and typically low amount of feedback.
- Fun fact 2: this (b) characteristic may either be correlated with the “tube sound” some people blather on about, or it may be a hallmark of the types of topologies that produce this sound (typically low feedback, simple topologies, square-law devices).
- Frequency Response. This is the range of frequencies that the device can reproduce, usually expressed with a variance. For example, "20Hz-20kHz, +/- 0.1dB." A device with this specification can reproduce all frequencies from 20Hz to 20kHz with no more than a 0.2dB variance. Yes, 0.2dB, not 0.1dB. You saw the +/- there, right?
- Why is frequency response important? Only because people expect to see it. Otherwise it’s pretty useless. Read on.
- Fun fact: getting flat frequency response is the easiest thing you can do in electronics. All electronic audio equipment measures flat in the audio band. It probably shouldn’t even be measured. The only exception might be Valhalla 2. In that case, the bass -6dB frequency is defined by the RC filter of the coupling cap and the load, or about 7-8 Hz for a 32 ohm load.
- Fun fact 2: anyone calling an electronic product “rolled” or “sucked out” and implying it is caused by frequency response is probably a bit touched in the head. Gross frequency response issues are the exclusive domain of transducers. Oh, and equalizers. Like Loki.
- Aside: Kinda makes speakers with 45-18,000Hz +/- 3.5dB not look so hot, hmm? That means they could easily be down 7dB at 45 and 18kHz.
- Signal to Noise Ratio. The Signal to Noise Ratio of a product is easily the most highly correlated to actual audible differences…simply because a noisy product is, well, noisy. Plug sensitive headphones into it and listen to it hiss. Or hum. Or both. Typically expressed as a dB number below a reference, with weighting, like: "-102dB, referenced to 2V RMS, A-weighted."
- Why is Signal to Noise Ratio important? Because it correlates to noise floor, and a noisy product is something you’ll absolutely hear.
- Fun fact: Pay attention to the reference and the weighting, because that’s where the first number can be gamed. Big time. A product measured referenced to 4V output has a 12dB advantage over one measured at 1V. A product measured with A weighting downplays any hum or buzz issues it might have.
- Fun fact 2: It’s fairly easy to make a device that is 99.999% quiet—as in, plug in any insanely efficient headphone (or speaker) and it won’t hiss or hum at audible levels at any meaningful point on the volume knob. But of course there will always be 135dB efficient IEMs and people who turn the knob all the way up on high gain (even though running music through it at that level would destroy the IEM, and even though they’re probably hearing noise from the source (or, if the source is not plugged in, they are hearing ambient noise—use shorting plugs for real results)).
- Fun fact 3: signal to noise ratio is the most correlated spec to subjective perception. A noisy component will be noisy…you’ll hear it.
- Fun fact 4: but we’re talking hiiiiigghhhhhhhh levels—as in, maybe -65dB for a speaker power amp, or -95dB for a headphone amp. Those nasty hissy Class D monitors on your desk, the ones you can hear like a leaky tire all night long…yeah, think more like -45dB.
- THD+N and Harmonic Breakdown.If you can read the FFT plot above, this measurement won’t tell you anything more than you already know. However, if you want a single number, like 0.0002% or -118dB THD+N, then this gives you a single number. The harmonic breakdown is more useful, so you can see if, say, second harmonic distortion is -85 or -105 dB or whatever.
- Why is THD+N important? Mainly because people expect to see it. When it’s hard to tell the difference between two amps with 1000x different distortion levels in blind listening, maybe we’re measuring the wrong thing.
- Fun fact: comparing the infinite variation of FFT plots to a single number is kinda fun—there are many ways to arrive at, say, a 0.01% number. Is it noise? Is it distortion? At what level? What harmonics? Is the power supply buzz or hum? Well, maybe fun to an engineer.
- IMD Level Sweep. This traces intermodulation distortion (CCIF) versus level, so you can see how a product performs from low level to its full output. Intermodulation distortion is sometimes thought of as a more “realistic” measurement, because it uses multiple sines, and music can be reduced to multiple sines (lots and lots of them, dynamically changing, not constant, but hey, why get into the complexity of things.)
- Why is IMD important? It’s somewhat more like music, and it’s good to see how a product acts over a wide range of output.
- Fun fact: when IMD is expressed as a single number, it’s important to ask, “At what output?” “Into what load?” “What kind of IMD?”—CCIF measures 19kHz and 20kHz tones and looks for a resulting 1k tone, whereas SMPTE uses 60 Hz and 7kHz.
- Fun fact 2: That’s why we do a sweep. Then you can see how it performs at all levels—including low levels, where noise dominates anyway, just like THD.
- IMD Frequency Sweep. This shows intermodulation distortion (again, CCIF) versus frequency across the audio band. Most good designs will show relatively flat IMD versus frequency.
- Crosstalk. This is how much of one channel bleeds over into another. You’ll usually see these numbers expressed as “-XXdB over Y-Z range,” like: "-68dB from 20 Hz to 20 kHz."
- Why is Crosstalk important? Only because people expect to see it. It’s like frequency response or THD+N.
- Fun fact: crosstalk is wayyyyyyyyyyyyyyyyy overrated. Some of the best stereo imaging can be had from turntables, with cartridges that have 25-30dB of channel separation. 25 dB down on one side? That’s “off.” As in, you’ll never hear it. We shoot for -70dB or so in headphone amps, which have to driver 32 ohm loads—and we rate it at the load. -120dB is easy for DACs.
- Stepped Level Sweep. This is a fun one. This actually shows the THD+N at various levels up to clipping for amplifiers, so you can see how it behaves across its entire working range. All amplifiers show a J-shaped plot, dropping to a minimum and then rising rapidly as they hit clipping. This plot shows you a ton of information, including what the noise floor is like, and how much power the amp puts out into various loads.
- Why is a Stepped Level Sweep important? Because it shows you how an amplifier works at various output levels, and verifies its output power.
- What’s the noise floor like? All amps have a J-shaped THD+N curve because noise dominates at low levels. Distortion doesn’t matter when an amp is putting out only a small amount of power, but noise does. If the amp has a high noise floor, the THD+N curve will be offset upwards when compared to quieter amps.
- How much power does the amp put out? Look at where the ascender of the “J” curve really takes off and goes to the sky. Look at the 1% level. Trace it over to the voltage output. 1% THD is pretty much the standard for maximum output.
- Fun fact: if the chart is in volts, like ours, use the formula P= V2/R, where V is the RMS voltage output, and R is the load resistance in ohms.
- Funner fact: if you’re looking at other charts, make sure they’re in Vrms and not Vpeak or Vpeak-peak, which will give very different results (much higher!)
- Aside: this is also another chart that may allow you to get a better idea of how a component sounds. Does it have a high noise floor? Does it clip cleanly? Does it have a smoothly rising distortion characteristic that makes the onset of clipping difficult to determine? All of these have a sound—the last of the three is typical of tube amps.
- Bandpass Level Sweep, AKA Linearity (DAC). This shows the output of a DAC from lowest-level signals (we start at -140dB) to highest. The ideal output is a straight line, with no deviation across all levels. A straight line is difficult to achieve at low levels due to noise and nonlinearity in the D/A converter itself.
- Why is a Bandpass Level Sweep important? Mainly to see how linear a DAC is, but we feel its importance is frequently blown out of proportion.
- Fun fact: dither is important to linearity. A Modi Multibit without dither begins deviating significantly from a straight line by -90dB. With dither, it’s -110dB. With the same DAC. The same 16-bit DAC. Yes, that’s higher linearity than its inherent resolution. Dither is magic. Well, not really.
- Jitter Level Sweep (DAC). So, there are a ton of ways to test “jitter” in an audio interface, and honestly, we think most of them are bunk. We include a jitter level sweep from the standardized AP test roster because, well, it is part of the standardized AP test roster that checks how measured performance is degraded by increasing jitter. It’s not a JTest or a single number (arrived at where and how?), but it is a standard test from the leader in audio test and measurement. So we include it.
- Why is a Jitter Level Sweep important? Heh, we don’t know. We run it because it’s a standardized APx test. (It may also show if jitter is going to cause gross problems, but we have yet to see a component that exhibits any.)
- Fun fact: during development, we concentrate on jitter measurements at the word clock, where it matters the most, using an interval analyzer. This is how we’ve always done it, since Mike was working on digital before Theta Digital was Theta Digital (you guys know Mike is one of the reasons we talk about jitter at all, right?)
Whew. That’s an insane amount of measurements, right?
Well wait, there’s more. In addition to the APx test results, we also specify:
- Output Impedance. This is most important in power amps, where a lower output impedance is better. Output impedance is expressed in ohms.
- Why is output impedance important? Because it gives you an idea how the amp may interact with a speaker or headphone. Unless you are a believer in “current output” amps, you typically want low output impedance—much lower than your speaker or headphone.
- Fun fact: 0.05 ohms and a “damping factor of 160 into an 8 ohm load” are the same measurements. Yeah. Blame old guys.
- Fun fact 2: It’s important that power amps be low output impedance so they do not affect the frequency response of the headphones or speakers they are powering (transducer impedance varies with frequency, whereas for amps and preamps it should not—or vary predictably due to a Zobel network…like I said, this gets deep really fast…)
- Fun fact 3: It’s important for preamps and source components to have a stated output impedance that is not too high, or long cable runs may be problematic.
- Fun fact 4: More feedback = lower output impedance. Woohoo!
- Fun fact 5: Output impedance can easily correlate to audible differences. If you use a high-output-impedance headphone amp with multi-driver IEMs, you’re going to run into gross frequency response problems (because they use crossovers). The same thing happens to a lesser extent with dynamic headphones that have impedance variations at certain frequencies. And the same thing happens with speakers. In addition, source components with too high of an output impedance and long, highly capacitive cables can roll off high frequencies.
- Maximum Power. This is how much power an amplifier can put out, usually expressed in watts RMS into a specified load, or multiple specified loads. For example, “1.5W into 32 ohms,” or “60W into 8 ohms and 100W into 4 ohms.” We rate maximum power at 1% THD for solid-state amps and 10% for tube amps.
- Why is Maximum Power important? Because it’s a good way to tell, at a glance, if the product will drive your headphones or speakers well.
- Fun fact: At one point, the Federal Trade Commission regulated how power output was to be measured, since the amp companies were gaming the numbers so much. They don’t pay very much attention to that today. But here’s what you’re looking for: power in RMS watts per channel, all channels driven, into a specified load.
- Full-Scale Output. This is a simple measurement for source components, typically DACs, specifying how much voltage it puts out for a full-scale signal (0dB). You’ll usually see this in terms of RMS voltage, like “2.0V RMS.”
- Why is Full Scale Output important? To ensure you have enough output to drive your system all the way to its maximum output—especially important if you are using, say, a passive preamp.
- Fun fact: The consumer standards, such that they are, are 2.0V RMS for single-ended sources, and 4.0V RMS from balanced sources.
- Power Consumption. This is a simple measurement of how much power a product uses. Well, usually simple. In the case of speaker power amps, the FTC mandates measurement at full output into the lowest specified load.
- Size and Weight. Yep, these are specs too. We take them. We provide them. No, they don’t correlate with how something sounds.
Quite a list, right? Well, not really. In fact, even with a book full of APx results, that’s really only scratching the surface.
All Measurements Are Belong To Us
Here’s the thing: measurement is an ongoing process, from the first breadboards to “naked” prototype PC boards to production qualifiers in chassis to production itself. I’ll give you a run-down of some of what we measure on the way to production, but before that, here’s an important question to ask:
Is this a “measurement” design or not?
As in, is this design aiming to please those who want exceptional performance with respect to the limited suite of conventional, steady-state measurements?
Or is it a design that isn’t governed by these rules?
Or a design that can’t meet these goals at all?
“Blasphemy!” some a crying now. “This is a chapter about measurements! Aren’t you implying they are the be-all and end-all of audio design?
No. Go back. Read the opening to this chapter again.
Or read the FAQ of many of our products, where we’ll say things like “yeah, we know Modi measures better, but we prefer Modi Multibit, and yeah, maybe you think we’re a bit touched in the head for this opinion, but neener neener don’t care.” Or, “we think there’s more than the number of zeros you can stack on the back of that 1 kHz sine wave measurement,” or “we persist in doing things like True Multibit and fully discrete topologies like Nexus and non-class-D-ten-billion-dB-feedback designs like Continuity because we think they have a place in the market, they are more technically challenging, and they sound better.”
Yes. Blasphemous words in some circles. But there you go.
So, step 0 in measurements is to decide how important they are to the design. In something that’s aiming at very low measured distortion and noise, measurements should be front and center, the first thing you start with. In other stuff, especially tube gear where there is little chance of “state of the art” measurements, it matters less.
Fun fact: I actually prohibited myself from measuring Folkvangr on the APx until the design was finalized. Because I knew it would measure pretty “bad.” Because, like Vali 2+, it’s a “measurements schmeasurements” design. Because it sounded just so, well, amazing and glorious, that I really didn’t care.
But I’m getting off track. Because Folkvangr and Vali 2+ were both measured plenty—just not on the APx. So let’s look at the process of measurements, from concept to production:
Early Stage Measurements. We’re talking breadboards and naked PC board prototype stage here. This is when we’re either just playing around (on breadboards) or trying something that may or may not end up being a product (on a prototype PC board, usually without a chassis, usually green rather than red…we play with quite a few things that never become products.
At this point, what do we look for?
- Gross problems. Whenever you start up a new design, the question is, “Will it work?” Or, in blunt terms: “Is this gonna catch on fire?” When we start up something for the first time, we’re usually using a variac to bring it up slow, looking at it with a FLIR to see if it’s getting too hot, and checking the output on a scope to see if it’s oscillating or unstable. These are super basic measurements that say, “Should we proceed farther, or shut it down.” If we assembled the prototype correctly, and if the design isn’t completely borked, then we should be able to get a clean sine wave out of the device, check out a square wave to see how close we are on the compensation, and proceed on to the next steps. If not, well…power down, stare at the board, fix it, and start again.
- Gain/phase margin. If you wondered why something might be a power oscillator rather than a power amplifier, an inadequate gain/phase margin is why. A Bode plot is a measurement that that determines how stable a feedback amplifier design will be. If you have inadequate gain/phase margin, you need to change the compensation of your amplifier (or, in the case of a current feedback design, you may need to alter the feedback impedance to decrease the bandwidth of the amp.)
- Open loop gain, THD, frequency response. These are the same as the gain, THD, and frequency response measurements outlined in the sections above, but applied to the gain stage without feedback, or “open loop.” Since we do primarily discrete designs, and primarily designs that have open-loop bandwidth greater than 20-20kHz, it’s important for us to characterize the designs in an open-loop state. This helps us optimize them for their particular usage. An amp that is more linear open-loop is typically better than one that needs to be “hammered flat” with tons of feedback, in our opinion.
- Clipping characteristics and maximum output. Okay, so it looks stable? Let’s add a load and run it all the way up to clipping. When an amplifier clips, it should do so cleanly and with little drama. If it doesn’t, more work is needed.
- THD characteristics. Now you have a clean-clipping amp. Now it’s time to look at the THD characteristics—not just single-number performance. Does it have low levels of high harmonics? Does the THD level decrease with frequency? Does it perform pretty much as expected? Amps can be notably sensitive to PCB layout, compensation, biasing…and when you throw in techniques like Continuity™, the opportunity to tweak distortion characteristics gets much greater.
- Maximum power output. Does it meet our expected specs? If not, why not? Is the transformer sagging, or are we losing too many volts in the bridge and output stage resistors? Some tweaking may be needed to achieve the target output.
- Noise performance. What’s the noise floor like? Is it as expected? Is power supply switching under control, or are there excessive peaks at 60Hz multiples? Is thermal noise acceptable for the design? There’s a big difference between the acceptable noise floor for a phono preamp and a 100W RMS speaker amp.
- Thermal performance. What’s getting warm on the board? This is where the FLIR comes in. It’s dead easy to see what’s staying within specs, and what’s getting dangerously hot—even if it’s a tiny SOT-23 part we miscalculated on. It’s also a great way to measure hands-off temperature of heatsinks and transformers. A shocking amount of engineering involves simply getting rid of heat, so it’s critical we stay on top of it.
- Input performance (digital). In addition to the above tests, digital devices get verified for different input frequencies and levels, as well as different levels of degradation of the input signal, to make sure they lock to the signals we specify.
- Jitter performance (digital). Also on the digital side, we’ll typically take a look at jitter performance once everything is on a PC board.
Qualifying Measurements. Once we get past the naked board measurements, we get into stuff that’s intended for production. So that means we’re now measuring boards in chassis…hopefully final boards, but the measurements let us know just how final they are.
This is where we repeat everything above, and pay more attention to the actual numbers across the board (including IMD, output impedance, and other stuff I didn’t mention above.) We’ll measure 4-6 prototypes and see if they are all similar, or if they’re all over the map. If they’re similar, that’s good, and we’re on the path to production.
Fun fact: in qualifying measurements, Nexus™ wasn’t similar. They were all a box of chocolates. Some were great. Some were the equivalent of a rat-turd-and-acetone bon-bon. That meant they weren’t ready for production. I had to go back and investigate why they were different—and, in the case of Nexus, they were different due to the parts not matching. That led to a whole raft of prototypes to explore what parts had to be matched, and how closely, in order to achieve consistent production. And that’s what led us to tighter matching in production than virtually any other audio product ever done before—and, uh, by the way, matching is measurement, too…
And there are a few more measurements:
- Multiple qualification. Yeah, the APx is the leader in audio measurement, but it’s nice to get confirmation on the Avermetrics as well. And it’s good to take a look on the GHz scopes to make sure there isn’t any parasitic oscillation that’s invisible in the audio band.
- Stress tests. This is also where we get to torture the gear—short the outputs, blast it with the ESD gun, stress the protection system with transient loads, and see if the products survive. It is also where we’ll run tube designs for a few weeks and re-measure tube characteristics to make sure there’s nothing unexpected going on.
- Custom tests. I mentioned matching Nexus parts as one custom test, but there are tons of custom tests we come up with, including some very sophisticated firmware tests. Ragnarok, Aegir, and Vidar all have “OBD2 reader” automated test fixtures, with a small display that shows problems with the board based on on-board microprocessor diagnostics.
Production Tests. And here’s where we do it all again—starting with the first articles of the first run, which is usually where we do the official “APx Test Report” for that product. We also create any specific in-line testing for each product. Some products are assembled and listened to, some products go through automated testing and listened to, and some products are burned-in and listened to later.
“What? You don’t run everything through the APx and run a report for each?”
No. Of course not. That would take wayyy too much time. How do you feel about a $199 Magni? Yeah, thought so.
However, different products have different needs.
And, you didn’t read the common thing with every product: every product is listened to.
As in, a $49 SYS is listened to. Same as a $1499 Ragnarok.
Yes, every single product we make is listened to.
This in itself is a test.
“Wait a sec!” some are saying. “That’s just listening! How can it be a test?”
Well, in some ways it’s the best test. An APx can’t find a scratchy volume pot. It can’t hear a pop when you flick a switch. It can’t say, “Hmm, that button doesn’t feel right.” Listening tests catch that. Because a proper listening test, like ours, checks every function and mode of the device.
Beyond listening, though, some products need automated testing. Every Mani is checked against an RIAA reference curve on our Avermetrics. Every Loki is checked against its frequency response reference curves on an Avermetrics. Every Modi Multibit and every Multibit DAC Card is checked for linearity on an Avermetrics (because these products have parallel-input DACs—if you lose one bit, you may not hear it, but a linearity test will catch it.)
Beyond that, production has its own set of measurements we need to do…measurements that cross over into the actual production process.
- Programming and verification. Lots of our stuff needs to have firmware installed, and the firmware needs to be verified. No testing can be done before the firmware is installed and verified.
- Biasing. Some of our products need manual biasing of the output stage, like Aegir, Vidar, and Ragnarok 2. This means we actually have to adjust potentiometers and measure voltages across an output resistor to set the bias…and then verify the rest of the measurements.
- Tube Matching. Some products use tubes that need to be matched. We do this in-circuit for the specific product the tubes are used for.
Measurements Schmeasurements
Last time I did this chapter, I said that we did our own multitone test, because it seemed to correlate better to perceived sonics. Now, there are tons of multitone tests out there, but most of them seem to be a blunt instrument—as in, let’s equally space the tones and run them at 0dB.
One problem: there’s exactly no music like that in the world.
Another problem: it’s also still steady-state tones.
A third problem: how do you interpret the results in terms of human perception?
Here’s the deal: we don’t really use our own multitone test anymore, because even if it did correlate a bit better with perception, the question is always: “Whose perception?”
I mean, Mike and Dave and I all differ in our sonic preferences. How does a harmonic spray translate to what we like?
Answer:
it doesn’t.
And that leads us to the final nail. Which is, even if the new 2025 model APx5555 had a “sonic quality measure” which displayed a number like “99.9% sonic quality,” or “78.3% sonic quality,” which set of ears determined that specific “quality?”
Mine?
Yours?
An untrained panel of listeners?
A trained panel of listeners?
If it’s not you, do they agree with you?
What if you have different sonic preferences?”
If you have different sonic preferences than the masses, is that “wrong?”
You start to see the problem now. No matter how we try to reduce sonic perfection to a number, the listener is always in the equation. And there we go, 100% back at subjectivity.
Sigh.
Yes, sigh.
Because it would be super-convenient if there was a single and absolute arbiter of quality. And it would be wonderful if sonic impressions lined up perfectly with conventional test metrics. But they don’t (seriously, try level-matched blind listening of a Vali 2+ and a Magni Heresy—one amp with 1000x the distortion of the other, and see what you think).
So does this mean measurements are useless?
LOL. No. Go back and read the chapter. We wouldn’t have invested literally hundreds of thousands of dollars in measurement gear and have it deeply embedded throughout our development process if it had no value. It’s just the measurements we make are much deeper and broader than one number on an APx. And they don’t lend themselves to oversimplification.
Let me pontificate for a moment.
Here’s the deal, as I see it. There are three possibilities:
- We really can’t hear the difference between any adequately-measuring gear, and we’re all fooling ourselves. The problem with this is that “adequately-measuring” may include gear that is really, really bad by today’s state-of-the-art standards. If it’s hard to hear the difference between -50dB THD+N (Vali 2) and -115dB THD+N (Magni Heresy), then “adequate” is much broader than we think.
- We are measuring the wrong things, and there is some as-yet-unknown magic measurement that can correlate with what we hear. The problem here is who is “we.” Is it you? Is it a panel? Are you weird and want something different than the mean? Is the mean gamed? Oh yeah and the process of defining such a measurement in the first place.
- There are small differences that aren’t well-defined by the standard suite of audio measurements, some people can hear these differences, and some people find them meaningful. The problem with this is it means there are no easy answers, and it all comes down to the wondrous spectrum of human capability and personal preference. There is no number. There is no capital-T Truth. There is just oeey gooey subjective experience, and that’s icky and messy and gross.
I think (3) above might be able to be explored by a blind-listening type of test, done properly with all the controls in place, but unfortunately that’s the bastion of real reseach, and, even if we could fund it, I don’t know how to get the ball rolling. Maybe that’s something for the future. We’ll see.
Whew.
If you were to take away some things from this epic-length screed, I’d say:
- Measurements are important for product development, but there are many more measurements than THD+N and frequency response.
- If you want to do high-resolution conventional measurements, you don’t need to spend $30K on an APx555—you can get very useful FFT plots from $250 recording interfaces and free software. There are also other pro options besides AP.
- Some designs may have different goals than minimizing THD+N on an APx…kind of like a Mercedes S-class and a Porsche 911 have radically different design goals. A 911 is a really schiity S-class and vice-versa. That’s because the goals are different.
- It’s OK to have goals other than pure measurements (gasp!). Yes. Seriously.
- We all may be fooling ourselves (see How We Fool Ourselves) or we may all be our own arbiters of perfection.
With that, I’ll close this chapter the same way I closed V1:
That’s why we still listen. And measure. And come up with new measurements. And listen again.
And I’ll leave it at that.
