2017, Chapter 14
Slouching Towards Kaizen
Don’t know what Kaizen is? Look it up.
No, seriously. You’re probably sitting in front of a machine connected to a significant part of the sum total of human knowledge right now. Either that, or you’re holding it in your hand. In any case, there’s no excuse for 8-year-old-in-1978 levels of huffing and groanery when you’re confronted with a word you don’t know.
“Okay, fine, I looked it up,” you say, moaning over the clunkiness of multitasking in iOS 11. “Japanese yada yada, making schiit better, what’s that got to do with anything?”
Well, hold onto that thought about Apple for a sec, because they provide a couple of really great examples of what this chapter is about.
Example the first: ask Apple if they’re developing a new iPhone. They’ll look at you with an expression like they’re talking to a 4-year-old with a learning disability. And, if they are being honest, they’ll say something like, “Well duh. Always.”
In short,
of course they are working on a better iPhone.
Continuously. Every day.
And,
of course, a better iPhone is coming, based on what they develop.
Everybody knows this, and nobody (well, almost nobody) gets their panties in a twist about it. Which makes it kinda ironic when a whole lot of people flip their lids when there’s even a
tiny hint of a suggestion that Schiit might be improving our products.
OMG, did Schiit change something about the DACs?
EEK, am I getting the latest one?
AH! Did they swap some parts in production to make it sound better?
What if they introduce something better after I buy mine? NOOO!
Dummies! Didn’t they ever try XYZ (feature, part) before they released this?
And so on.
Well, here’s a hint: the answer to all of the above is “yes.”
Sometimes things change because we can’t avoid the change. Sometimes things change because we think it’ll make the product better (and the risk is low). Sometimes things change because we don’t think they matter. And we’re always developing something better, so yes,
eventually something better
will exist after you buy it. And we try lots of crazy things during development, but not all of them make it into production.
Don’t worry, I’ll get into all the above in detail—with examples—shortly. But that last sentence about development brings me to the next Apple example.
Example the second: So, yeah, you accept that Apple is always working on a better iPhone. This means they do lots of development work.
And so, during their development, someone sold them on the idea they could do a fingerprint reader under the screen on the new iPhone X. And, it’s totally obvious that feature didn’t work. Like, not at all.
So what did they do?
They said, “Hell with it, people want a new phone this fall, we got face ID, we’re good.”
Yep.
Just leave the fingerprint reader off. Screw it, not worth messing with.
(Need I remind you that this is the most valuable company in the world—or maybe the second, I don’t really care—and this is their best answer.)
And, in delicious irony, this amazing failure will probably rewrite the book on what people expect from a high-end smartphone over the next few years. Fingerprint sensor? Naaaahh, that’s so 2016, broh! You need the super-duper laser-shiny soul-sucky face recognizer now!
(And please, let me hammer this home: fingerprint sensors will be dead like fried chicken in a couple of years due to a
failure in development.)
And that’s the reality. You try a bunch of things. Sometimes they don’t work out. And sometimes you’re big enough to, well, just change the whole market.
We ain’t that. But we try lots of things. And things do change. And we want to make things better. So let’s talk about all that.
“If It Ain’t Broke…”
Before I get into the whole “how things change,” deal, I should probably give you a little background on modern manufacturing. Or antique manufacturing, because it’s been like this for a looonngg time.
In manufacturing, you typically want to avoid change. You want everything you make to be exactly the same. This makes sense for tons and tons of reasons, the top 3 of which are:
- It’s most efficient, because your line doesn’t have to learn a bunch of variations, and the line isn’t stopped or slowed down by random custom versions.
- It lowers costs, because you’re ordering lots and lots of the same parts, over and over again.
- It’s best for service, because you don’t have to wonder which version you’re working on—change is hell on support.
And, a whole lot of what we do is based on avoiding change. Whether it’s re-doing the setup on the robotic pick-and-place machines that do the board assembly, or re-training the people on the line, change isn’t necessarily a good thing.
To avoid change, we have a Bill of Materials. This lists every part in a product, where it is, what the manufacturer’s part number is, where we get it, acceptable alternates, and whether or not it’s a critical part that needs to be reviewed if it needs to be changed.
“Hold on a sec,” you might be saying. “What are those last two? Acceptable alternates? Critical parts? That sounds like you’re planning for change.”
Well, yes, we are. And, in fact, it gets even crazier. Because there are a whole lot of parts on the board that we consider noncritical, and therefore we don’t care where they come from. Those can be changed at will. Our distributors can substitute any part they want, as long as it’s functionally the same as what we specified.
“Oh, hey, I don’t like that,” you say. “That’s a little, uh, sloppy, don’t you think? Wouldn’t it be better to have everything 100% nailed down?”
In short, no.
Why? Because you don’t want to wait 6 months to get your Magni 3.
Stay with me here for a sec. If we specified every single part, down to the resistor that feeds the LED on the front panel and the capacitor that forms the timer for the relay mute, we’d go bonkers trying to get those exact parts. Because, a fact of manufacturing life is…schtuff goes out of stock. Sometimes unpredictably. You need to be able to roll with the punches.
And (I’ll repeat this a bunch of times), this is how
every single manufacturer works. Period. If they say different, they’re lying.
Aside: Now, before you go thinking we’re all evil and schiit, let me say this: we do specify an amazing amount of parts by manufacturer or type, right down to the thin-film SMD resistors used on Magni 3. This is quite a big bag of crazy in a $99 product. Plus, see below.
Aaaanddd…that’s why we have “acceptable alternates,” and “critical parts.”
“Acceptable alternates” mean they’re really the same thing. This happens when we consider, say, filter caps by Nichicon or Rubycon or United Chemi-Con interchangeable. (And we do.) It also happens when multiple manufacturers make the same active components. (They do.) If we know that something is the same, or functionally the same, then we may put an “acceptable alternate” on the BOM.
“Critical parts” means “if you can’t get this specific part, contact Mike or Jason to see how to deal with it.” Not being able to get a specific part can happen for a few different reasons, most of which are bad.
- The part is out of stock. This is least bad, because other distributors might have it, or it might show up in time for the production run. And, worst case, there are usually dicksmokers (er, I mean alternate distributors) who load up on parts and then scalp the crap out of you (er, I mean charge an adjusted market price) where you can find stock.
- The manufacturing plant got taken out by tsunami, earthquare, hurricane, or fire. No, I’m not making this up, sadly. And it happens more often than you might think. But the reality is, natural disasters can upset the flow of critical parts. If you’re particularly unlucky, the natural disaster has hit something like “electrolytic capacitor city” where 80% of the world’s electrolytic capacitors are made, and then things get really interesting. But, as with the above, this will eventually pass.
- The part has gone obsolete (and stock is gone.) I had to add the “and stock is gone” part because there are a lot of obsolete parts that exist in a hazy half-life for 5-10 years after they go obsolete. Either they have warehouses full of parts, or the manufacturer will half-heartedly make another run or two when the orders get big enough. But when the part is well and truly gone, you’re well and truly boned. Especially if you designed in something pretty special.
“Oh, but parts going obsolete can’t happen that often!” you cry.
Yeah. Uh-huh. I guess that’s why the output devices used in Magni 2 went obsolete (and gone) twice in 3 years.
And that’s a great segue to…
Surprise! Dealing With Unexpected Change
During the production of a product, when everything is humming along nicely, and everything is coming off the line just fine, sometimes I get nervous. Because I know that it’s exactly at that moment that Alex is likely to step into my office, looking somber, and announce:
“Ah, we might have a problem.”
That problem can be lots of things, but it usually has to do with parts shortages. Most of these shortages can be dealt with by specifying alternates (and, worst case, testing them to make sure they really really are the same—I did this recently for the diodes we use to bias Jotunheim.)
Aside: some “obsoletes” are mostly BS. The manufacturer may change the production process slightly, and, in doing so, also changed the part number. The part itself is really no different, but complies with some obscure new regulation about the percentage of rat feces allowable in the composite material. However, this means we still have to check them. Thank you, legislatosauruses of the world.
But sometimes they’re real. Which brings me to two cases in point, one analog and one digital.
Case in point: Magni 2 and Uber. Both of these products started their (production) lives using the same output devices as the original Magni. These D-Pak sized parts were nice, high-beta, linear, inexpensive, and readily available.
That is, until they weren’t. Available, that is.
This was one of the most sudden instances of unavailability that I can remember. One day they were there, one day they weren’t. I didn’t think it was a huge deal, so I switched to an “audio” D-pack pair that was very popular with companies making gear costing a whole lot more than ours. After testing a prototype to make sure it didn’t oscillate and tested within our specs, we went to that pair of parts.
End of problem, right? Well, yeah, until those parts went obsolete too, about 6 months after the first swap. Argh.
Actually, that was OK by me, because the “audio” parts were, well, kinda poopy. In circuit, they met the specs we set for the product, but they had lower beta, and were slightly more nonlinear.
I’d actually already been looking for alternate parts, seeing what was available in D-pak format with highly linear beta, high beta, high ft, and complementary pairs. I’d been testing a couple of 2SC/2SA parts which were a whole lot better than both the original outputs and the “audio” pairs.
But these parts had a twist: they didn’t just drop in and work. (Which is weird, BJTs should be pretty much interchangeable, right? Some days I wake up and think, “I got this.” Other days this schiit is 100% voodoo.)
Oh no, this one wasn’t going to be so easy. The compensation was fine for these parts, but they wanted to thermally run away, where the previous parts hadn’t had a problem. Which meant we had to change how we biased them—from resistor to diode bias.
And so, after making this fairly significant change, and verifying that the end result was thermally stable and met our published specs, we put the third type of output device into production. And those outputs continue into Magni 3. And lots of other places.
Aside: it turns out the Second Magni Output Debacle wasn’t a “real obsolete,” but actually some process-change BS. Oh well, the new parts are much, much better. So much better we use them damn near everywhere now. Hopefully they’ll be around for the long run.
Aside 2: I could write a whole chapter on how to pick parts from datasheets and curve testing, because there are a ton of parts that are great for audio that aren’t billed as being for audio at all. A lot of the whinging about how “there ain’t no great parts anymore” comes from people who don’t look past the audio parts, and/or will not consider surface mount parts. The reality is that there are plenty of parts that are far better than the “golden age” NOS devices, if you’re willing to look. Well, with the exception of P-channel JFETs, maybe.
And so, with no grand reveal, with no “3 series” moniker, with no outside change at all, Magni 2 went through two fairly big output changes.
“But which one sounds better?” someone cries. “Which one did I get?”
No clue. I don’t know the answer to either question.
“Why not? You’re unfeeling monsters, teasing us like that!” someone might ask.
Because
it doesn’t matter. This is the kind of thing that happens all the time. Things change. And we deal with it. We make sure it meets the specs, listen to it to make sure it’s as good or better than the old version, and it goes into production. It doesn’t matter which version you got, because there was no option to get the old version. That bit of history is past. It will never happen ever again.
Again, let me repeat:
this is no different than any other manufacturer. If they protest that this isn’t so, they’re lying. Or making products that cost $150K and require an 18-month preorder. Or they’re having everything contract manufactured in China and they have no friggin idea what kind of parts are in their product.
Case in point: Modi 2 and Uber. Whaaaaaa? Current products now, with changes? Yes, and yes. Modi 2 and Uber followed in the footsteps of their forebear, Modi, and used the AK4396 DAC. This was a great DAC—nice, linear, inexpensive, pleasant-sounding, and clean.
That is, until AKM killed it.
Shortly after we started production on Modi 2 and Uber, we learned about a new range of AKM DACs coming to market—first, the AK4495, which Mike kinda sniffed at and declared “not good enough for the money, I’ma gonna finish these here multibits, OK.”
Then, the AK4490. This one actually seemed to be worth looking at, plus it was inexpensive. Still, we didn’t do much with it, until one day when we were looking through the AKM site and noted that the AK4396 was “not recommended for new designs,” or NRND.
NRND can be bad, or it can be meaningless. NRND parts can be made for another decade. Or they can go away real fast.
Yeah. Guess which kind the AK4396 was.
Oh yeah. Next thing we knew, the AK4396 started showing up as “obsolete” at our distributors, though they still had plenty of stock. Alex stepped in and placed a couple of very large orders to ensure we had enough stock for about a year, and Mike and Dave started putting the AK4490 into Bifrost…and Modi 2 and Uber prototypes.
The good news was that the AK4490 was not a bad part. It could have been. Then we would have been stuck with a real problem, because changing to another manufacturer’s DAC would be much more profoundly disruptive.
The bad news was that the AK4490 was not 100% an upgrade. It was better in many ways than the AK4396, but some people liked the older DAC’s dryer, just-the-facts presentation better. I always thought the AK4490 was the winner, but I could definitely understand that it was a personal thing.
So, in the end, what did we do? We did a running change to the AK4490 on the Modis—first on the Uber, then on the 2, without any announcement.
Why no announcement? Why no “Modi 3?” Go two paragraphs up. Like I said, it was shades of gray. Although the AK4490 was the current darling of the delta-sigma DAC world at the time it went into Modi 2—being used in products fully 30x the price—it was not head-and-shoulders above the AK4396.
Was it as good as the AK4396? In my opinion, and in Mike’s opinion, yes. Was it better? In my opinion, yes. In Mike’s opinion, it was tradeoffs.
Which meant it was the same product, not a “3.”
Which meant it would be best to minimize nervosa and not trumpet the change (despite it being the darling DAC of the year).
So we didn’t.
Aside: and to be perfectly clear, we’re talking about $99 and $149 products here. Come on, be realistic.
So which one did you get? No clue.
Which one sounds better? To me, the AK4490. To others, maybe not.
I can absolutely say they all met the specs we provided, with no interim changes. In fact, the AK4490 measures a bit better, if that’s important to you.
And again, this is
how manufacturing works. Everywhere.
Change for the Better (We Hope)
Of course, not all production-line change is caused by “oh schiit” moments. Sometimes we see an opportunity to improve a product, and we take it.
No. Wait. That’s over-simplifying.
It’s more accurate to say, “We see something that looks like it might be an opportunity to improve a product, so we built a few that way (and maybe some other ways), test them, listen to them, discuss with other people like Alex about the ramifications to production if we make the change, and then, if everything looks good, and the risk looks low, we make a change for the better.”
Soooo….maybe, “Tiptoeing towards kaizen,” or something to that effect.
Why would we violate the old rule of “if it ain’t broke, don’t fix it?” It’s simple: there IS progress in the world. Parts can get better. Good parts can get cheaper. New, superior parts can appear. And we can learn a thing or two about our own products, stuff you can only learn after making a few thousands, or tens of thousands, of them.
And yes, I know, this is 100% contrary to some who will whine and carry on about how we’re past “the golden age of audio parts,” while desperately clinging to their expensive, inconvenient, and unavailable NOS through-hole components. While I’ll agree with them 100% that if you look only at active parts that are billed as “for audio use,” yeah, there’s a good deal of suckitude in the current catalog, you have to look beyond that. Most of the best actives are not billed as being suitable for audio. You need to look at the curves (and verify them yourself), and you’ll find some parts that are far better than any “golden age” device.
Aside: much of the magic of the “golden age” came from relatively crappy parts, which necessitated obsessive testing. When you hand-match outputs on a curve tracer (because their process control was, well, beyond crap in today’s terms), you can get really nice results. But this is a hell of a labor-intensive thing to do. Don’t expect to see us doing that any time soon (though we do have a solution, based on modern technology and techniques, coming—more on that after the end of the year.)
And when you get past the actives, yeah, the golden age can go bite a donut. Because you didn’t have easy (and cheap) access to stuff like 0.5% thin-film resistors and 2% film capacitors. Per. I. Od.
Now, things like 0.5% thin-film resistors are less than $0.01. This is a relatively recent development, and that brings me to the first example of us screwing with the “if it ain’t broke, don’t fix it,” mantra.
Case the first: 0.5% thin-film resistors. When I did design work at Sumo, we chose the 1% metal-film resistors we used carefully. They were more expensive than the 5% carbon-film products, and we were making inexpensive products. But because there’s progress in the world, we used nothing but 1% film resistors at Schiit from the get-go.
When we started in the garage, a lot of them were thru-hole parts. When we began doing significant numbers of surface-mount products (Magni and Modi), though, things changed. Yes, we were still using 1% film resistors, but they were thick-film resistors, not thin-film resistors. Thin-film is a better process, with superior temperature and voltage coefficients. But thin-film, at the intro of Modi 1, cost about 15x more than thick-film. So Mike would brag about using “a couple of thin-film resistors” in critical places. We were very picky about where we used them. It was a fairly big deal.
But in the last 5 years, the whole resistor market has changed. First, thin-film got about 3x cheaper. So, when we designed new products, we used more of them. Then, the bottom fell out of the thick-film resistor market, and new thin-film resistors appeared at about the same price we’d been paying for thick film. Suddenly, we could use thin-film everywhere.
The problem? We had tons of products—and tons of BOMs—that were a mix of thick- and thin-film, from a whole host of manufacturers. The only solution was to sit down, go through them, update them all for thin-film parts (from one manufacturer), and roll that into the line.
This is literally the least risky thing we could do. We were taking 1% resistors and replacing them with 0.5% (and 0.1% in some cases). No other differences. It just made the product a little bit better. And now, we’re at the point where everything is running thin-film. And everything is a little bit better.
Which one do you have? Again, we don’t know. And again, it doesn’t matter.
This is how all manufacturing works. Do you really think iPhones start and end their production lives using 100% the same parts everywhere?
Yeah. Like, no.
Case the second: the glitch fix. Okay. Here’s the thing that started this whole chapter. Mike’s already written about this on his blog, but here’s the summary: a couple of years ago, someone got their nose out of joint about a “glitch” the Yggdrasil exhibited around zero-crossing. We’re talking stuff that’s 120dB down. This is despite AtomicBob measuring the Yggy (showing the glitch) and being unperturbed. But this dood was super-emphatic about being able to “hear the glitch.” He went on and on and on and on about it.
Aside: and this is how internet memes get started—one guy gets a bit, ah, obsessed, and spreads his views far and wide, far and wide, and spends all his time online defending them, because, you know, that’s what matters, not family or fun or friends or actually engaging in a productive debate where you learn something, and then someone who searches for the product gets Google barfing up all his words on the screen, because he wrote far and wide on the subject, he must be an expert. Shoot me now.
Mike and Dave and I kinda looked at each other and shrugged when we learned about it, because we all knew about “the glitch” and had discussed it during development, eventually deciding just to leave it alone, because it didn’t have any audible consequences.
But this guy kept going. And going. And going and going and going (Note to other sites: this is what moderation is for.)
Finally, Dave says, “Well, we can fix the glitch. It’s just a ROM change.”
Mike, who by now is weary of reading about his “incompetence,” says, “Yeah, **** it, go ahead, let’s compare the two ROMs again and see if there’s any difference.”
So he did. And we compared. There was no sonic difference, just as there hadn’t been any difference during development. No big shock.
“So what do we do now?” Dave asked.
“**** it, put in the glitch fix in all current production.”
“For Yggy?”
“For everything—Yggy, Gumby, Bimby,” Mike said, waving a hand.
“And tell people about it, for an upgrade?” I asked, a little nervously. It was a bit early in the production cycle to change Yggy, and Gumby was really, really new, and Bifrost Multibit had just started shipping.
Mike paused, then grinned. “No. Let someone measure it, like, a year from now, and find out it has no glitch.”
“So never tell them?”
Mike shrugged. “Tell them in a year. Or two.”
I nodded. It really didn’t matter. It didn’t sound any different, so why call attention to it? It would be like admitting the glitch was bad, if we responded to this one crazy guy.
So we applied the deglitchified ROMs and shut up.
That is, until some crazy rumors about upgrades to our multibit DACs started circulating early this month. That’s when Mike had to step in and say, “Yeah, it’s a terrible conspiracy, we made a change 18 months ago that makes your DACs measure better, but not sound any better in any way.”
Now, does that mean that there are no upgrades coming? Of course not. The products are upgradable. And we just announced Gen 5. And, are we working on something new? Of course, and always. Just like Apple. But that doesn’t mean it’ll work. You know, kinda like the thumbprint sensor under the screen.
Anddddd…wow, that’s a pretty good segue into talking about development.
Dev Chaos: Change of Plans
I’ve said before that we try a lot of stuff, but a lot of it never makes it to production. I even did a chapter on dead product ideas. Those are the ones that went a lot farther down the road than many of our experiments, though.
I mean, in the last week, I’ve done the following:
- Finished up development, including chassis tweaks, thermal and power testing, and A/B measurements for an upcoming product.
- Built three different versions of a current product to see if we could take it to the next level with relatively minor changes. (Thank you, new PCB prototype source that is the magic combination of inexpensive and fast.)
- Worked out a next-level change for that product based on the results of the different versions, then tried drawing it up, and then abandoning it (it wouldn’t fit in the chassis)
- Did extensive testing on another step-up product which may or may not see the light of day.
- Built a new variant on a current product to see if we like it—if we do, it can use the same chassis, which is nice.
- Figured out a third variant on a current product using the same chassis again, and laid it out so we can try that, too. (Again, thank you new PCB source—I no longer have to think about how many protos I want to do.)
- Tested a production change to a current product necessitated by a parts change (yep, again, but this time not Magni).
- Breadboarded a bunch of variations on one of our current topologies, to see their relative performance pluses and minuses.
And this is just me, in one week, on the analog side. Amongst all those items, only one will fer-shure end up in production (because it has to). The rest are still out there, nebulous possibilities in the Schrodinger’s Box of What Might Be.
Aside: Ah, and—competitors better sit down—the pace will increase in the next months, as we are bringing on a technical hire who will further unburden me from oddity repairs, and who will build many of the new prototypes, something I used to do exclusively on the analog side.
Mike and Dave continue to work on their own projects, including Sol and The Gadget, as well as our own USB receiver implementation (yeah, I know, Gen 5 is just introduced, don’t get too excited, this will take a good LONG while to get right, but it’s a key part of our plan to further drive up USB performance, while at the same time making it less power-hungry, and less expensive), and a few ideas that Mike has said, “are too wacky to even mention yet.”
So, are we working on new stuff, is new and better stuff coming?
Of course, and always.
Embracing Change
Yes, I know. You came into this chapter believing that every product coming off the line is exactly the same, from start to finish. You had hoped that you could just buy the best, and there would never be anything better in the future.
And here I am, throwing rocks at those dreams (and furthermore, being completely unapologetic about it.) What a terrible person I am! How unfeeling, how uncaring! How can I do this?
Because it needed to be done. There’s way too much hyperventilating about stuff we can’t control. Change during manufacturing is a fact of life. Better products in the future are a given. Heck, I should throw in the fact that not every device that comes off the line is exactly the same—there’s a thing called a “tolerance stack” that means there are “good ones” and “better ones.”
“Which are the better ones?” you ask, breathlessly. But I bet you already know the answer.
Yep: “Don’t know, don’t care.”
And no, we won’t sort out “the better ones” and charge a higher price for them. Because then
everything would cost more. I shudder to think of the investment in test equipment and personnel to do this.
Aaaand—here’s the real kicker—the “better ones” would have to be sorted on the basis of standard tests like THD and noise. Which doesn’t necessarily correlate with what sounds better. So “better” would not necessarily be better. Yeeeessshh.
Yes, we’re terrible people. But that’s OK. That’s the reality of manufacturing. So repeat after me, one more time:
this is how all manufacturing works, everywhere.
Breathe in, breathe out. Relax. Have a drink. Listen to some good music.
Things change.
And that’s OK.
