2016, Chapter 3:
The Importance of Service
Okay, so let’s go to a place I haven’t talked a lot about: service. Or, more specifically, what happens when something breaks.
Yes, I know, this may seem like a topic that’s about as interesting as a plumbing manual, or discussing the finer points of automotive wrecking, but in actuality, service is really fascinating. In my opinion, doing service well is the primary thing that separates a mature company from a hobbyist venture or early-stage enterprise—and it is absolutely key to providing the highest possible value.
“Wow, that sounds pretty over-the-top,” some of you might say. “Isn’t a great product more important?”
Well, exceptional service starts with exceptional products—products that are designed to break as little as possible. And before you go into full “Well, duh,” mode, this isn’t as easy as it seems. Even the simplest product is a complex assemblage of parts…and it’s usually the first time all of those parts have been put together as a whole. This means there are complex interactions that you may not see (at all) until you’ve built a few hundred—or a few thousand—of them.
And those “complex interactions” will come in first through service.
The amp is making a funny noise. A channel is down. The DAC won’t lock to a digital input. Radio is playing through the phono preamp. A volume control sounds scratchy. Get too many of any one of the above, and you’re looking at a complex interaction…which means you need to make a change.
Which means: if you aren’t paying close attention to service, you may miss critical data that will help you make your products better.
Again, I know, this sounds like a “well, duh,” moment. But it really isn’t. Many organizations—even very small ones—have a service department that is kept as far as possible from the lofty towers of the CXO suite. (Yes, even when a “CEO” title is laughable…I personally wouldn’t think about taking such a grandiose title until I was looking at an organization of several hundred employees. At least. So let’s just refer to the upper management as the “HMFWICs”—head MFers Who’s In Charge.)
Why hide service from the HMFWICs? Tons of reasons:
- The HMFWICs think it is beneath them
- The HMFWICs pontificate about how their time is better spent in processes, logistics, research, engineering, deal-making, golf-playing, drinking and schmoozing, going to fancy retreats on the company dime, etc, the latter three usually being the most honest
- The staff truly wants the HMFWICs to spend their time better at processes, logistics, research, engineering, etc, so they divert service concerns away from them
- The staff below the HMFWICs want to hide a perceived “dirty” part of the organization from the upper management
Well, here it is, for all HMFWICs: service is
not beneath you. And spending your time better at processes, logistics, research, engineering, etc
includes service as a
primary component.
Or, in other words:
HMFWICs, pay attention to service, or else.
And please note these are not empty words. Every day at Schiit, I spend time in service. Sometimes even doing repairs directly—stuff that’s puzzling, off the beaten track, or that occurs in clusters comes to me.
Yes, you read that correctly: it’s entirely possible your broken amp was repaired by yours truly. Yes, even today, with all of our growth.
“Well, that’s an entire bag of crazy-snacks,” some CEOs will say. “You could be designing new products…working on products…making deals.”
Yes, I could. And our products would suffer as a result.
Because I take some of the service burden directly, I feel it keenly. I’m highly incentivized to make sure our products are as dead reliable as possible. And this incentivization results in a feedback effect, so things get better and better.
This is why our service load is so light. Between Tony, Jesse, and myself, we’re spending only about 12-18 hours per week (total, not each of us) for well over 100K products in the field.
So is everything rosy and perfect? Not at all. Things break. Weird things happen. But through service, we see them quickly…and we get them fixed.
But service isn’t just about stuff that breaks after a while in the field. In fact, service really starts with the warranty, how you troubleshoot, and how you handle DOA products.
So let’s talk a bit about all of those aspects…
Warranties: Not Just Rolling the Dice
You’ve probably noticed that our products have different warranty periods. The larger products, from Asgard 2 and up, have 5-year warranties. The smaller products like Magni 2 have 2-year warranties. Fulla has a 1-year warranty. And tubes are covered for only 90 days.
How did we arrive at these numbers, you might ask? Did we roll the dice and see where they fell? Did we use a giant Wheel of Fortune? Did we sacrifice small animals to read their entrails?
No. The warranty periods were chosen deliberately in every case. The decision was based, roughly, on two factors:
- How long we figured a product would last, given typical usage. Note the “typical usage.” This will be important later.
- How long the average competitors’ warranties were. We wanted to be longer than most of them, without going full-crazy like some outliers offering 20 years (or even more.) More on this later.
“So does that mean a Magni 2 is made like crap, so it has 2.5x less warranty than an Asgard 2?” you might be asking. “And, if some companies can do 20-year warranties, why can’t you?”
Both good questions. Let’s look at them in turn.
First, no, there’s no reason a Magni 2 won’t last as long as an Asgard 2, given the same usage. Or a Ragnarok. Or a Gungnir. They’re all built to substantially similar standards, with the same gauge of sheetmetal used in the chassis (well, except Ragnarok and Yggdrasil, which are thicker), the same FR4 2- or 4-layer boards, the same premium-but-not-audiophile-insane components, etc.
But note the “given the same usage.” Magni 2s (and the other small components) typically get used harder than the larger gear. Magni 2s are much, much more likely to get thrown in backpacks, carted around many different places, hooked up into different systems, etc. Because of this different usage, and to keep prices down, we decided to limit the warranty on smaller products like Magni 2. Fullas are even more likely to get bashed around. So the warranty is even shorter. Tubes? 90 days covers DOAs, and that’s basically all we need for NOS tubes.
And yes, we said it.
“And to keep prices down.” When you’re doing $99 products, this is absolutely a valid concern when setting warranty period. Sorry, but that’s reality.
So what about those 20-year warranties? Why not up the warranty term for everything in the line? Isn’t that a great way of saying how confident you are in your products?
Well, yes…but in the era of lead-free solder, it also may be a little foolhardy.
“Wait, what?” you may be asking. “Isn’t lead-free a good thing?”
Absolutely, when it comes to not growing third arms out of your forehead and having kids that can do basic math and be trusted not to eat the furniture. However, in terms of electronic products, lead-free solder still has some question marks around it. Although we haven’t seen it in our products yet, lead-free solder can grow “tin whiskers,” which can cause shorts on the board—and all sorts of problems. Perhaps this won’t be an issue down the road…but we don’t know. Where we used to be comfortable with a 20+ year lifespan for leaded solder, lead-free may be more like 10 years max. Or maybe not.
But would you want to stake your company on a “maybe not?”
Yeah. There you go.
And, finally, although it sounds a bit morbid to say this, every product needs an expiration date. 20-year warranties expose you to the “what if lead-free is a big issue in 10 years” problem, but infinite warranties expose you to something even worse—infinite liability. And companies typically don’t like infinite liabilities. They don’t do good things for balance sheets.
Melodramatic? Maybe. But if 80% of your boards fail in 15 years due to tin whiskers and you have a 20-year warranty, weeeeellllllllll….that’s a huge bag of bad stuff. And, if you have a boatload of products with infinite warranties, good luck explaining (to an investor, bank, or corporate suitor) your strategy for servicing them 30, 40, 50, or more years down the line…when parts may simply not be available.
And that’s why you’ll see 5 years from us, max. Excuse us for wanting to keep prices down…and excuse us for wanting to be around for the long haul.
“We’ve Got a Deader”
Okay, now let’s move on to the first “service case,” to use some corporatese: the DOA product.
Yes. It happens. You open up a great new shiny wonderful thing, plug it in, hook it up…and it doesn’t work. Just plain dead. No workee.
And yes, it happens to everyone. Us included. No matter how much we test, no matter how long we burn something in, there are going to be DOA products. This may be due to shipping damage…or it might just be its time to expire.
So what do you do?
If you’re Schiit, you do what’s known as a “Rapid Return/Exchange.” We send you a shipper so you can send the defective product back to us at zero cost, and we send you a replacement as soon as the defective one hits the mail.
Yes, I know. Nothing groundbreaking, at least in terms of large-company policy. However, the more discerning may have noticed that we say nothing about “Rapid Return/Exchanges” on the website, just some stuff about “If you need warranty service, you’ll pay shipping one way, and we’ll pay the other.”
Yep, sure, that’s the letter of the law. And that’s usually what happens, when your Asgard expires three years after you bought it.
But if it’s new and dead, we will do a Rapid Return/Exchange every time we possibly can. Which is about 99.9% of the time. But if you happen to buy the very last black-chassis Asgard 2 and it shows up DOA, weeelllll….then it might have to come back to us. But that would be on our dime.
The point is, we can’t guarantee a Rapid Return/Exchange every single time, 100% of the time…which is why it ends up being undocumented on the site.
The Art of Troubleshooting
Okay. Let’s say the shiny new product shows up…but it doesn’t work quite right. It’s noisy, or it glitches when hooked up to your laptop’s USB port. Or it works happily for a year or two, then drops a channel. Or suddenly refuses to show up in your computer sound output panel.
In these cases, it’s time to troubleshoot.
And this is where it sometimes gets fun. Because there are some people out there who absolutely, positively refuse to believe there can be anything wrong with their system other than their Schiit product, and will spend much time and effort defending their system, rather than doing some simple troubleshooting.
Or so it sometimes seems. I see a lot of the support emails, and it’s amazing how some like to argue that their 100% electroplated yak-hair and virgin rubber cables couldn’t possibly be the problem because they cost several times the retail price of the gear they’re using it with, when a simple cable swap would show that one of the cables had expired. Or that there is absolutely no way that Microsoft could update their USB drivers without them noticing it, despite the fact they’re running Windows 10 on a computer that’s constantly connected to the internet.
In fact, running through the numbers, about 7 out of 10 “problems” with our gear are not related to the gear at all, but is usually one of three things:
- USB port power management problems. Number one with a bullet. We have very few problems with drivers these days…but getting a USB port not to throttle power sometimes seems to take an act of God.
- Bad cables. No kidding. You would not believe how many bad cables are out there.
- Operational error. You would not believe how many Bifrost problems can be resolved by asking if the right input is selected.
Now, we know that the USB port power management problems are real problems, and we continue working with C-Media (and through them, Microsoft) on minimizing them. But the reality is that companies like Seagate and Logitech also have problems with their hard drives and input devices being powered by USB ports (do a quick Google search if you don’t believe me). Plus, El Capitan has been nicknamed “El Crapitan” by many USB DAC users (not just ours.) Again, don’t believe us, Google it.
And if companies like Seagate and Logitect can’t get 100% compatibility….aaaaand if Apple has worked for 8 months to undo the problems with El Crapitan and are only finally getting it worked out in 10.11.4 (beta), weeeeeeelllllll…there are
gonna be problems.
Those top three issues are why we provide a number of troubleshooting guides on the site (USB Problems, DAC Problems, and System Problems). These guides are quite detailed, and will almost always allow us to determine if there’s a problem with our product or not. If there’s something wrong, we’ll get the product back, fix it, and get it back to the customer as fast as possible.
But sometimes, people don’t want to go through the troubleshooting. If they don’t, or can’t, we have only one choice—to take the product back and have a look.
And many times, there’s nothing wrong with it.
And that’s arguably the worst thing that can happen. “Fault not found,” is not cool. The customer doesn’t want to hear this. And we don’t want to send it back without really trying to get the product to fail. That’s why a “fault not found” repair gets a whole lot more scrutiny:
- It’s run through two separate technicians, to ensure that one isn’t missing something (one of them is usually me.)
- It’s burned in for a while to see if it only happens when warm or after a period of time.
- If it’s a digital product, it’s checked on several computers.
And yes, sometimes there really is no fault to be found. Those are painful to send back, but I’m absolutely certain that our current process will find something wrong, if there is something to find.
Of course, there are always problems that don’t slot neatly into a “type.” Those will take some more back and forth, maybe even with the help of some photos or screen-captures. In every case, it is much better to do this via email, since we have a complete record of what we’ve said, what we’ve recommended, and what the customer has tried.
Hence our insistence on email support rather than phone support. I can’t tell you how many times I’ve seen phone support end up in endless loops due to lack of information…or completely inefficient as they have to enter all information into the ticketing system in real time.
And, since email support is so close-coupled to service, we know what typical problems are real…and which may be phantoms. Have a Lyr of a certain vintage with noise in one channel? You’re gonna get one email from us asking you to swap the tubes from front to back, and if the noise doesn’t change channels, one more email to send it in. (Bad batch of gain resistors, believe it or not.) Have an Asgard 2 that’s wonky? We’re going to be pretty insistent on you going through the entire System Problems troubleshooting guide. Because they sooooooo rarely break.
Beyond that, what breaks is mainly what you’d expect. In order:
- Switches. Things that move are usually what break. See the top three items on this list. Note that this doesn’t mean the switches are crappy…they’re just the most used, and the most likely to be snapped off if the product is mishandled in shipping.
- Connectors. By this, I’m including everything—RCA jacks, tube sockets, AC connectors, etc. Another mechanical part. Not surprising they sometimes have issues. However, the Neutrik headphone jacks have been stunningly reliable. I’ve only seen a single case of a broken jack…ever.
- Potentiometers. Yes, sometimes potentiometers break. Usually in shipping. This is despite using very nice Alps pots.
- Wall-warts. Despite pre-testing wall-warts, sometimes they arrive DOA, buzzy, or they expire after a few months.
- Transformers. Every once in a while, we’ll get a transformer that expires before the warranty. Note we’re getting into the “Every once in a while” phase here. Once in a looonggg while.
- Everything else. Output transistors sometimes show up dead, but it’s so rare that the occurrence is probably in the low double digits. Sometimes input transistors show up dead. How that happens, I’m still guessing. Sometimes we get a noisy capacitor or resistor (yes, it happens). And sometimes a relay driver dies. However, we have yet to have a truly dead relay.
But if you have problems, take a deep breath. We’ll get it fixed up…whether that means getting you to swap a bad cable, or if we have to bring it back and replace the whole thing.
In the Shop
Once a product comes back to us for service, it gets just that: service. We do very little “board-level” or “module level” service, and essentially no “swap for refurb” service.
These euphemisms mean, in order:
- Swapping a whole board without troubleshooting the existing one. Fast, and very inefficient. If you’re running a typical high-margin niche product (read, overpriced at 8x+ the BOM cost), sure, you can just swap boards and scrap the deaders. But this really isn’t an efficient way to run.
- Swapping a module without troubleshooting the existing one. At times we’ll do this on complex boards, like Yggdrasil’s USB input or DSP board. But in either case, those boards go into a queue for troubleshooting later.
- Swapping the whole product for a refurbished one. When a product is essentially unrepairable, or only repairable at great expense (or with low-cost labor), this is a common strategy. None of our stuff is designed like this, so it doesn’t apply.
So what do we do? We actually go through the board, troubleshoot it, and fix what’s wrong. This requires a much higher level of technician than one who is simply swapping boards and modules, but it’s absolutely worth it. It’s worth it, because the cost of troubleshooting an existing product is lower. It’s worth it, because it promotes greater understanding of what actually went wrong. And it’s worth it, because it encourages us to keep our products easily serviceable. All of this contribute to a feedback loop that further reduces service problems and service costs.
And now you see why I do some of the service. It’s very useful to dive into a board and see what’s really happening. And once you’ve seen a few with the same problem, it’s easy to go upstairs and make a change to improve the next run. That’s why we’ve seen less and less problems over time…and less problems with new product introductions. We learn from service…and that learning is instantly applied to improving the product.
And I’ve seen some pretty bizarre stuff. Some of it’s our bag, some of it is the customer’s bag, and some are mysteries.
Our bag:
- The aforementioned Lyrs and Valhallas that got bad gain resistors. Yep. No way to predict it. And no way to know it would be a problem, until they started failing a couple of years later.
- Some Bifrost Multibits with a distortion problem on sample rate change. It took us many weeks to replicate this and find a firmware fix, but we have one now. If you have a funky Bifrost Multibit, let us know and we’ll take care of it.
- Old Asgard 2s with humming transformers. These still show up from time to time.
Their bag:
- An Asgard that a cat threw up in. Cleaned it up. It worked fine.
- An Asgard that got orange juice dropped in it. Again, cleaned up and worked fine.
- A Ragnarok with a near full-grown cockroach in it. No clue if it caused the problem…still wondering how it got in there.
- A Mjolnir with about 5 pounds of dust inside of it, nicotine stains on the outside, and various other stains I don’t want to know about. When cleaned, the noise it had been making went away. Not a huge shock.
- A Mjolnir that had been disassembled…including having holes drilled in the bottom (why?), a broken potentiometer, screws snapped off and chassis gouged to heck and back. This was supposedly an attempt to install an audiophile fuse. That repair bill was pretty big.
Mysteries:
- A Magni 2 Uber that literally smoked the output stage—as in, it burned the output resistors and charred the board. That one got a whole new board. Why? Unless it was overdriven into a shorted load for a long time, I have no clue.
- A Valhalla 2 that hummed (through the headphones) for no reason at all. Checked all the grounds. Replaced a bunch of parts. No dice. Nothing fixed it. That one got scrapped.
- A Mjolnir 2 that came back, “eating” expensive NOS tubes. I took that one, figuring it would be easy—a bad heater regulator, overbiased, oscillating, something like that. Nope. Heaters were perfect. Biased perfectly. Not oscillating. Sounded great. Stuffed it on the burn-in rack for days with stock tubes. No problems. That had to go back to the customer with a “no fault found” diagnosis.
The Penalties of Inattention
I’ve seen companies brought to their knees by their service load, and I’ve heard stories about many more. Ed Miller’s description of the last days of Great American Sound—killed by unreliable amplifiers, desperately trying to get ahead of a tidal wave of broken gear—really resonates.
But those are big, grandiose kinds of ends. It takes a lot of bad product, for a very, very long time, to really kill a company. Allowing your service problems to get to that level is probably pretty rare. Because, at some point, the troops have to go to the HMFWICs and say, “You know, we really have a problem.”
But I’ve also seen companies not much larger than Schiit labor with 7-10 full-time service technicians…and consider that 100% normal.
And that’s probably the real problem with inattention to service…it can seem to be relatively benign. It can seem to be “normal” and “all right,” as long as the company is operating, producing a reasonable profit, and the team of service techs are keeping ahead of the burden.
The problem is, “normal” and “all right” continue to get redefined, until they aren’t. Until the company isn’t doing so well. Until the service techs can’t keep up.
Doing service well is one of the primary differences between an established, viable company and a hobby business or start-up. And keeping service close to the top is, I’ll argue, one of the primary differences between a company delivering top value…and all the rest.
I’ll say it one last time:
service is worth your attention.
.
