Chapter 14:
Technical Help Via Time Warner, and The World’s Most Irritating Failure Mode
Sharp-eyed readers will note that I swapped the position of this chapter in the book, moving it up before “DAC in a Toilet Paper Roll.” This isn’t because I’m a terrible person and want to keep you in suspense for another week. It’s because this chapter really comes before the next one, chronologically.
“Well, ya coulda made the outline right,” quips someone.
Yeah, and I was probably drinking when I did the outline, so there you go.
In any case, this is the story of Tony, our second employee and first technician. It’s also a story of an amazingly hard-to-diagnose production problem that, to this day, is not fully explained. You’ll see why soon, but first, the wrap-up:
- At this point in time, it’s late October 2011
- We have 4 products: Asgard, Valhalla, Lyr, and Bifrost
- Mike has yet to bring me his DAC in a Toilet Paper Roll and change the course of the company
- But, we are both thinking about what’s next…the bigger badder as-yet-unnamed balanced amp and DAC
- We’re still in the garage, though Eddie has been banned from smoking by now
- We’ve re-organized the garage to give Eddie more space, and a real table to work on—which meant moving the Corvette to Mike’s extra garage space for a while
And, even before we brought out Bifrost, it was clear I couldn’t do all the testing and repair. At least not do it, and remain sane. I was running the marketing company full-time—and, with it being busy, sometimes more than full time. I was also contracted by Penguin to write a couple of crazy books about giant robots.
Yes, I am an idiot. But hey, might be fun, right?
Yoda and Rain Man
In any case, this is how we got Tony: me freaking out from insane time-pressure, and Mike stepping in to help with a suggestion. Now, for all of you who think Mike is a Yoda-like sage who comes in with words of wisdom and a perfect plan, consider this conversation…
“Mike, I can’t do this all,” I told him. “Between Centric, the books, and the orders ramping up, I’m doing 16-hour days.”
“Well, we should look for a tech,” Mike told me.
“To work on a garage?” I said, doubtfully.
“Eddie does,” Mike shot back.
“Eddie is special,” I said. And he was. Eddie had long worked in the entertainment biz, so he was used to weird hours and spotty schedules. And he knew us. And he liked the free food.
Mike’s expression brightened. “I think I know the perfect tech for us.”
“Who?”
“Tony,” Mike said. “He used to work for Time Warner Cable. He did all the tough installs that nobody else wanted to do.”
I shook my head doubtfully. “Mike, a Time Warner tech is a little different—“
“He’s a little like Rain Man,” Mike said. “But once you show him something, he never forgets it.”
Rain Man? I thought. The image of a Time Warner-uniformed, confused Dustin Hoffman flickered through my mind. It wasn’t a reassuring thought. “Um, Mike—“
“He’s a good guy, I’ll call him,” Mike said, whipping out his phone.
“But he’s still gotta work in a garage,” I protested, as Mike put the phone to his ear.
“No problem,” Mike said. “He’s my stepson.”
Oh, great. Better and better. “But Mike, we can’t guarantee hours, and we don’t have healthcare and stuff—hell, we don’t even have real employees—isn’t he better at Time Warner?”
“Doesn’t matter,” Mike said. “He’s been laid off.” Then, into the phone: “Hey Tony, want to come down and work on some Schiit?”
“Mike!” I snapped
Mike held up a hand. “Yeah. Sure. Now is good. I just spent half an hour listening to my girly business partner talk about how he’s overworked. Yeah. Here’s the address…”
When Mike hung up, I grumbled, “Unemployed Rain Man stepson…does he even know how to solder?”
“Probably,” Mike said. “And if not, you only have to show him something once.”
“What if he doesn’t work out?” I asked.
“Then he doesn’t work out,” Mike said. “Look, I’m not trying to stuff him down your throat, but you’re about as bad as me at admin stuff. I figure, Tony’s available, let’s give him a shot.”
“But…Rain Man?”
Mike grinned. “Yep!”
The Tequila Bottle Theory of Hiring
Okay, you don’t need to be a brilliant CEO to know that what we just did there isn’t how you hire people. Not if you want to be successful. Or at least not by the rules of any manual on
Getting The Best And Brightest, ghost-written for a CEO dropped into an established $10B company by his friends from Harvard.
Because, of course, you have to have a rock-solid vetting process that includes questions like “How many marbles fit in an average toilet?” and psychological profiling to determine if the candidate is a right fit. You have to set Goals and Responsibilities and identify Key Result Areas. And you must carefully analyze the results in order to determine the best possible candidates, then offer them a can’t-refuse package that includes all the latest perks, from a fitness club to a concierge.
(Or you can go play some basketball with the founder and get a job offer, or have gone to school at Stanford with the CIO, or have compromising pictures of the CMO with a chicken. The real world doesn’t always work by the numbers.)
In my businesses, I’ve hired probably a couple hundred people over the last 20 years. In the early days, I went by the formula and the checklist. I agonized over who to hire. And, a lot of the times, the formula won over my gut.
And every time the formula won over my gut, I screwed the company.
Because people can’t be distilled down to a 2-page resume and a 1-hour interview. There are a ton of candidates skilled in the art of looking good on paper. There are plenty who can be friendly, affable, and make all the right responses to the standard interview questions.
And yet they can still fall on their face. Because it’s easy to recite a formula, but a lot harder to deal with the unpredictable Real World. People who interview well usually fall down at one of these things:
- Only Doing it By the Book, or “This is how we’ve always done it in my previous jobs.” Yeah, that’s fine, but the world is changing. Open your eyes. If you’re incapable of adapting, then it’s going to be a bad day.
- Stunning Lack of Initiative, or “Well, you didn’t tell me to do that syndrome.” Look. You’re not computer software. You’re a person. Small businesses don’t have time to program your every move. You know what needs to be done. Take some initiative.
- Prima Donna-itis, or “You’re so lucky to have me.” Hey, I know you’re a rockstar, but that doesn’t mean you need to redecorate your entire division before you do a single ounce of work. And no, we will not sort your M&Ms. You’re here to work. Get over yourself.
Your standard interview—or even the creative Google-esque interview—isn’t going to identify those kinds of people. They’re not even particularly good at identifying the go-getters, unless you get off the script and ask them
why and
how they did something, rather than just focusing on
what they did.
For example, let’s look at this scenario:
- Laid off from last job. Big red flag, in traditional hiring. Do not interview.
- Laid off from last job—for recognizing the inefficiency in the ordering system and building a custom database to automate it, thereby eliminating their own job. Hire this person. Immediately.
Today, when I’m hiring, I don’t ask any of the traditional questions, or the stupid trick questions, or give people tests. I just sit and talk to them, usually about what they’re most interested in—which may or may not be work-related. Because you won’t get to really know someone until they’re comfortable talking with you.
(I joke that my ideal interview would be sitting down with the interviewee and a bottle of tequila. But of course we can’t do that.)
The results? Today, we have a lot less churn in the business, and a lot more long-term employees. Selecting people with potential and ambition beats experience every day.
Tony and the Popping Lyrs
So where does that put us with Tony, you ask?
Well, Mike gave me a hint at his potential with his comment about “he used to do all the difficult installs.” That’s not the behavior of someone who just wants to collect a paycheck.
And Tony was excited to work with us. He’d been hearing about Schiit from Mike for a while. And even before he was laid off from Time Warner, he was done. He wanted to be part of something where he could make a difference.
And Schiit, yes, he could make a difference. Mike was right. Tony didn’t take long to train at all. It didn’t hurt that most of the stuff we were making was pretty easy to qualify, and it didn’t hurt that our boardhouse’s quality is very good. By putting Tony in the mix, all I had to do was look at a few broken products from time to time. My schedule was back to a manageable level, and everything seemed to be going well.
And, when we introduced the Bifrost, Tony loved it—because it involved programming and computers. Tony is our resident Android/Linux guy, though he also has PCs and Macs as well. He’s the guy who qualified our DACs on Linux, and he’s the guy who knows the most about how to deal with, say, failed Windows driver installations.
So everything was going great, until we get the next run of Lyrs in from the boardhouse.
Tony came into the house, jiggling a finger in his ear. “The Lyrs are popping,” he said.
“A small pop is normal as the relay engages,” I told him.
“No,” Tony said. “Popping. Blow out your ear popping.”
“It’s probably a bad servo. I’ll look at it later.”
“They’re all doing it,” Tony said.
“All of them? And they’ve been through the pre-test?”
Tony nodded. “Yep. I pre-tested them, pre-biased, then Eddie put them in the chassis, and I set the bias again, just like you showed me.”
I frowned. If they passed the pre-test and biased OK, they pretty much had to work. We hadn’t changed anything since the last run of Lyrs, and those had worked flawlessly.
I went out to the garage to have a look. I grabbed a Lyr—one labeled with a post-it note saying, “popper,” took the back chassis off so I could probe around and see what was happening, and put it on the bench.
It tested perfectly. Right gain, right bias, right voltages, THD looked fine, no problems anywhere.
“Are you sure this is a popper?” I asked Tony.
“Yep!”
“Check another one, then give it to me.”
Tony ran another Lyr through the sound check. As the relay engaged, a loud POP! came from the open-backed HD650s—loud enough to echo in the garage.
Tony winced, taking the headphones off his head.
“You know you can test it without having them on your head,” I told him. “Or, better yet, run it on the scope.”
“Okay,” Tony said, handing me the Lyr.
I took the back off and tested the “confirmed popper.”
Like the first one, it measured perfectly.
What the hell? I’d just heard it blast a headphone so loud it was amazing the thing still worked.
I turned off the Lyr, then put the scope in one-shot mode to see what kind of DC spike was happening when the relay engaged.
A few millivolts. Nothing to worry about.
“What is it?” Tony asked.
“I don’t know. It’s fine. No problem.”
“You just heard what it did,” Tony said.
I nodded. Yep, I’d heard.
I power cycled it a couple more times, and the Lyr continued to behave. No problems at all.
“I don’t know,” I told Tony.
“Maybe it fixed itself.”
“Maybe,” I said. Knowing things never fix themselves. I buttoned the Lyr back up and handed it back to Tony for sound check.
BANG! Another explosive pop reverberated through the garage.
“Is that the same one I just gave you?” I asked Tony.
“Yep!”
“You sure?”
“Yep!”
Great, I thought,
an intermittent problem. Those were the worst. I took the back off the Lyr and ran it through its paces again.
No problem. No DC spike. Nada.
This made no sense! There wasn’t anything different about our test rig and the listening setup.
Except—
I’d taken the back chassis off the Lyr.
Nah. Impossible. It made no sense.
Shaking my head, I put the back chassis back onto the Lyr, and plugged it back into the test rig. I powered it up and waited for the relay to click on.
BANG! A huge pulse of not-just-DC, but multi-megahertz oscillation hit the scope, the moment the relay closed.
“Oh, you’ve gotta be kidding me,” I groaned.
“What?” Tony asked.
“I think it only pops when the back chassis is on.”
“Huh?”
I nodded. “Exactly.”
I went back to the bench and quickly confirmed that three out of three Lyrs all worked fine with the back chassis off, and oscillated with the back chassis on were .
“Why does it do that?” Tony asked.
I shook my head. MOSFETs are weird. Something as simple as the chassis being close to the output devices might cause enough parasitic capacitance or inductance to make the amp unstable. But that was extremely unlikely. None of the rest of the Lyr runs had shown this kind of instability. It was a no-feedback amp, it didn’t have loop problems.
And, the biggest problem? When you have an oscillating amp, you can usually poke around with a finger to see where the problem lies, or at least the basic area. But with the Lyr bolted all the way together, there was no doing that.
“How are you going to work on it, if you can’t get into it?” Tony asked, echoing my thought.
“I don’t know.” I admitted.
The Brute Force Fix
The “how” of fixing Lyrs that only oscillated in the chassis turned out to be a tedious, trial-and-error process of endless disassembly and reassembly. I did all the normal stuff you do when you have an amp that isn’t behaving—additional bypassing, bigger gate stoppers, etc—and none of it worked.
The only thing that worked was to replace the MOSFETs. And, since many of the “poppers” also took out the servo and part of the Dynamically Adaptive stage through oscillation, we replaced those, too.
That finally killed it—the brute-force replacement of about a dozen critical parts.
Why did it fix it? No idea.
Yes, that’s right. To this day, I have no idea what caused the problem. That single run of Lyrs is the only run that ever popped. Nothing before, nothing since. No parts were changed, the boardhouse was the same—it should be absolutely, totally the same. But it wasn’t.
I have some theories, of course, but they’re all pretty iffy.
The most plausible idea is that the output MOSFETs were somehow damaged by static. And that’s not saying much. Our PCB assembly house is an ISO-certified facility. I doubt if their assemblers suddenly decided to leave off their static straps and wear angora sweaters to work en-masse. And we usually don’t handle the parts outside of their own protection.
Or, it could have been a bad run of MOSFETs. The ones we replaced them with had a different lot code. Again, not very plausible.
Finally, tolerance stacking—the boards, the solder, the parts and the assembly were just different enough this one time to make it unstable. Again, not very plausible, probably the most farfetched of all.
I suspect we’ll never know what the problem truly was, for the most irritating failure mode in the world.
Final word: Tony. To this day, Tony is our lead tech. He’s tested and/or programmed tens of thousands of products. Pretty good after Mike’s first intro, right?
and of course now with the futuristic looking Schiit stuff...
