Retired Quarterback

Over the last several months, I've implicitly adopted a change in my management style that's been (I'm informed) quite a relief to the folks around me.

Several years ago, a candidate I was interviewing asked me an interesting question: "Would you describe your relationship to your employees as more like a manager, or more like a coach?" My answer was that I saw myself as more like a quarterback: directing the game, but playing on the same field as everyone else, taking the same hits, and working to develop more-or-less the same skills. In other words, I saw myself as engaging in the same debates as everyone else, letting my ideas compete with everyone else's ideas on an even playing field, and expecting and hoping everyone would throw themselves into the argument with the same vigor.

In retrospect, about the time she asked me that question is when my answer stopped being the right one. It sounded good at the time, and up until that point, I think it had worked well. When we were a smaller team, when I understood the details of most decisions that needed to be made, it was all right for me to jump feet-first into every conversation, throw my idea down first, and then let everyone argue with it.

But Zango has grown and changed an awful lot over the last several years (and especially in the last year). Among other things, the size of our technology team has grown. When there were only a dozen of us, sitting within a few yards of each other, we knew each other well, we understood each other, and most everyone was comfortable with the rough and tumble of our chosen rapid development strategy. But our technology team now consists of almost 90 people, spread across the US, Canada and Israel. Sometimes I have a hard time remembering everyone's name, and I simply can't have a close, personal relationship with everyone. (I hate that realities, for what it's worth, and I'm working hard to keep up to speed.) It's not that people won't argue or debate with me, but they're not as comfortable with it – and to be honest, neither am I. It's easier for one or the other of us to get defensive, and once that happens, it's hard for me not to win the argument: and that's obviously problematic.

In addition, I'm a lot dumber now than I was then. When we had two or three projects going at once, I could go deep enough into the details to contribute meaningfully to each. But last I checked, our portfolio consisted of some 30 active projects. There are too many details to manage, and I just don't understand the nuances of most of the debates well enough to put my opinion out first and have it be meaningful. When I have to skim as quickly as I do, my first take on an issue is generally going to be wrong.

Finally, and perhaps more controversially, it's more problematic when I lose a debate these days. This is something that I didn't understand for a long time, and even now, I'm still not sure I accept it: but I can see its effects. As a leader, it's critical to admit when I'm wrong – but the further I get from being able to have a personal relationship with everyone, the more important it becomes that I not be badly wrong in my stated opinions very often. There is such a thing as managerial/leadership capital, I think, and it's not unlimited.

So I've reluctantly dropped my "quarterback" metaphor. I still don't know whether I would more describe myself as a "manager" or a "coach," but the practical effect is that I try to sit back and listen to the whole debate before I weigh in; I ask a lot of questions; and if I finally speak up, it's if I have something meaningful to say. Generally, if I do have something meaningful to contribute, it won't be specifically technical: those sorts of things can and should get solved before I weigh in. My opinions need to be about higher level issues: the overall architecture that we're choosing, the relative priority of a given business issue, what level of resources to throw at a given problem.

Mt. Adams

I'm off to Mt. Adams for a couple of days, to try to drag my sorry carcass up the south climbing route. I'll take pictures, and see if any are interesting enough to post when I get back. See y'all soon.

MSR Whisperlite International

Yikes. OK, for what it's worth, I don't recommend this stove. I've had an MSR DragonFly for years, and all things considered, it's been fine: no complaints, except maybe it's a bit heavy. But when I finally lost some key pieces to it (it's a long story, involving an airport, a full fuel bottle, and a dunderhead trying to combine the two), I thought I'd try something a bit lighter, and replaced it with a Whisperlite.

I'm reasonably confident I'll be returning it to REI this afternoon.

To start with, the directions have this encouraging quote: "A soccer-ball sized ball of flame is normal." Certainly my testing (in my garage just now) seemed to bear that out. The directions didn't indicate whether it was also normal for the stove to start spilling liquid burning fuel everywhere, but I'm guessing even MSR would consider that problematic. I managed to avoid burning the house down only by grabbing the still-burning, still-dripping-burning-fuel stove in my hand and running with it outside onto my driveway, where it could sputter and drip and burn in relative peace.

I'm not sure what went wrong, and quite possibly it was something I did: but I can say this, that in the last eight or so years I've been using my DragonFly, I've never actually felt like I was in danger of blowing myself up. A friend of mine once said, in reference to rock climbing, "I'm a father: I'm not going to participate in a sport where the penalty for a mistake is death." I'm not sure I agree with his sentiments about rock climbing, but I'm fully in sympathy when it comes to backpacking stoves.

And even after I got the WhisperLite working, i.e., not actually spilling burning fuel, I wasn't impressed. It seemed to put out barely any flame, and what flame did come out wasn't the sturdy blue flame I've come to expect from my DragonFly, but a weak, wobbly yellow thing that seemed unlikely to heat anything interesting. Maybe that's how it's supposed to be – but I'm thinking it's not what I want.

Two Holy Men

A couple months ago, when my cousin Brian was on his way to climb Mt. Everest (see pictures here, here and here), he ran across the Indian holy man below:

I've never been to India, but I understand that they're a common site.

I had occasion to think of this tonight, when I was with my friend Doug at a Devereaux concert, during the Bite of Seattle. I noticed a very similar gentleman, presumably homeless, dressed largely in what looked like women's scarves, twirling and swaying, sort of in time to the music. A group of hip looking teenagers were not-so-subtly laughing at him.

My suspicion is that the two men are alike in more than appearance, i.e., I suspect that by traditional medical standards they both suffer from some form of psychosis. But they differ in one key way. In India, there's a place for people like that. They know how to fit them into their culture, and in their own way, they are respected. I doubt that you want your son to grow up to be like that: but if he does, you know that he's not going to begging for change down on skid row. There's a way for that person to be understood, to fit, in a way, into society. And of course, in America, we don't. They're a problem to be solved, not a holy man to be revered. They carry the whiff only of spirits, and never of the Spirit. Think on that for a while, and ponder what we're missing.

Best Practices for Remote Dev Offices

As I mentioned in my last post, there's no doubt that managing developers in remote offices is tough. Here are some things that I try to do, which have helped:

1. Build trust. It's astonishing how much more gets done when teams trust each other. Two tactical recommendations along these lines:
   
   1. Travel. There's no substitute for butts on airplane seats. Debating database architecture long distance has an entirely different tone if the person at the far end of the call took twenty bucks off you at poker two weeks ago.
      
   2. Use video-conferencing. A person's face takes up more room in your brain than their voice. Plus, you're less tempted to check email while they're talking. We use the Polycom VSX system for our conference rooms and Polycom PVX for our desktops; the UI is annoying and buggy on both, and I have ongoing problems with the desktop audio, so I can't really recommend it, but it works better than just a phone call.
      
2. Communicate. Schedule regular 1:1 meetings with as many people at the remote office as possible. If folks at our Tel Aviv office stay late, and folks at our Bellevue office come in early, we can get maybe 8 hours of overlap a week. I try to have all 8 of those hours filled with meetings; usually, I'm about half successful.
   
3. Adopt the same processes, and fairly quickly. Two keys to doing this:
   
   1. Pick one set of processes and have the other office/company adopt it. Normally the dev process that will be chosen is, naturally enough, the acquiring company's. But don't just have an argument about processes in the abstract, or try to merge the two. Pick one, and move on it. If you're the acquiring company, be open to adjusting your own processes based on the other company's best practices – but everyone should expect that one set of processes will provide the framework for the discussion.
      
   2. Structure the organization at the remote office to support the process changes. You need to have people on the ground who have bought into the new processes wholeheartedly, and are willing to fight for them. Sometimes the only way this can happen is to bring in new people (e.g., product or project managers), who aren't wedded to the old way of doing things.
      
4. Choose the right managers. Managing remote team members is a difficult skill, and it needs to be learned. Just because a manager has been a success with local team members doesn't mean they'll be successful with remote employees. (I'm a great example of this.) If someone hasn't managed remote employees successfully before, keep a close eye out for problems, and step in quickly. This is a good opportunity for mentoring.
   
5. Be very patient. To a much greater extent than you can know, you don't understand the context for decisions made in remote offices. Always give them the benefit of the doubt. Sometimes that benefit isn't deserved, but in the end you'll get further (and be right more often) if your assumption is, "There must be something I don't understand," and not, "That was stupid."

Merging Dev Teams

One of the hardest things I do as a manager is to oversee remote offices and employees. In addition to our headquarters in Bellevue, Washington, Zango has development offices in Montreal and Tel Aviv. We hadn't intended to open offices there, but took them on (and are glad we did) as a result of various acquisitions and mergers.

When we first acquired CDT in Montreal, I'll confess, I don't think I did a very good job managing the office. The only people I had been managing for the previous five years had been sitting outside my door, and I had developed some bad habits. I took for granted the trust we had built up over years of working together, and I was used to communicating in a sort of short-hand. Most of the developers here could complete my sentences, and we thought about (or at least argued about) architecture in very similar ways. We had the same philosophy about where to invest extra architectural efforts, and where to take short-cuts, and if we had different perspectives on a given issue, we trusted each other enough to not take the debate personally.

But those same techniques didn't transfer well to a newly acquired company in a remote office. Because they weren't right outside my door, I didn't talk to the folks in Montreal regularly, and it took longer to develop a sense of trust than it should have. I didn't pay close attention to what they were doing. I didn't make much of an effort to visit the office or establish strong relationships. Not surprisingly, we have had more turnover there than I wanted. Even so, the team there has worked very hard, they've done a great job, and have accomplished a great deal – but at least originally, it was probably more in spite of my management than because of it.

The good news is that I've got very good managers working for me. As it turns out, some of them are very experienced with remote management, and I've learned a lot by watching them. So when the time came for our merger with Hotbar a little over a year ago, I managed to avoid most of the same mistakes I made after the CDT acquisition. There's still been turnover: you can't avoid it in any sort of M&A situation. But we did things differently this time, and our Tel Aviv office got off to a smoother start than our Montreal office.

Still, there's no getting around it: managing remote teams takes a great deal of effort, and an abundance of patience. In my next post, I'll try to cover some of the specific things I've been doing lately to help our remote offices be a success.

What would you do?

A little over an hour ago, I was finishing up a hike up and down the cable line trail up Tiger Mountain, near work. I had been mostly running down the trail, so I came around the last bend of the trail, and onto the pavement, pretty quickly.

The first thing I saw was a thin, sallow-looking man pulling a woman's purse out of the broken driver's side window of a black Lexus, two cars down from mine. There was glass lying all around the car. He was about 10 feet away from me. He looked startled, to say the least.

What would you do?

1. Swear at him. Loudly.
2. Whack him with your hiking pole. Hard.
3. Chase him down the street.
4. Call 911.

The correct answer, of course, is (e) All of the above.

Real People

As part of my ongoing attempts to keep up with the industry, I've been reading through the Sunbelt and VitalSecurity blogs. For those of you who don't know, folks at Zango and folks at those two companies haven't always, shall we say, seen eye-to-eye on everything.

That's perhaps not too surprising. On the one hand, Zango has made mistakes: we've (unwittingly) done business with folks it would have been wiser to stay away from, and we didn't police our distribution networks like we should have. We've acknowledged as much in the past, and we've paid our dues (and then some). (And for what it's worth, anybody here at Zango can tell you that nobody's shouted louder or gotten more angry than yours truly whenever we found a bad distributor. Frankly, I probably yelled louder than I should have: repairing relationships with some of the co-workers I got angry at has taken a while.)

On the other hand, it's no ding on them to point out that companies like Sunbelt and Facetime have their own vested interests. They've got investors and balance sheets too, and the more people whose machines they can identify as "infected", the more software they can sell. In other words, they look pretty good any time they can make Zango look bad. It's possible this doesn't influence them as much as I suspect, but you can't deny that this is the dynamic. In addition, and mostly to their credit, they have all the righteous zeal and indignation of any narrowly focused and highly committed community. As we all know, there are plenty of bad guys out there well deserving of their wrath. (It's been rather fun watching PaperGhost take out YoGangsta50.)

But for some time now, Zango's business partnerships and practices have been as solid and trustworthy as we can make them in this fallen world. In light of that, it's disappointing but probably not surprising that our critics have given us very little credit. And the result, as you can imagine, is that there's real tension between our company and theirs.

So reading through their blogs has been enlightening. I was struck especially by this post, which shows a picture of the people who banded together to help Julie Amero. I have to confess, I was grateful to that entire community for the work they did to help her. For those who don't know the story, it's summarized here, though the blank Wikipedia prose scarcely does justice to the situation. It scares me how badly our justice system went awry, and in the end, it wasn't the justice system that saved her: it was, well, the righteous zeal and indignation of a narrowly focused and highly committed community.

In addition, it's good to see that, as frustrating as they can be sometimes, our critics are real people too. They're a community; they're friends. They go out to dinner: they laugh at each other. They inexplicably reference stupid rap videos. There's another side to them that doesn't exactly come out in the incessant arguments and debates. That's good for me to know.

In the interests of full disclosure: part of my hope in posting this is that our critics will also realize Zango isn't quite what we've been made out to be. We support our community. We do fund-raising for the American Cancer Society. We don't take ourselves seriously; we do take money from each other at poker. We've been identified as one of the best companies to work for in Washington State for three years running. And although it's difficult to demonstrate to anyone who isn't in on our near-daily compliance meetings, we try really hard to do the right thing.

Microsoft’s Strategic Adware Initiative

In my last post, I questioned how MS was planning to leverage its investment in extensive web service API's, otherwise known as the "Cloud OS". Along those lines, I've been perusing the Windows Live SDK, and noticed, oddly enough, that MS included the adCenter API as a part of it. That seems like an odd place to put it, given that adCenter's focus is a completely different audience than Windows Live (and the rest of the Windows Live SDK). Unless, of course, MS planned to make advertising a key part of their web services initiative.

In other words – shocker – it sure looks to me like MS is planning to use advertising to make money off of their web services.

Nothing terribly revolutionary there: advertising has always been the way you make money off of services people won't pay for. But the hard part about making advertising money off of web services is the ad format. How do you make money off of a bare API call?

There are three ways I can think of to do it, none of them mutually exclusive:

1. Include advertising elements in the objects returned from the API call, and require or encourage their use. Microsoft's Virtual Earth UI controls don't do this yet – but there's no reason they couldn't. Or you could also give websites that consume MS web services special breaks if they also become adCenter publishers. Or you could just say, "If I don't see 'x' number of ad impressions or clicks resulting from 'y' API calls, we'll shut down your access."
   
2. Use the database of information gathered from these API calls to compile behavioral profiles on users, and target advertising based on past behavior. See #1.
   
3. Build advertising into the base OS, so that API calls (and the data riding in them) become just another monetizable context, the way that URL's often are today.

Then a friend pointed out a Slashdot posting on an MS patent application recently made public. This patent application outlines an architecture for "an advertising framework [which] registers context data sources and advertising display clients". And this "advertising framework" is specifically mentioned as being "a part of the OS, an application, or integrated within applications".

In other words – shocker – at least one way MS is thinking about monetizing its server-side investment is through client-side, desktop advertising. "Adware", in so many words.

No wonder MS was so interested in Claria back in 2005.

Cloudy OS

I've been watching the various Internet platform plays with some interest recently. It's old news that Google is trying to leverage its search engine might into an overall dominance of the web services space. Amazon's S3 initiative is similar, and pretty useful to a developer, too, but I don't get the impression that Amazon is trying to take over the world with S3: it smells too experimental, like it's nothing more than a side-line to their e-commerce business. I've been playing around with Facebook's API recently, and there are a lot of things to like about their approach – but again, Facebook isn't going to take over the world either (though they seem well positioned to grind MySpace into the ground, at long last).

But a recent ZDNet post about Microsoft's "Cloud OS" got me thinking. Unlike Amazon and Facebook, but just like Google, MS does have the resources and the vision to take over the world. And the fact that they've (sort of) chosen the metaphor of an "operating system" to describe what they're doing shows the breadth and extent of their commitment to providing a set of basic, necessary and extensible services to Internet developers. Google is moving in the same direction, of course, what with the introduction of Google Gears and similar services, and I suspect that when all is said and done, they're going to have a fairly full-featured set of API's.

But I'm only hearing the "OS" metaphor from Microsoft (let me know if Google is talking that way anywhere). That's natural, to some degree, of course: MS has earned their current pinnacle of world dominance precisely through their control of the desktop operating system, so of course they would want to extend this to their next attempt to dominate the world.

But I wonder if it will actually work. I don't think you'll be able to make money – or much money – off of web services directly. You can't charge for a web-serviced based "operating system" the same way you can charge for a desktop OS. (Amazon's trying it with S3, but that's only in the enterprise market, and it doesn't lend itself well to your standard mash-up.) The point of this sort of platform play really has to be about controlling the platform, and that's the one thing you can't do with web services. If you're writing an application for Windows, and don't like the Win32 GDI, you can't really replace it with X-Windows (even though there technically ways to do it), or with Quartz. But if you build your entire application around web services, it's not that difficult to rip out Microsoft's mapping application and replace it with Google's (or vice versa). Unless I'm missing something obvious, you really can mix-and-match web service API's in a way that you can't mix-and-match OS API's.

So I'm not saying that this isn't all pretty darned cool – really, it's quite exciting, a new revolution in the making. And probably Google and MS will find ways of making money off of their API's indirectly, in ways that aren't really clear to me. But I'd be quite interested in hearing how executives at either company answer the question, "So when we've got this all rolled out, how's it going to make us any money?"

Being Dumbledore

"You know who I want to be like?" I asked a friend tonight. "Dumbledore."

"You want to be like Dumbledore?" Doug said. "You mean, dead?"

Which wasn't quite the point.

The point – to the extent there was one – is that as much as I can admire a fictional character for anything, I admire Dumbledore's ability to take things in stride, to somehow be above it all.

I just finished The Halfblood Prince tonight (yeah, yeah, two years after everyone else), and I found that I was surprisingly moved by the ending. When Sirius died at the end of book 5, I found I didn't really care: Rowling hadn't done enough (in my opinion) to really make me care about him as a character. Lupin was much more interesting and sympathetic. But I was genuinely saddened by Dumbledore's death. It had to have been quite difficult for Rowling to kill him off (much like it must have been difficult for McMurtry to let Gus die at the end of Lonesome Dove). I will miss Dumbledore.

The thing that made him so interesting to me is his ability to be in charge of a situation, completely unthreatened by anyone else, while maintaining a bemused but quite genuine humility through the whole process. Granted that it's easier to write that sort of character than to be that sort of character, it's still something to which I aspire. I invariably get defensive in an argument, and it almost always works against me. The ability to keep your cool while defending your perspective is as rare as it is valuable. It's rather odd that this equanimity is so difficult to find: and even more odd that, although I recognize its value, it's still so difficult to achieve.

Dorothy Lake

Galena was busy today, catching up with friends, so I took the opportunity to head into the mountains. I looked through our map of the Alpine Lakes Wilderness, and decided on the trail to (and past) Dorothy Lake, which I hadn't hiked before.

The weather was beautiful, and the trail wasn't overly crowded – I probably passed 30 or 40 people over the 13 miles I hiked. A few pictures:

Turns out that most of the pictures I thought were any good from this hike were of plants and wildflowers more than scenery. Most likely that's because it was pretty harsh sunlight most of the hike -- which makes for great viewing, and a great time, but not so hot pictures.

As a side note, I should mention that I love my Nikkor 18-200 VR lens. Recommended.

Keith’s Blog

Turns out that Zango's CEO – Keith – my "little" brother – has a blog going. It's interesting enough to read. Recommended.

Getting Stuff Done

An exchange I had today with another senior manager here at Zango gave me an opportunity to put words to my philosophy around changes to our production environment. It can be summarized as:

1. Move fast enough to make mistakes.
2. When you make a mistake, slow down enough to figure out what went wrong, and don't do it again.

Both parts of this philosophy are critical. If you don't have #1, you'll never get anything done. If you don't have #2, you'll keep screwing up the same way over and over again.

I've worked in environments where #1 wasn't the case, where change control processes were out of control, and crippled people's ability to get things done. At one company I worked at, before you could make a change to the environment, you had to do a formal write-up, then send it to a committee which met once a week, and only after that committee had approved it could you actually make the change. The net result was that nothing got done, and employees (not all of them in IT) got very frustrated. If you're AT&T, and any outage is immediately visible to millions of people, I admit that you do need to be this careful. But for most companies – and Zango is one of these – you'll end up getting more done in the long run if you're willing to take a few risks and make a few mistakes along the way.

I've also worked in environments where #2 wasn't the case, where we kept making the same mistakes over and over. But at Zango, any outage triggers our post-mortem process: we describe what happened, what the impact on revenue or installs was, and do our best to drill down to the root-cause of the problem. Then we document the steps we'll take to make sure it doesn't happen again, and follow up to make sure those steps were taken. Over time, this process cuts down on the number of needless mistakes more than any decision to "go slowly and carefully" could.

And a final unnecessary point: out of a somewhat macabre sense of humor, I insist on calling them "post-mortems," not "outage reports" or anything like that. Ever since I worked at a funeral home during college, the image of a body – or in this case, a project or a rollout -- lying open on a slab, with people gathered around and pointing, has been a rather potent one for me. You can't help but pay attention.

Rereading Potter

Like millions of other people, I'm looking forward to the two new Harry Potter releases, the film this week, and the book next. Galena and I have been rereading the entire series in preparation: I finished The Order of the Phoenix last night, and when my allergies got me up at 1:00 am, managed to make about 100 pages of my way into The Halfblood Prince.

It's interesting rereading the series. Even though I enjoyed Rowlings' work, when the first Potter books came out, and were widely compared with Lewis' Chronicles of Narnia, I was insulted by the comparison. "Fifty years from now," I said to anyone who was interested, "people will be asking 'Harry who?', but they'll still be reading Narnia." I still think that the Chronicles will last longer than Harry Potter, but my opinion about Harry Potter has become somewhat more nuanced.

Lots of people have pointed out that the plots have become darker, and in that way, more interesting. But Rowlings' character development has also been better than I expected it to be. Harry isn't all good, any more than his parents were: he's moody, he doesn't trust his friends, he gets angry, and at one point he uses an illegal cruciatus curse. Lots of characters who are on the "right side" are still unpleasant people, and in that way, it's a good deal like real life. Harry's disturbing trip through Snape's pensieve, or his misplaced attempt to "save" Sirius at the Ministry of Magic, were not what I expected – and are therefore more interesting than I expected.

Still, there are aspects to the books which have disturbed me this time around. I don't like the way that the characters – all the characters – lie so casually. For a book which revolves around the fight between good and evil, it's not always clear to me what Rowling sees as the difference between the two. I'm not complaining, of course, that the characters are themselves ambiguous and complex. Jane Austen's novels portrayed complex characters with very mixed motives – but the narrator's moral perspective is always clear: Emma Woodhouse is portrayed sympathetically, but the narrator leaves you no doubt as to her failings. But I don't get the same sense from Rowling that she's quite as clear about the moral problems with many of Harry's actions. If the narrator judges them at all, it is from a distance, and almost as an afterthought.

Perhaps I'm misjudging here: I've only read the books twice, after all, and it'll be another couple weeks before I can read the last one. But I get the impression that Rowlings' somewhat muddy moral perspective reflects, not surprisingly, the muddy waters of the 21^st century in which she (and all of us) must wade.

Pico Computing

I had a fascinating lunch today with a friend I met a couple weeks ago. He's a Christian, a graduate of Seattle Pacific University (where I taught briefly), and a hardware engineer at Pico Computing. Pico makes massively parallel FPGA computers that get used by anyone who needs to solve compute-intensive, data-light problems. (I've heard that certain government agencies, for instance, have those needs.) My friend is one of those quiet, humble, intense intellects that you don't run across very often: the sort of guy who enjoys discussing chip design as much as discussing C. S. Lewis.

We sat in an Indian restaurant in Ballard, just north of the Ship Canal, enjoyed a rare hot day in Seattle, and talked about life in a startup. Pico's about where Zango was at 7 years ago: they seem to have some very smart people there, and have the potential to tap into a big and growing market. (Not everybody who wants to model protein folding, or crack the latest cryptographical algorithm, has access to a network like folding@home or World Community Grid.) After lunch, we walked back to Pico's offices, and he walked me through their design process, and he showed off a prototype of their latest not-quite-launched product. It's an interesting gig.

Pico has some big competitors – Nvidia's Tesla product line is going after roughly the same market, with a very different technology (Graphical Processing Units) – but if they can get past the bootstrap stage, I think they've got an interesting future ahead of them.

Goat Rocks Wilderness

We're back in touch with civilization, after a very relaxing week spent mostly hiking and backpacking through the Cascades. Rather than bore everyone with the details of our trip, I'll just post a couple pictures from Goat Rocks and Mt. Hood.