It’s About Trust: Muglia’s Silverlight Screw-Up

Every year for the last several years, Microsoft has used their Fall PDC conference to tout (among other things) the improvements they’re making, about to make, or have just made to the Silverlight platform.  But over the last year, there’s been a fair amount of discussion in the Silverlight developer community about the future of Silverlight vs. HTML 5, given that many of the things that you need a plugin for will eventually be available in HTML 5 (specifically, video and smooth animations).  This has left folks investing in Silverlight (like myself) a little bit on edge.  Microsoft has a history of touting a platform, only to drop it when something bigger and better comes by.  This may be good for Microsoft, but it’s not so good for the folks who have invested hard in that particular platform.  My assumption, however, has been that with the advent of Windows Phone 7, which is based on an old but reasonably functional version of Silverlight, that Microsoft has lots of reasons to continue investing in and improving the Silverlight platform.

Still, I was surprised, and not in a good way, that the entire emphasis at PDC 2010 was on IE9 and HTML5.  If the word “Silverlight” crossed Ballmer’s lips at the keynote, I missed it. There wasn’t a single session dedicated to Silverlight (it got mention in a few), and after a regular drumbeat of 12-month releases, there wasn’t a single hint about what might be coming in Silverlight 5.  This worried me.

My worry turned to angst, and from angst to anger, when I read Bob Muglia’s comments about why Silverlight had been de-emphasized at PDC 2010:

“Silverlight is our development platform for Windows Phone,” he said. Silverlight also has some “sweet spots” in media and line-of-business applications, he said.

But when it comes to touting Silverlight as Microsoft’s vehicle for delivering a cross-platform runtime, “our strategy has shifted,” Muglia told me.

Silverlight will continue to be a cross-platform solution, working on a variety of operating system/browser platforms, going forward, he said. “But HTML is the only true cross platform solution for everything, including (Apple’s) iOS platform,” Muglia said.

Coming on the heels of Silverlight’s no-show at PDC 2010, this is clearly damnation by faint praise, and it’s spawned no end of tweeting and blogging and reporting about the future of Silverlight.  At the moment, there are lots of comments coming out from MS folks that Muglia misspoke or was misinterpreted, and that there’s going to be a walk-back of some sort in the near future. And if you’re a Silverlight developer, there remain plenty of reasons for hope, besides the obvious one that Windows Phone 7 is clearly a huge bet for Microsoft.  After all, as lots of folks have pointed out, the HTML 5 standard won’t be finished for maybe another 10-12 years. No current browsers support even the full set of tags that have been defined so far, and the ones that do don’t support them all in the same way.  The fact that the HTML 5 committee can’t even agree on a video codec standard is the classic example of this, but there are others.  And even when browsers have caught up, the tooling behind HTML5 simply isn’t there yet.  For starters, none of the MS tools can output HTML5, and even when Blend does finally support it, the various debugging environments for JavaScript/JQuery have many years of improvement ahead of them before they’re even within shouting distance of C# (which is, in my opinion, the best language ever, running in the world’s best IDE by a significant margin).  It’s precisely these things which make it so odd that Muglia has declared the Silverlight vs. HTML5 debate over – and which make it so likely that Muglia is going to “clarify himself” in the near future.

Still, I’m worried.  I love C#, and I love the fact that I can use Silverlight to bring C# to client-side web applications.  It makes writing the UI logic of an application dramatically easier than trying to code it in JavaScript.  But I’ll confess that I like Silverlight less than I like the idea of Silverlight.  The tooling remains poor by Microsoft standards.  XAML is a verbose, ugly language that is, at best, an awkward fit onto the simplicity and elegance of C#.  Blend is powerful, but the UI is a royal mess.  The XAML editor in VS2010 is light-years ahead of VS2008, but it remains slow and clunky.  Debugging databindings has always been a significant PITA, and it shouldn’t be.  Unlike Flash, Silverlight doesn’t even support streaming webcams and microphones.  Yes, it gives you access to the raw bits coming out, but if you want to do anything with those raw bits, you have to write your own framework to compress and stream them somewhere. God help you if you actually want advanced DSP like echo cancelling.  (I just got echo cancellation working on Alanta’s platform, by the way.)  It’s really nice that SL4 gave us access to the Net.TCP WCF binding – but it does so without security or authentication.  And of course, Silverlight has only reached about 60% of desktops, vs. some 96% for Flash.  Requiring a third of your audience to download a plugin can be a significant drag on your own application’s adoption.

In other words, Silverlight is still a pretty raw platform.  Version 4 was probably the first mature release, but the ecosystem still has a long way to go.  Nevertheless, I’ve felt comfortable investing in Silverlight despite its rough edges because I believed that Microsoft was committed to fixing the multitude of problems that remained.  I saw – and see – the potential that Silverlight has to be a premiere application platform. It could have a truly awesome design and development environment.  It could run faster than HTML and JavaScript could ever hope.  It could give you access to the best parts of the CLR on every desktop.  It could let you run raw GPU-accelerated code.  It could let you really achieve “write once, run everywhere (that Jobs will let you)”.  It could iterate orders of magnitude faster than the W3C or WHATWG could possibly hope to.  It could develop into an awesome, premiere realtime collaboration environment that users love because they don’t need to know anything about it and developers swear by because they do.

It could.  Silverlight could do all these things.  It doesn’t do them yet.  And the only way that it might ever do these things is if Microsoft puts its full weight and authority and development resources and marketing muscle into play.  Silverlight has always been a hard sell, an also-ran in a crowded space dominated first by Flash and now by HTML 5. But it has the potential to punch its own weight if Microsoft is committed to fixing all the things that remain wrong with it.  And that’s precisely what Muglia seems to have committed Microsoft to not doing.  Yes, there’s going to be another version.  (But there’s no point to Silverlight unless there are going to be five more versions, at a minimum.)  Yes, it is the platform for Windows Phone 7.  (Which has precisely, let me check, oh, umm, zero market share, in a market dominated by companies which have handily out-competed Microsoft for years now.) Yes, it can still fill a couple “sweet spots” in media and line of business applications.  (But these “sweet spots” can be addressed just fine with Flash and .NET client apps.) 

But here’s the real issue.  I’ve spent two years learning Silverlight because I trusted Microsoft to bring me a good return on my investment.  But unless Microsoft makes it clear very shortly that Muglia was simply wrong (not just “unclear”), you’d be foolish to make that same investment now.  WP7 looked mildly interesting because I might be able to leverage the Silverlight ecosystem on a new platform. But if that ecosystem is going to get de-emphasized, what mild interest I had in WP7 vanishes.  What Muglia has said, in effect, is: Silverlight is a bad bet for your technology future.  You shouldn’t waste your time learning it. Go learn HTML5 and JavaScript and JQuery instead.  And that’s a real problem for Microsoft. Because if I’m going to be limited to HTML5 and JavaScript, and can’t use C#, I don’t feel any particular need to do my coding on the Microsoft stack.  IE9 looks interesting, but at the moment, IE has had three versions in a row which are slower and uglier and less standards-compliant than any other browser on the market.  MySQL is mildly interesting, but Hadoop and CouchDB and MongDB look pretty darned fascinating.  Linux has clearly come into its own as a server platform.  I’ve had a hankering to learn Python for a while now (PHP is popular, but ugly, and aesthetics count for a lot with me), and RoR sounds like a pretty sweet ecosystem as well.  Muglia has just made a very persuasive argument that these technologies are going to be more important to my future than anything coming out of Redmond.

I hope he’s wrong.

Sermon: The Stories God Tells

(Note: Every now and again, I’m asked to preach at a local retirement center.  I haven’t had any blog updates for a while, so I figured I’d post the sermon that I preached yesterday.)

Introduction

For a long time, one particular aspect of the story of Stephen has puzzled me greatly. I suppose we all know who Stephen was – the first martyr of the church – and we’ve heard his story many times since our Sunday School days. As you will likely recall, Stephen was a man full of faith and of the Holy Spirit who had been selected by the Apostles as a deacon, a servant of the church. His main responsibility was to administer the church’s care and feeding of the poor and needy, to tend for the widows who had nobody else to help them. Despite this “day job”, however, Stephen found time to do some preaching and evangelizing on the side, and it wasn’t long before his outspoken proclamation and defense of the apostolic message landed him in trouble with the same authorities who had quite recently been responsible for Jesus’ death.

The accusations leveled against Stephen were certainly plausible enough. His accusers had two primary charges: they claimed first that he had blasphemed the law of Moses; and second, that he had blasphemed the holy temple in Jerusalem. Luke identifies those making the accusations as “false witnesses”, but this may be a critique more of their motives than of the actual content of their testimony, because what they accused Stephen of saying did not go very far at all beyond what Jesus Himself had to say.

But here’s the part that’s always puzzled me. Imagine for a moment that you’re Stephen, called before the Sanhedrin, on trial for your life. What would you say? How would you respond to these charges? I suppose one option would be to deny that you had actually said the things of which you were accused. Another would be to clarify your earlier statements. And I suppose a third, gutsier alternative would be to repeat your controversial views, to re-assert emphatically what had already gotten you into trouble. But Stephen doesn’t do any of these things. What he does do is something else altogether.

The Jewish Story

Stephen tells a story. And not even a new story. He starts by telling the oldest story of them all, a story that everyone listening would have known since childhood. Stephen tells about how God called Abraham out of the land of Ur, and promised him, against all odds, a land that his descendants would inherit. He tells, at some length and in great detail, how God brought first Joseph and then Jacob down to the land of Egypt, then how God brought them back out again through the leadership of Moses. He tells how the people rebelled against Moses, and were forced to wander in the wilderness for forty years, before God finally brought them into the land He had promised Abraham, generations and generations before. And he tells about how Solomon built a temple for God, before the people rebelled once again, and were led into exile in Babylon.

As I said, it’s always puzzled me why Stephen frames his reply in this way. On the surface, he doesn’t seem to be saying anything new, and certainly isn’t telling anyone in the Sanhedrin anything they don’t know quite as well as he does. But Stephen, we must remember, was a Jew, and being a Jew means that you come from a long line of professional debaters. Stephen was the heir to centuries of subtle and sophisticated arguments. The rabbis didn’t need to draw out every train of thought, they didn’t need their opponents to make every implication perfectly clear. When you’ve been arguing about the Law for as long as the Jews have, a nudge, a hint, an allusion is sufficient to get your point across. And Stephen’s speech contains quite a few hints and number of not-so-subtle nudges. So as I read through this chapter recently, I noticed that Stephen tells the story in a very particular way, simply through his selection of incidents and passages from the Old Testament. And as I thought about how he was telling the story, I began to piece together his motives for conducting his defense in this way.

Reason #1: Critique of the Temple

Stephen’s first motive for conducting his survey of Jewish history was to point out that the temple was not perhaps all that his accusers thought that it was. For starters, he says, the temple was not a part of the most ancient Jewish traditions. It played no part in the religion of Abraham, Isaac, Jacob, or Joseph. God commanded Moses to build, not a temple, but a tabernacle: not an expensive building which was confined to one place and which could never move, but an inexpensive tent, which was always on the move, a mobile sanctuary that would serve as a constant reminder that Yahweh was not the God of a special place, but the God of a special people. God commanded Moses to build a tent, a σκηνη; David wanted to build him a dwelling place, a σκηνομα: but what Solomon built Him was a temple. At one point, Stephen quotes an accusation made by the prophet Amos that the Israelites had “built a tabernacle for Moloch”, one of the ancient and brutal gods worshiped by Israel’s neighbors. In quoting this accusation, Stephen comes perilously close to saying that the temple itself was pagan. The burning bush was holy, Stephen says; the tabernacle was holy; the temple, by implication, not so much.

Reason #2: Critique of the Jewish Leadership

Stephen’s second motive in retelling the story of the Jewish people was simply to point out that what had happened to Jesus was not at all out of the ordinary. Stephen knew that the real motive behind his trial was not whether he had said ‘x’ or ‘y’ about the temple, but rather his contention that the Jewish leaders had killed the very Son of God. Through his retelling of the story of Jewish history, Stephen made it clear that Jesus was not the first hero chosen by God to be rejected by the establishment. Joseph was sold as a slave by his brothers. Moses had to flee Egypt when a fellow Israelite threatened to turn him in. Later, the Israelites again rejected Moses’ authority, and were forced to spend 40 years wandering in the wilderness as punishment. And even after they had taken possession of the land God had promised to Abraham, the people continued to rebel against the prophets, until the whole nation came to disaster. In other words, Stephen is saying, the fact that the Jewish leaders wouldn’t accept Jesus’ message proved nothing about whether Jesus’ words were true. “Which of the prophets did your fathers not persecute?” Stephen asks, and the story that he has just told makes the answer uncomfortably clear.

Reason #3: Affirmation of the Jewish Story

But I think the final reason for Stephen’s long historical discourse is to make it clear to his accusers that, like them, he was himself a faithful Jew. He recited the long story of Jewish origins, their failures featured prominently against God’s faithfulness, so that his judges would know that he was one of them. Stephen told a story because stories define who we are. All of us, you and I as much as Stephen and the Sanhedrin, know who we are because of what we have been. I know that I am a father, because I can tell you the story of the long years when my wife and I tried to have children and failed; and how God worked to bring three children into our life who are as amazing as they are exhausting. That story of past sorrow and present joy and ongoing toil tells me who I am; and in the same way, the Jewish story of sin followed by slavery followed by rescue followed by more sin, over and over, told the Jews of the first century who they were.

Nor have they yet forgotten these stories: every year, at Passover, the Jewish people tell each other, We were slaves to Pharaoh in Egypt; and the Lord brought us out of Egypt with a mighty hand. In the same way, Stephen begins his speech by saying, “Brothers and fathers, listen to me! The God of glory appeared to our father Abraham while he was still in Mesopotamia.” These words give Stephen’s message and his trial a context: they place him and his hearers within the great story of God’s action and self-revelation throughout history. They show that Stephen worshiped the same God, revered the same Scriptures, and followed the same teachings. For all Stephen’s critique of the temple and of the current Jewish leadership, he wants to be clear that his faith in Jesus makes him nothing more and nothing less than a faithful, observant Jew. Stephen recounted the story of the Jewish people to the Jewish leaders because it was not just history: it was his story.

The Jesus Story

We all have our stories. I love to sit and listen to the ones my grandfather tells about his childhood in Montana, how he survived the Great Depression, about the winter he spent gold mining on the Salmon River in Idaho. He tells great stories about how he eloped with my grandmother the week after she graduated from high school, about how they moved to Oregon and bought property and built a house and raised a family. I love stories, and our family is famous for telling them.

But sometimes we find ourselves in stories that we didn’t expect. My grandfather’s story isn’t over yet, but it’s taken a hard turn. About fifteen years ago, he lost his eyesight. Five years ago, he broke his ankle, and began using a cane. Three years ago, he lost his beloved wife of 70 some years to cancer. Two years ago, he broke his hip. Last year, he had to leave the property he had owned for 60 years. The brilliant and brash young man who conquered his wife’s heart as easily as he conquered the Salmon River is now lonely, crippled and blind. Every day, he hurts. Every day, he cries. Last week, he asked me, “Ken, why won’t God just take me home?” People like to hear his stories about how, during the Great Depression, he found work and a wife and a home. Nobody wants to hear stories about how he has now lost his work, his wife, and his home. People like to hear about how he cheated death hunting in the mountains. Nobody wants to hear about how he now longs for death to take him. Not all of our stories have the simple, happy endings that we would choose.

Stephen found himself in the middle of just such a painful story. Stephen was a young, ambitious, charismatic follower of Jesus. He was intelligent, well-spoken, and persuasive. He was full of confidence and excitement about the Lord in whom he had come to believe. Yet the situation in which he found himself was one nobody could relish. His confession of faith in Jesus, his rejection of the Jerusalem temple, his criticism of the Jewish leadership, was all quite likely to lead to his death; and he knew this. So why did Stephen speak so boldly and without fear? Stoning is a horrible death. So why was Stephen so reckless with his life?

I think it was because Stephen recognized the story that he was in. He had seen this happen before, and he knew the ending. Everything that was happening to him had already happened to Jesus. Like Jesus, Stephen had been accused of wanting to destroy the temple and to abolish the Law of Moses. Like Jesus, Stephen had been hauled in front of the Sanhedrin where he was confronted by false witnesses. Like Jesus, Stephen was scorned by the High Priest, then looked up to heaven and spoke of the coming of the Son of Man. Like Jesus, Stephen commended his spirit into God’s hands. And like Jesus, Stephen asked God to forgive the very people who were killing him.

Stephen, you see, is not just the archetype of every Christian martyr. Stephen is the archetype of every Christian. Not every Christian will be hauled before the Sanhedrin: but every Christian is called to be crucified with Christ. Jesus said, “If any man would be my disciple, let him first deny himself, and take up his cross daily, and follow me.” Paul said, “I am crucified with Christ; and nevertheless I live, yet not I, but Christ who lives inside me.” You see, each of us is living out a part of the story of Jesus. At times, each of us finds ourselves praying in Gethsemane, wincing under the lash, standing silent before our accusers, stumbling down the Via Dolorosa. Each of us carries within our hearts a small sliver of the cross on which God died.

I don’t know your story. Maybe it includes a divorce. Perhaps you’re estranged from your children. Maybe you’ve watched your husband die. Maybe you’ve had to move out of your house, feel your health decline, your eyesight fail, your mobility and independence fade. Whoever we are and whatever we’ve done, there are parts of our stories that we can only wish were different. But it’s precisely when we walk through these parts of our stories that we walk closest to Jesus. The scenes from our stories which are the most painful to endure are the parts which most closely resemble the story that God told when He came to earth. I don’t know why God hasn’t called my grandfather home yet: but I know that even in his present suffering, he is partaking of the life and the death and the resurrection of our Messiah.

And if our story really is the story of Jesus – if all our pains and sorrows and disappointments are really the hand of God working through us – then there is another conclusion we must also draw. We are not just in God’s story: God is in our story. We are not just carrying the cross of Christ: Jesus is also carrying our cross. We may sometimes be blessed to play the role of Simon of Cyrene, and suffer with our Lord through his tribulations: but more often, I expect, we find Jesus suffering with us through our tribulations: alongside us, inside us, lifting us up, giving strength to our stumbling feet, courage to our faltering hearts, steel to our fading wills. For wherever we are, our Lord has been there too; and whatever our story, it is the story of our Lord. And if we turn our eyes to heaven, like Stephen, we may yet see the Son of Man coming in glory, with all his angels.

Valid Reasons for Using Threads

We had an interesting debate this morning in our daily Alanta standup meeting.  We’re still in the process of implementing our proprietary media server (long story), and I’d noticed during a recent debugging session that with three clients connected, our media server had spawned something like 118 different threads.  That didn’t seem right to me, so we had a long discussion about whether we actually needed that many threads and whether it represented a significant problem for our architecture.  The answers that seemed obvious to me – no, we didn’t, and yes it was – weren’t as obvious to everyone else.  We were eventually able to come to a consensus about how to proceed, but I thought I’d jot down some thoughts about when it’s appropriate to spawn threads, when it’s not, and some best practices about how to achieve the right balance in your architecture.

Why you don’t want a lot of threads

First things first: you really do want to minimize the number of threads in your application. One well known reason for doing this, of course, is that there’s overhead every time the processor has to switch between threads. The processor has to save the state of the currently executing thread, then load the previously saved state of the next thread (and if the threads are in different processes, it needs to reload the virtual address translation tables as well).  This results in a practical limit of about 30 active threads per process per processor.  Any more than that, and you’ll be spending more time dealing with thread overhead than with getting anything done.

However, the argument advanced this morning was that if the threads weren’t active, i.e., were just waiting to be signaled, then they weren’t chewing up CPU cycles with context switches, and hence even very large numbers of them were harmless.  The problem with this argument is that threads have a significant virtual memory overhead even when they’re inactive.  As Raymond Chan has pointed out, each thread you spawn needs its own stack, and by default, the stack size on a Windows box is 1 MB.  This means that every time you spin up another thread, you’re adding another another 1MB to the virtual memory your process is consuming.  So on a 32-bit machine, since 2GB is generally reserved for the kernel, the maximum number of threads that any one process could reasonably hope to create is ~2000.  (On Linux, the default stack size is 2 MB, which introduces a practical limit of ~1000 threads per process.)  Of course, this is only on 32-bit machines, and you can get away with quite a bit more on 64-bit machines, but the point remains: every thread chews up allocatable memory, and quite a lot of it.  One caveat: each thread chews up 1 MB of virtual memory, not physical memory (at least, not right away).  In other words, if your process creates 500 threads, you won’t necessarily see your process chewing up 500 MB of physical memory in task manager.  That’s because the pages in physical memory that back your stack aren’t allocated until they’re referenced (as described here).  So you can’t tell you’re running low on virtual memory until you’ve actually run out of address space, and the first indication you’ll have that you’re about to die is a strange and otherwise inexplicable Out of Memory error.

I should note that it’s theoretically possible to create a threading model that doesn’t suffer quite so badly from the stack problem noted above.  Rob Behren and some compatriots wrote a great paper on the topic, in which they demonstrated a system running with over 100,000 active threads.  Unfortunately, their recommendations haven’t been widely adopted by tool and framework vendors.  For instance, they point out that a compiler/linker which dynamically adjusted stack sizes would lower the memory overhead for each thread pretty substantially.  This is true, but I’m not aware of any mainstream compilers that actually do this.

So in brief, if you’re writing papers for a conference, create as many threads as you want. If you’re writing software with mainstream compilers, frameworks and operating systems, limit the number of threads you create.

Three reasons for creating threads

So it’s pretty clear why we don’t want a whole bunch of threads.  Similarly, there are a pretty limited number of reasons for ever creating threads in the first place:

1. To take advantage of multiple processors.  This is the classic reason.  If your software will typically be running on a multiprocessor or multicore machine, and you really do have more work than one processor can handle in a timely fashion, it makes sense to split that processing up into multiple independent threads of execution.
   
2. To move specific processing off of the UI thread.  This is a much more pragmatic reason.  In the normal Windows world, you typically create threads so that a given background process doesn’t choke your UI. In the Silverlight world where I’ve been living lately, this is less of a problem: Silverlight forces every sort of IO into an asynchronous pattern, and that tends to keep your UI thread from blocking.  But the opposite can sometimes be an issue: the UI thread can get so busy that you need to spin up background threads so important tasks (like audio encoding or decoding) can get dispatched quickly enough.  But the idea is the same: you want to keep your users from getting grumpy because important parts of their application appear to have ground to a halt.
   
3. To simplify complicated asynchronous calling patterns.  Unless you actually need to take advantage of multiple processors, it’s almost always possible to get the effect of threads through a completely different mechanism, namely, using events to pass control from one part of the program to another.  (This is called the Lauer/Needham Duality.  In a famous 1979 paper, Lauer and Needham showed that “message-oriented” and “procedure-oriented” systems – read event-driven vs. multithreaded – were duals of each other and hence were logically equivalent architectures.)  But although thread synchronization is difficult to get right, it can be even more painful to write a program using entirely asynchronous calls.  The APIs for doing so are often complicated and obscure, and they require that you split your program’s logic across various artificial boundaries.  Depending on the complexity of your application, they may also require a that you implement a cooperative multitasking model, where any given function can be requested to yield to other functions, and only later pick up where it left off.  These sorts of issues can make debugging and maintenance quite difficult.  Threads are plenty complicated, but apart from the places where you start a thread, wait for a thread to finish, or lock some resource, multithreaded code looks reasonably similar to synchronous single-threaded code.  And that’s almost always a good thing.

If none of these three reasons apply, you don’t need to create threads.  It’s as simple as that.

Maintaining the right balance

In the real world, you’re typically going to need to balance legitimate reasons for threads against the virtues and vices of asynchronous programming.  But the basic guidelines for scalable, high-performance systems are reasonably clear:

1. Use asynchronous IO with callbacks or events when you need to interface with the outside world.  If a client connects to your server, you may be tempted to spin up a thread dedicated that client, and then let everything within that thread happen synchronously.  In other words, you create a thread for each client that connects, and then in that thread, you send some data to the client, block until the client sends some back, and so forth, until finally the client disconnects, at which point you terminate the thread.  Certainly that’s the simplest model, and if you’re confident that you’ll never have more than a couple dozen clients connected simultaneously, it may be a reasonable approach.  But as I discussed above, it’s surprisingly memory intensive, and a load as small as a few hundred clients could bring your server to its knees (even if the server’s CPU was completely idle).  A better approach is to use the asynchronous version of whatever network API you’re using, and simply handle whatever processing needs to be done in the callback.  On the native Windows API, you do this using I/O completion ports; on Linux, you can use select() or poll() or (better) aio_read() and its compatriots.  If you’re using C++, the best way to do it is probably to use the boost::asio namespace; and if you’re using .NET, you should use the asynchronous versions of whatever piece of the framework is applicable. (There’s a reasonable, if incomplete, example of how to use Socket.BeginAccept/Socket.EndAccept here on the MSDN site, for example.)  Depending on how your framework is implemented, this may also be a good place to use a threadpool: instead of handling the event on whatever thread it comes in on, hand it off to your threadpool, and go back to waiting for the next signal/callback/event.
   
2. Don’t try to do everything via asynchronous message-passing. Threads are really a pretty helpful abstraction.  I’ve seen high-performance servers implemented entirely using asynchronous, event-driven code, but they’re a pain to troubleshoot and debug.  The vnc-reflector project on SourceForge is a good example of this.  It’s very performant, even though it runs on a single thread.  But it’s a pain to debug or modify, because so much of the logic consists of passing static function pointers around in a very complicated main() loop.  Asynchronous is good, but don’t bother trying to make everything asynchronous.  If you can simplify your code by kicking off a preemptively multitasked thread, rather than implementing a whole bunch of your own mechanisms for cooperative multitasking, by all means, kick off the thread.
   
3. Use a threadpool whenever you don’t know precisely how many threads you’ll be creating.  If you want to dedicate a thread to some specific task that can run in the background, there’s not much point to using a threadpool.  But if you need to handle an unknown n number of clients, don’t try to spawn a dedicated thread per client.  Use a threadpool instead, and service the requests out of the pool.  On a side note, I’ve known folks who were tempted to write their own threadpools.  Granted that in their simplest versions they’re not all that complicated, but still: unless you have a damned good reason to do so, don’t.  Recent versions of Windows have their own built-in threadpool APIs; .NET has a reasonable ThreadPool in the System.Threading namespace, or you can use Ami Bar’s SmartThreadPool; and although Boost strangely doesn’t have one, Philipp Henkel has made a boost::threadpool available that works cleanly with Boost threads.  And there are tons of other examples.  It’s just silly to try to reinvent the wheel when there are so many thorough implementations available for basically the cost of browsing to a website and looking at some sample code.
   
4. A good compromise may be to use a small number of queues to move data through your system.  Queues are a classic asynchronous mechanism, but they can also be used effectively with threads and threadpools.  For instance, one of the functions of a media server might be to mix audio coming in from clients in a conference call, and then send the result back to each.  One of many possible architectures would be to implement this as a series of queues with a threadpool servicing each of the queues.  One threadpool would parse the incoming data and then place the compressed audio data in a queue to be decompressed; a second set of threads would decompress the audio and then place it in a the next queue to be mixed; a third set of threads would mix the decompressed audio and then place the results in a final queue; a fourth threadpool would then construct the mixed audio data into packets and deliver them back to the clients.  You could also implement this same basic architecture with more or less queues, depending on other constraints and design goals.  And I should also note that the only reason for using threadpools rather than individual threads would be to take advantage of multiple processors.  If you only had four threads servicing the three main queues, but were running this on an eight-processor box, you’d effectively be leaving four of the processors unused.

If you take this last approach, however, just make sure that your queues and especially your threadpools are limited in number.  The problem that kicked off this whole blog posting is that our media server currently creates a separate set of queues for each connected client, and then creates a separate threadpool for each queue.  The result is that each connection spawns 37 new threads.  I don’t think it’ll be too painful to fix this: but it does need to be fixed.

Rained out on Labor Day (again)

We tried to go camping for Labor Day (again), but like last Labor Day, we got rained out, gave up and came back home early.  Still, we had a reasonably good time while we were there.  

http://picasaweb.google.com/smithkl42/20100904LaborDayCamping#

Our weekend started on Friday afternoon, when Caedmon and I drove to the Middle Fork Campground, ahead of everyone else, so that we could be sure to get our spot.  

The next morning, Saturday, Caedmon and I tried our hand fishing, but didn't have any luck.  Probably would have done better with flies, but it's difficult enough to get a three-year old to handle a spinning rod without tangles and snags everywhere.  I'll leave the fly-fishing lessons for another day.

After fishing, but before lunch, Caedmon and I went on a hike that ended up being longer than we expected.  We got about a mile and a half out before finally turning around.  Caedmon did great, but was a little disappointed that we turned back before the end of the trail.  I explained to him that we needed to go back to camp so we could have some food, before we got all tired and grumpy.  He wanted to know why we might get tired and grumpy, so I said, "Well, it's our brain's way of telling us that we need to get more food."  A while later, when we were almost back to our campsite, I was carrying him on my shoulders, and I heard him whisper to himself, "It's all right, brain. You'll get some food in a couple minutes."

Later Saturday afternoon,  Galena, Brendan, Calista and Kirstin McGhee got there.  Kirstin's been helping Galena with the kids most of the summer, so she knows them very well, and when her boyfriend, Tony, decided to spend Labor Day weekend in Las Vegas, she asked if she could hang out with us.  That gave us the excuse we needed to try camping with three such small kids.  And she was very handy to have around -- you don't want young children outnumbering adults on a camping trip.

Brendan loved the ice-cold, freezing water.  Must help to have a layer of blubber for insulation.

This was Calista's first camping trip, and she seemed a little stunned by it all.

That evening, after dinner, we made s'mores, which were a substantial hit, not least because they gave Caedmon an excuse to wave a flaming torch around.

It started sprinkling just after we put the kids to bed, and by the time we were in bed ourselves, it was coming down pretty hard.  It rained all night, and it turns out that 3-season tents are really preferable to 2-season tents in situations like that.  I had Caedmon and Calista in my tent, and we were all dry, but Galena, Kirstin and Brendan were all pretty wet by the time morning rolled around.  So we made a quick breakfast, cleaned up as best we could in the rain, and headed home.  I like to think of it less as "giving up" than "staying married".

Rainier Summit via Kautz Ice Chute

Late last December, my pastor, Charlie Swartz, pulled me aside and asked if I’d be interested in climbing Rainier with him.  I’d tried twice before, succeeding once, but I didn’t feel comfortable enough with my skills to lead a team myself.  Consequently, I asked my cousin Brian (who has 10 Rainier summits and an Everest summit on his resume) if he’d be interested in leading the team.  He was, and we started planning.

The only route that I’d done before was the Schurman/Emmonds approach on the other side of the mountain.  This time, however, my cousin had talked us into trying something a little more challenging: the Kautz Ice Chute, a couple miles climber’s left of the standard Disappointment Cleaver route.  Gauthier’s book describes it as a Grade II/III, basically because of the difficulty of the ice chute, which has a one 50 degree pitch, and a second 60 degree pitch.  That seemed challenging for a bunch of newbies, but folks liked the idea, so we went with it.

Over the next six months, probably two dozen people “joined” our team and then backed out, but when we finally left the Paradise parking lot on Wednesday morning, we had 11 folks hoofing it up towards Pan Point.

Unfortunately, the two most interesting members of our party had to back out first.  Lhakpa Sherpa has five Everest summits under his belt, but his wife Maya wasn’t feeling good at elevation, so they peeled off after crossing the Nisqually Glacier, and headed back to Seattle, which left nine of us heading up out of the Nisqually and up onto Wapowety Cleaver.

Once we were on Wapowety Cleaver, we crossed over the Wilson Glacier then began working our way towards the base of the Turtle.

What with one delay or another, along with the logistics of keeping a large group moving, we ended up getting to camp later than we had hoped, about 7:00 pm.  We setup camp at about 9500 feet, right around the base of the Turtle, ate dinner, watched the sunset, and then headed to bed.

Our original plan had been to take Thursday as a rest day, and then summit and descend on Friday.  However, most folks were feeling fairly good, so we decided to try for a 3:00 am start on Thursday morning.  The result was that I got four hours sleep, and pretty much everyone else got less – several folks didn’t get any, and two members of our team were feeling too exhausted to try for the summit.

The seven remaining members of our team made it to the top of the Turtle just as the sun was coming up.  We roped up, adjusted our crampons, and headed onto the Kautz ice chute.

Luckily for us, the route was in excellent shape.  Usually, by mid-July, the Kautz is more ice than snow, but we still had a great snowpack, so even the two steep pitches didn’t give us much trouble going up.  We eventually made it up off the Kautz, and at about 13,000 feet crossed over to the Nisqually.

Unfortunately, this is about when one of our party began to suffer some of the symptoms of AMS.  Stephanie Spence was an experienced mountaineer, and had climbed at the 14,000 foot level in the past, but this time she began throwing up at around 13,000 feet.  She insisted on continuing, but by 13,500 she was done.  Graciously, Eric Dalzell, another Everest veteran, volunteered to give up his summit and descend with her. 

The last 1000 feet were brutal.  It was a step, and then two breaths, and then another step, and another two breaths, the entire way.  Anne Timblin declared that this was significantly harder than the half-ironman she’d done the year, and that she should have turned back.  But she kept going, and so did the rest of us.

Finally, a little after noon, we crossed over the summit rocks and stood on the summit crater. 

The weather was still perfectly clear, though the winds were beginning to pick up.  Several of us hunkered down in the rocks near the steam caves and took long naps.

After an hour and a half or so on the summit, we began descending, which was a long, slow process.

The trickiest part came while descending the top pitch of the Kautz Ice Chute.  The snow was pretty soft, which made for good down climbing, except for a 10 yard stretch of ice.  My cousin Brian had four of us on belay as we were descending the chute, so we were fairly well protected, but we were still trying to be careful.  I got to the icy stretch, and between general fatigue, nervousness, and inexperience, I couldn’t keep my crampons in the ice.  Unfortunately, although I had slowed down, the climber above me hadn’t, so there was a significant amount of slack in the rope when my crampons finally popped out and I went for a tumble down the slope.  I yelled “Falling!!”, and luckily the folks above me had enough time to go into self arrest.  I ended up sliding some 20-30 feet, and gave the rope a good yank when I finally hit the end, but they held on, which was good, as I didn’t have any desire to test the strength of the pickets up at our belay station.

After we’d reached another belay station towards the bottom of the pitch, we unroped, and then I belayed my cousin down.  The belay was mostly just for appearances, as we hadn’t placed any protection on the way down (Eric was carrying our extra pickets when he left to accompany Stephanie down the mountain): if Brian had fallen, he would have swept past us and then to the end of the rope before we could have stopped him.  Luckily, he was able to bypass the icy patch, and made it safely down to our belay station. 

The only other interesting part of our descent came a couple hundred feet further down the chute, towards the bottom of the lower pitch.  I was kickstepping my way down the chute, facing into the slope, when I felt the snow give way under me.  I went into self-arrest, but not before I found my legs and chest dangling inside a large crevasse.  I elbowed my way out – style points didn’t seem important at that point – and then our team carefully navigated around the crevasse, and we continued our descent to camp without any further incidents (other than a nice glissade down the Turtle).

The next day, we made ourselves breakfast, then headed out. 

The snow was soft enough that we were able to glissade maybe 75% of our way down to the Nisqually.

And damn, it felt good to get back to the parking lot.

I’ve posted the full assortment of the pictures I took here.  Anne’s pictures are on Facebook here.

Some lessons I learned:

1. If we ever do a newbie trip again, we’ll plan for four full days on the mountain.  If we’d had a rest day between our ascent to high camp and our summit day, probably all nine remaining members of our party would have been able to summit, instead of just five of us.
   
2. Folks without ice climbing experience have no business being out on the Kautz Ice Chute.  We got lucky in that the ratio of snow to ice on the Kautz was roughly 20:1, but I get the impression that’s not at all typical for this time of year.  If it had been a normal year, we would have struggled a great deal more.  Before I do that route again, I need to get some real experience on hard ice.
   
3. Training with a heavy pack at altitude is the best conditioning, and I hadn’t done enough of it.  I’ve been training regularly and hard for six months, and I’ve made trips up to both Muir and Schurman this season, but I could easily have used another two 10,000+ climbs before trying for the summit.
   
4. Large groups travel slowly.  A team of 12 people is just too large, and moves, well, glacially.  Next time, I won’t go with a group larger than six people.
   
5. I need a smaller camera than my DSLR to take on these trips.  Not only was my Pentax K10D heavy (I didn’t need the extra two pounds), but it’s awkward, with the result that I had to leave it in my pack most of the time.  It doesn’t matter that an SLR takes better photographs if I can’t get the picture because I can’t get at the camera.
   
6. Once again, I was carrying too much: my pack weighed nearly 80 pounds out of the parking lot.  Options for dropping weight include switching over to a jet boil stove, taking less food (I had ~10 pounds left over), and carrying less water (see #7).
   
7. I need to figure out how to drink less water.  During our two climbing days, I drank probably twice as much water as anyone else – nearly six liters on each of our climbing days.  I sweat a lot, but still, this seems excessive.  My suspicion is that I was probably suffering from hyponatremia, and that some Gatorade mix, or even just some Nuun tablets, would have made a significant difference in the amount of water I felt I needed.
   
8. The Asolo AFS Evoluzione climbing boots that I rented from REI were worthless.  They have no flex in them whatsoever, and while they might be good for technical ice climbing, they’re vastly inappropriate for general-purpose mountaineering.  Specifically, they’re nowhere near as good as the Koflachs that REI used to rent.  It was like climbing in plastic ski boots: I ended the climb with large painful blisters and bruises on the front of both shins, and numerous places elsewhere on my feet.  If the Asolo’s are still what REI has available the next time I head up a peak, I’ll either have to bite the bullet and buy my own boots, or rent elsewhere.

Major Problem and Major Progress

The Major Problem

Brendan has been very excited about pictures lately, especially when they're hanging on the wall.  Last week, when Galena’s parents, John and Sue, were up visiting, John built an oversized picture frame for us that we could place at toddler eye-level. 

This morning, during a temporary lull in the insanity, Galena and I decided to mount that picture frame.  It was big enough that I figured we should attach it to the studs in the wall, so I used my handy-dandy stud finder to locate the first stud, and with the help of four toddler hands, managed to pound in the first nail.  While I was looking for the next stud, Galena asked, "What's that noise?"  With three kids under three, there's a lot of strange noises in our house, and I was preoccupied with keeping Brendan's hands off the stud-finder, so I didn't pay much attention to her question.  But she repeated it several times, and then suddenly yelled, "Ken, we've got a major problem.  There's water coming out of the wall!"

And there was.  From right behind the frame, about where I'd put the first nail, there was a small but steady stream of water dripping down onto the floor.  I realized almost immediately what had happened: I must have pounded the nail through a pipe in the wall.  Not knowing what else to do, and wanting to get a better look at the source of the leak, I pulled the frame away from the wall, pulling the nail out along with it.  That was when the real fun started, as a spray of water emerged from the wall powerful enough to drench the other side of the kitchen.

I don't remember everything that got said in the excitement that followed, but suffice to say that as I was running around trying to find the water main, the stream turned too hot for Galena to block with her bare hands.  She temporarily abandoned her attempt to contain the spray, and scooped Brendan up to get him out of the way of the increasingly hot water.  (Brendan exhibited extreme consternation at being manhandled in this way: he'd been enjoying himself tremendously.)  Caedmon, meanwhile, had run into the family room and was crouched out of the way of the spray, yelling excitedly, "Major problem! Major problem! MAJOR PROBLEM!"  Galena grabbed my Goretex jacket from the back of a chair, and with that in front of her, waded back into the scalding stream of water.  As this was happening, I was busy running in and out of the house in my bare feet, looking for pliers, and checking on results, as I first managed to turn off the gas to the entire house, and then the gas to the hot water heater, neither of which was as helpful in this situation as you might think.  Finally I found the right valve, and I heard the gushing from inside the house subside.

The plumber will supposedly be here sometime this afternoon.  I'm guessing if we're lucky, this will only cost us $1000.  In the meantime, the floor is covered with towels soaking up the water, and Caedmon has asked for the "major problem" to be re-explained to him at least 50 times.  "Why major problem, Mommy?"  It's a question I'd like to know the answer to myself.

The Major Progress

This is almost anticlimactic, but Brendan took his first consecutive steps today.  After we'd managed to get all the water soaked up, we fed the kids lunch.  Since the water is still turned off, I wiped Brendan down afterwards with a washcloth, and set him down on the floor.  Without even thinking, he took five steps in a row over to a chair and grabbed onto it.  Galena said, "Brendan, that's major progress!"  He looked up at us with a huge smile on his face: he knew he'd just accomplished something worthwhile.  If only the same could be said for my morning . . . :-(

Bedtime Routine

Back at the beginning of September, Caedmon finally figured out how to climb out of his crib, and how to turn the light on in his room.  We tried very hard for several nights to get him to stay in bed on his own, but we eventually gave up.  Ever since, every night, our routine has been like this:

• Give him a bath and brush his teeth.
• Read three stories together.
• Give him half an hour to play quietly in his room.
• Tuck him in, pray together, and turn off the light.
• Wait outside the room (at most 30 seconds) before he's up and turning on the light or opening the door.
• Remove the lightbulb, and lock his door.
• Wait for the screaming to die down.

For the last week or so, we've been talking about trying something different – partly because we're tired of the fight, but also because we don't like having to resort to a physical restraint, like a lock.  We'd rather that he be able to control himself, well, by himself.

So this afternoon, we explained to Caedmon that we'd be doing something different tonight.  We had four new bedtime rules: (1) Stay in bed; (2) close your eyes; (3) stay quiet; (4) put your head on your pillow.  We gave him hand motions for each of the rules, and rehearsed them with him repeatedly throughout the afternoon and evening.  We also explained that if he got up out of bed, we would immediately put him back to bed, without looking at him, and without saying anything.

So Galena drew the short straw tonight.  Everything went well, up to the point where she turned off the light, left the room and closed the door. 

At about the half hour mark, I came up to see how things were going.  She was standing outside the door with her teeth clenched.  In-between missions, she said, “Thirty-three.”  The door opened again, she disappeared inside, then re-emerged.  “Thirty-four.” The door opened again, and she disappeared once more. “Thirty-five,” she said when she came out.

I came back about half an hour later.  She was still standing outside the door, teeth still clenched, but she had removed her sweater and her arms were bare.  “One hundred thirty.”  “One hundred thirty-one.”  “One hundred thirty-two.”

Somewhere around 150, I could hear Caedmon's giggles switch to crying.

I popped my head into the hallway a bit later.  “One hundred sixty-two,” she said, but there was triumph in her eyes.  From within his room, I could hear Caedmon screaming, “No, Mom! No! Go away! Daddy! Daaaaddy!"  The door opened again, and in she went.

Caedmon is now asleep.  Galena had to put him back in bed 169 times before he finally stayed.

It's my night tomorrow.  Pray for me.