Valid Reasons for Using Threads

We had an interesting debate this morning in our daily Alanta standup meeting.  We’re still in the process of implementing our proprietary media server (long story), and I’d noticed during a recent debugging session that with three clients connected, our media server had spawned something like 118 different threads.  That didn’t seem right to me, so we had a long discussion about whether we actually needed that many threads and whether it represented a significant problem for our architecture.  The answers that seemed obvious to me – no, we didn’t, and yes it was – weren’t as obvious to everyone else.  We were eventually able to come to a consensus about how to proceed, but I thought I’d jot down some thoughts about when it’s appropriate to spawn threads, when it’s not, and some best practices about how to achieve the right balance in your architecture.

Why you don’t want a lot of threads

First things first: you really do want to minimize the number of threads in your application. One well known reason for doing this, of course, is that there’s overhead every time the processor has to switch between threads. The processor has to save the state of the currently executing thread, then load the previously saved state of the next thread (and if the threads are in different processes, it needs to reload the virtual address translation tables as well).  This results in a practical limit of about 30 active threads per process per processor.  Any more than that, and you’ll be spending more time dealing with thread overhead than with getting anything done.

However, the argument advanced this morning was that if the threads weren’t active, i.e., were just waiting to be signaled, then they weren’t chewing up CPU cycles with context switches, and hence even very large numbers of them were harmless.  The problem with this argument is that threads have a significant virtual memory overhead even when they’re inactive.  As Raymond Chan has pointed out, each thread you spawn needs its own stack, and by default, the stack size on a Windows box is 1 MB.  This means that every time you spin up another thread, you’re adding another another 1MB to the virtual memory your process is consuming.  So on a 32-bit machine, since 2GB is generally reserved for the kernel, the maximum number of threads that any one process could reasonably hope to create is ~2000.  (On Linux, the default stack size is 2 MB, which introduces a practical limit of ~1000 threads per process.)  Of course, this is only on 32-bit machines, and you can get away with quite a bit more on 64-bit machines, but the point remains: every thread chews up allocatable memory, and quite a lot of it.  One caveat: each thread chews up 1 MB of virtual memory, not physical memory (at least, not right away).  In other words, if your process creates 500 threads, you won’t necessarily see your process chewing up 500 MB of physical memory in task manager.  That’s because the pages in physical memory that back your stack aren’t allocated until they’re referenced (as described here).  So you can’t tell you’re running low on virtual memory until you’ve actually run out of address space, and the first indication you’ll have that you’re about to die is a strange and otherwise inexplicable Out of Memory error.

I should note that it’s theoretically possible to create a threading model that doesn’t suffer quite so badly from the stack problem noted above.  Rob Behren and some compatriots wrote a great paper on the topic, in which they demonstrated a system running with over 100,000 active threads.  Unfortunately, their recommendations haven’t been widely adopted by tool and framework vendors.  For instance, they point out that a compiler/linker which dynamically adjusted stack sizes would lower the memory overhead for each thread pretty substantially.  This is true, but I’m not aware of any mainstream compilers that actually do this.

So in brief, if you’re writing papers for a conference, create as many threads as you want. If you’re writing software with mainstream compilers, frameworks and operating systems, limit the number of threads you create.

Three reasons for creating threads

So it’s pretty clear why we don’t want a whole bunch of threads.  Similarly, there are a pretty limited number of reasons for ever creating threads in the first place:

1. To take advantage of multiple processors.  This is the classic reason.  If your software will typically be running on a multiprocessor or multicore machine, and you really do have more work than one processor can handle in a timely fashion, it makes sense to split that processing up into multiple independent threads of execution.
   
2. To move specific processing off of the UI thread.  This is a much more pragmatic reason.  In the normal Windows world, you typically create threads so that a given background process doesn’t choke your UI. In the Silverlight world where I’ve been living lately, this is less of a problem: Silverlight forces every sort of IO into an asynchronous pattern, and that tends to keep your UI thread from blocking.  But the opposite can sometimes be an issue: the UI thread can get so busy that you need to spin up background threads so important tasks (like audio encoding or decoding) can get dispatched quickly enough.  But the idea is the same: you want to keep your users from getting grumpy because important parts of their application appear to have ground to a halt.
   
3. To simplify complicated asynchronous calling patterns.  Unless you actually need to take advantage of multiple processors, it’s almost always possible to get the effect of threads through a completely different mechanism, namely, using events to pass control from one part of the program to another.  (This is called the Lauer/Needham Duality.  In a famous 1979 paper, Lauer and Needham showed that “message-oriented” and “procedure-oriented” systems – read event-driven vs. multithreaded – were duals of each other and hence were logically equivalent architectures.)  But although thread synchronization is difficult to get right, it can be even more painful to write a program using entirely asynchronous calls.  The APIs for doing so are often complicated and obscure, and they require that you split your program’s logic across various artificial boundaries.  Depending on the complexity of your application, they may also require a that you implement a cooperative multitasking model, where any given function can be requested to yield to other functions, and only later pick up where it left off.  These sorts of issues can make debugging and maintenance quite difficult.  Threads are plenty complicated, but apart from the places where you start a thread, wait for a thread to finish, or lock some resource, multithreaded code looks reasonably similar to synchronous single-threaded code.  And that’s almost always a good thing.

If none of these three reasons apply, you don’t need to create threads.  It’s as simple as that.

Maintaining the right balance

In the real world, you’re typically going to need to balance legitimate reasons for threads against the virtues and vices of asynchronous programming.  But the basic guidelines for scalable, high-performance systems are reasonably clear:

1. Use asynchronous IO with callbacks or events when you need to interface with the outside world.  If a client connects to your server, you may be tempted to spin up a thread dedicated that client, and then let everything within that thread happen synchronously.  In other words, you create a thread for each client that connects, and then in that thread, you send some data to the client, block until the client sends some back, and so forth, until finally the client disconnects, at which point you terminate the thread.  Certainly that’s the simplest model, and if you’re confident that you’ll never have more than a couple dozen clients connected simultaneously, it may be a reasonable approach.  But as I discussed above, it’s surprisingly memory intensive, and a load as small as a few hundred clients could bring your server to its knees (even if the server’s CPU was completely idle).  A better approach is to use the asynchronous version of whatever network API you’re using, and simply handle whatever processing needs to be done in the callback.  On the native Windows API, you do this using I/O completion ports; on Linux, you can use select() or poll() or (better) aio_read() and its compatriots.  If you’re using C++, the best way to do it is probably to use the boost::asio namespace; and if you’re using .NET, you should use the asynchronous versions of whatever piece of the framework is applicable. (There’s a reasonable, if incomplete, example of how to use Socket.BeginAccept/Socket.EndAccept here on the MSDN site, for example.)  Depending on how your framework is implemented, this may also be a good place to use a threadpool: instead of handling the event on whatever thread it comes in on, hand it off to your threadpool, and go back to waiting for the next signal/callback/event.
   
2. Don’t try to do everything via asynchronous message-passing. Threads are really a pretty helpful abstraction.  I’ve seen high-performance servers implemented entirely using asynchronous, event-driven code, but they’re a pain to troubleshoot and debug.  The vnc-reflector project on SourceForge is a good example of this.  It’s very performant, even though it runs on a single thread.  But it’s a pain to debug or modify, because so much of the logic consists of passing static function pointers around in a very complicated main() loop.  Asynchronous is good, but don’t bother trying to make everything asynchronous.  If you can simplify your code by kicking off a preemptively multitasked thread, rather than implementing a whole bunch of your own mechanisms for cooperative multitasking, by all means, kick off the thread.
   
3. Use a threadpool whenever you don’t know precisely how many threads you’ll be creating.  If you want to dedicate a thread to some specific task that can run in the background, there’s not much point to using a threadpool.  But if you need to handle an unknown n number of clients, don’t try to spawn a dedicated thread per client.  Use a threadpool instead, and service the requests out of the pool.  On a side note, I’ve known folks who were tempted to write their own threadpools.  Granted that in their simplest versions they’re not all that complicated, but still: unless you have a damned good reason to do so, don’t.  Recent versions of Windows have their own built-in threadpool APIs; .NET has a reasonable ThreadPool in the System.Threading namespace, or you can use Ami Bar’s SmartThreadPool; and although Boost strangely doesn’t have one, Philipp Henkel has made a boost::threadpool available that works cleanly with Boost threads.  And there are tons of other examples.  It’s just silly to try to reinvent the wheel when there are so many thorough implementations available for basically the cost of browsing to a website and looking at some sample code.
   
4. A good compromise may be to use a small number of queues to move data through your system.  Queues are a classic asynchronous mechanism, but they can also be used effectively with threads and threadpools.  For instance, one of the functions of a media server might be to mix audio coming in from clients in a conference call, and then send the result back to each.  One of many possible architectures would be to implement this as a series of queues with a threadpool servicing each of the queues.  One threadpool would parse the incoming data and then place the compressed audio data in a queue to be decompressed; a second set of threads would decompress the audio and then place it in a the next queue to be mixed; a third set of threads would mix the decompressed audio and then place the results in a final queue; a fourth threadpool would then construct the mixed audio data into packets and deliver them back to the clients.  You could also implement this same basic architecture with more or less queues, depending on other constraints and design goals.  And I should also note that the only reason for using threadpools rather than individual threads would be to take advantage of multiple processors.  If you only had four threads servicing the three main queues, but were running this on an eight-processor box, you’d effectively be leaving four of the processors unused.

If you take this last approach, however, just make sure that your queues and especially your threadpools are limited in number.  The problem that kicked off this whole blog posting is that our media server currently creates a separate set of queues for each connected client, and then creates a separate threadpool for each queue.  The result is that each connection spawns 37 new threads.  I don’t think it’ll be too painful to fix this: but it does need to be fixed.

Rained out on Labor Day (again)

We tried to go camping for Labor Day (again), but like last Labor Day, we got rained out, gave up and came back home early.  Still, we had a reasonably good time while we were there.  

http://picasaweb.google.com/smithkl42/20100904LaborDayCamping#

Our weekend started on Friday afternoon, when Caedmon and I drove to the Middle Fork Campground, ahead of everyone else, so that we could be sure to get our spot.  

The next morning, Saturday, Caedmon and I tried our hand fishing, but didn't have any luck.  Probably would have done better with flies, but it's difficult enough to get a three-year old to handle a spinning rod without tangles and snags everywhere.  I'll leave the fly-fishing lessons for another day.

After fishing, but before lunch, Caedmon and I went on a hike that ended up being longer than we expected.  We got about a mile and a half out before finally turning around.  Caedmon did great, but was a little disappointed that we turned back before the end of the trail.  I explained to him that we needed to go back to camp so we could have some food, before we got all tired and grumpy.  He wanted to know why we might get tired and grumpy, so I said, "Well, it's our brain's way of telling us that we need to get more food."  A while later, when we were almost back to our campsite, I was carrying him on my shoulders, and I heard him whisper to himself, "It's all right, brain. You'll get some food in a couple minutes."

Later Saturday afternoon,  Galena, Brendan, Calista and Kirstin McGhee got there.  Kirstin's been helping Galena with the kids most of the summer, so she knows them very well, and when her boyfriend, Tony, decided to spend Labor Day weekend in Las Vegas, she asked if she could hang out with us.  That gave us the excuse we needed to try camping with three such small kids.  And she was very handy to have around -- you don't want young children outnumbering adults on a camping trip.

Brendan loved the ice-cold, freezing water.  Must help to have a layer of blubber for insulation.

This was Calista's first camping trip, and she seemed a little stunned by it all.

That evening, after dinner, we made s'mores, which were a substantial hit, not least because they gave Caedmon an excuse to wave a flaming torch around.

It started sprinkling just after we put the kids to bed, and by the time we were in bed ourselves, it was coming down pretty hard.  It rained all night, and it turns out that 3-season tents are really preferable to 2-season tents in situations like that.  I had Caedmon and Calista in my tent, and we were all dry, but Galena, Kirstin and Brendan were all pretty wet by the time morning rolled around.  So we made a quick breakfast, cleaned up as best we could in the rain, and headed home.  I like to think of it less as "giving up" than "staying married".

Rainier Summit via Kautz Ice Chute

Late last December, my pastor, Charlie Swartz, pulled me aside and asked if I’d be interested in climbing Rainier with him.  I’d tried twice before, succeeding once, but I didn’t feel comfortable enough with my skills to lead a team myself.  Consequently, I asked my cousin Brian (who has 10 Rainier summits and an Everest summit on his resume) if he’d be interested in leading the team.  He was, and we started planning.

The only route that I’d done before was the Schurman/Emmonds approach on the other side of the mountain.  This time, however, my cousin had talked us into trying something a little more challenging: the Kautz Ice Chute, a couple miles climber’s left of the standard Disappointment Cleaver route.  Gauthier’s book describes it as a Grade II/III, basically because of the difficulty of the ice chute, which has a one 50 degree pitch, and a second 60 degree pitch.  That seemed challenging for a bunch of newbies, but folks liked the idea, so we went with it.

Over the next six months, probably two dozen people “joined” our team and then backed out, but when we finally left the Paradise parking lot on Wednesday morning, we had 11 folks hoofing it up towards Pan Point.

Unfortunately, the two most interesting members of our party had to back out first.  Lhakpa Sherpa has five Everest summits under his belt, but his wife Maya wasn’t feeling good at elevation, so they peeled off after crossing the Nisqually Glacier, and headed back to Seattle, which left nine of us heading up out of the Nisqually and up onto Wapowety Cleaver.

Once we were on Wapowety Cleaver, we crossed over the Wilson Glacier then began working our way towards the base of the Turtle.

What with one delay or another, along with the logistics of keeping a large group moving, we ended up getting to camp later than we had hoped, about 7:00 pm.  We setup camp at about 9500 feet, right around the base of the Turtle, ate dinner, watched the sunset, and then headed to bed.

Our original plan had been to take Thursday as a rest day, and then summit and descend on Friday.  However, most folks were feeling fairly good, so we decided to try for a 3:00 am start on Thursday morning.  The result was that I got four hours sleep, and pretty much everyone else got less – several folks didn’t get any, and two members of our team were feeling too exhausted to try for the summit.

The seven remaining members of our team made it to the top of the Turtle just as the sun was coming up.  We roped up, adjusted our crampons, and headed onto the Kautz ice chute.

Luckily for us, the route was in excellent shape.  Usually, by mid-July, the Kautz is more ice than snow, but we still had a great snowpack, so even the two steep pitches didn’t give us much trouble going up.  We eventually made it up off the Kautz, and at about 13,000 feet crossed over to the Nisqually.

Unfortunately, this is about when one of our party began to suffer some of the symptoms of AMS.  Stephanie Spence was an experienced mountaineer, and had climbed at the 14,000 foot level in the past, but this time she began throwing up at around 13,000 feet.  She insisted on continuing, but by 13,500 she was done.  Graciously, Eric Dalzell, another Everest veteran, volunteered to give up his summit and descend with her. 

The last 1000 feet were brutal.  It was a step, and then two breaths, and then another step, and another two breaths, the entire way.  Anne Timblin declared that this was significantly harder than the half-ironman she’d done the year, and that she should have turned back.  But she kept going, and so did the rest of us.

Finally, a little after noon, we crossed over the summit rocks and stood on the summit crater. 

The weather was still perfectly clear, though the winds were beginning to pick up.  Several of us hunkered down in the rocks near the steam caves and took long naps.

After an hour and a half or so on the summit, we began descending, which was a long, slow process.

The trickiest part came while descending the top pitch of the Kautz Ice Chute.  The snow was pretty soft, which made for good down climbing, except for a 10 yard stretch of ice.  My cousin Brian had four of us on belay as we were descending the chute, so we were fairly well protected, but we were still trying to be careful.  I got to the icy stretch, and between general fatigue, nervousness, and inexperience, I couldn’t keep my crampons in the ice.  Unfortunately, although I had slowed down, the climber above me hadn’t, so there was a significant amount of slack in the rope when my crampons finally popped out and I went for a tumble down the slope.  I yelled “Falling!!”, and luckily the folks above me had enough time to go into self arrest.  I ended up sliding some 20-30 feet, and gave the rope a good yank when I finally hit the end, but they held on, which was good, as I didn’t have any desire to test the strength of the pickets up at our belay station.

After we’d reached another belay station towards the bottom of the pitch, we unroped, and then I belayed my cousin down.  The belay was mostly just for appearances, as we hadn’t placed any protection on the way down (Eric was carrying our extra pickets when he left to accompany Stephanie down the mountain): if Brian had fallen, he would have swept past us and then to the end of the rope before we could have stopped him.  Luckily, he was able to bypass the icy patch, and made it safely down to our belay station. 

The only other interesting part of our descent came a couple hundred feet further down the chute, towards the bottom of the lower pitch.  I was kickstepping my way down the chute, facing into the slope, when I felt the snow give way under me.  I went into self-arrest, but not before I found my legs and chest dangling inside a large crevasse.  I elbowed my way out – style points didn’t seem important at that point – and then our team carefully navigated around the crevasse, and we continued our descent to camp without any further incidents (other than a nice glissade down the Turtle).

The next day, we made ourselves breakfast, then headed out. 

The snow was soft enough that we were able to glissade maybe 75% of our way down to the Nisqually.

And damn, it felt good to get back to the parking lot.

I’ve posted the full assortment of the pictures I took here.  Anne’s pictures are on Facebook here.

Some lessons I learned:

1. If we ever do a newbie trip again, we’ll plan for four full days on the mountain.  If we’d had a rest day between our ascent to high camp and our summit day, probably all nine remaining members of our party would have been able to summit, instead of just five of us.
   
2. Folks without ice climbing experience have no business being out on the Kautz Ice Chute.  We got lucky in that the ratio of snow to ice on the Kautz was roughly 20:1, but I get the impression that’s not at all typical for this time of year.  If it had been a normal year, we would have struggled a great deal more.  Before I do that route again, I need to get some real experience on hard ice.
   
3. Training with a heavy pack at altitude is the best conditioning, and I hadn’t done enough of it.  I’ve been training regularly and hard for six months, and I’ve made trips up to both Muir and Schurman this season, but I could easily have used another two 10,000+ climbs before trying for the summit.
   
4. Large groups travel slowly.  A team of 12 people is just too large, and moves, well, glacially.  Next time, I won’t go with a group larger than six people.
   
5. I need a smaller camera than my DSLR to take on these trips.  Not only was my Pentax K10D heavy (I didn’t need the extra two pounds), but it’s awkward, with the result that I had to leave it in my pack most of the time.  It doesn’t matter that an SLR takes better photographs if I can’t get the picture because I can’t get at the camera.
   
6. Once again, I was carrying too much: my pack weighed nearly 80 pounds out of the parking lot.  Options for dropping weight include switching over to a jet boil stove, taking less food (I had ~10 pounds left over), and carrying less water (see #7).
   
7. I need to figure out how to drink less water.  During our two climbing days, I drank probably twice as much water as anyone else – nearly six liters on each of our climbing days.  I sweat a lot, but still, this seems excessive.  My suspicion is that I was probably suffering from hyponatremia, and that some Gatorade mix, or even just some Nuun tablets, would have made a significant difference in the amount of water I felt I needed.
   
8. The Asolo AFS Evoluzione climbing boots that I rented from REI were worthless.  They have no flex in them whatsoever, and while they might be good for technical ice climbing, they’re vastly inappropriate for general-purpose mountaineering.  Specifically, they’re nowhere near as good as the Koflachs that REI used to rent.  It was like climbing in plastic ski boots: I ended the climb with large painful blisters and bruises on the front of both shins, and numerous places elsewhere on my feet.  If the Asolo’s are still what REI has available the next time I head up a peak, I’ll either have to bite the bullet and buy my own boots, or rent elsewhere.

Major Problem and Major Progress

The Major Problem

Brendan has been very excited about pictures lately, especially when they're hanging on the wall.  Last week, when Galena’s parents, John and Sue, were up visiting, John built an oversized picture frame for us that we could place at toddler eye-level. 

This morning, during a temporary lull in the insanity, Galena and I decided to mount that picture frame.  It was big enough that I figured we should attach it to the studs in the wall, so I used my handy-dandy stud finder to locate the first stud, and with the help of four toddler hands, managed to pound in the first nail.  While I was looking for the next stud, Galena asked, "What's that noise?"  With three kids under three, there's a lot of strange noises in our house, and I was preoccupied with keeping Brendan's hands off the stud-finder, so I didn't pay much attention to her question.  But she repeated it several times, and then suddenly yelled, "Ken, we've got a major problem.  There's water coming out of the wall!"

And there was.  From right behind the frame, about where I'd put the first nail, there was a small but steady stream of water dripping down onto the floor.  I realized almost immediately what had happened: I must have pounded the nail through a pipe in the wall.  Not knowing what else to do, and wanting to get a better look at the source of the leak, I pulled the frame away from the wall, pulling the nail out along with it.  That was when the real fun started, as a spray of water emerged from the wall powerful enough to drench the other side of the kitchen.

I don't remember everything that got said in the excitement that followed, but suffice to say that as I was running around trying to find the water main, the stream turned too hot for Galena to block with her bare hands.  She temporarily abandoned her attempt to contain the spray, and scooped Brendan up to get him out of the way of the increasingly hot water.  (Brendan exhibited extreme consternation at being manhandled in this way: he'd been enjoying himself tremendously.)  Caedmon, meanwhile, had run into the family room and was crouched out of the way of the spray, yelling excitedly, "Major problem! Major problem! MAJOR PROBLEM!"  Galena grabbed my Goretex jacket from the back of a chair, and with that in front of her, waded back into the scalding stream of water.  As this was happening, I was busy running in and out of the house in my bare feet, looking for pliers, and checking on results, as I first managed to turn off the gas to the entire house, and then the gas to the hot water heater, neither of which was as helpful in this situation as you might think.  Finally I found the right valve, and I heard the gushing from inside the house subside.

The plumber will supposedly be here sometime this afternoon.  I'm guessing if we're lucky, this will only cost us $1000.  In the meantime, the floor is covered with towels soaking up the water, and Caedmon has asked for the "major problem" to be re-explained to him at least 50 times.  "Why major problem, Mommy?"  It's a question I'd like to know the answer to myself.

The Major Progress

This is almost anticlimactic, but Brendan took his first consecutive steps today.  After we'd managed to get all the water soaked up, we fed the kids lunch.  Since the water is still turned off, I wiped Brendan down afterwards with a washcloth, and set him down on the floor.  Without even thinking, he took five steps in a row over to a chair and grabbed onto it.  Galena said, "Brendan, that's major progress!"  He looked up at us with a huge smile on his face: he knew he'd just accomplished something worthwhile.  If only the same could be said for my morning . . . :-(

Bedtime Routine

Back at the beginning of September, Caedmon finally figured out how to climb out of his crib, and how to turn the light on in his room.  We tried very hard for several nights to get him to stay in bed on his own, but we eventually gave up.  Ever since, every night, our routine has been like this:

• Give him a bath and brush his teeth.
• Read three stories together.
• Give him half an hour to play quietly in his room.
• Tuck him in, pray together, and turn off the light.
• Wait outside the room (at most 30 seconds) before he's up and turning on the light or opening the door.
• Remove the lightbulb, and lock his door.
• Wait for the screaming to die down.

For the last week or so, we've been talking about trying something different – partly because we're tired of the fight, but also because we don't like having to resort to a physical restraint, like a lock.  We'd rather that he be able to control himself, well, by himself.

So this afternoon, we explained to Caedmon that we'd be doing something different tonight.  We had four new bedtime rules: (1) Stay in bed; (2) close your eyes; (3) stay quiet; (4) put your head on your pillow.  We gave him hand motions for each of the rules, and rehearsed them with him repeatedly throughout the afternoon and evening.  We also explained that if he got up out of bed, we would immediately put him back to bed, without looking at him, and without saying anything.

So Galena drew the short straw tonight.  Everything went well, up to the point where she turned off the light, left the room and closed the door. 

At about the half hour mark, I came up to see how things were going.  She was standing outside the door with her teeth clenched.  In-between missions, she said, “Thirty-three.”  The door opened again, she disappeared inside, then re-emerged.  “Thirty-four.” The door opened again, and she disappeared once more. “Thirty-five,” she said when she came out.

I came back about half an hour later.  She was still standing outside the door, teeth still clenched, but she had removed her sweater and her arms were bare.  “One hundred thirty.”  “One hundred thirty-one.”  “One hundred thirty-two.”

Somewhere around 150, I could hear Caedmon's giggles switch to crying.

I popped my head into the hallway a bit later.  “One hundred sixty-two,” she said, but there was triumph in her eyes.  From within his room, I could hear Caedmon screaming, “No, Mom! No! Go away! Daddy! Daaaaddy!"  The door opened again, and in she went.

Caedmon is now asleep.  Galena had to put him back in bed 169 times before he finally stayed.

It's my night tomorrow.  Pray for me.

Really Missing Serialization Callbacks

I just ran into another feature whose absence from Silverlight is sorely missed.

I’m using the WCF generated proxy classes as the basis for binding to some UI objects.  If you’re using a full MVVM pattern, the way that you’d normally do this is wrap the proxy-generated classes with a ViewModel, so that instead of binding to, say, the User class generated by “Add Service Reference”, you’d bind to a UserViewModel class that acts as a facade for the User class.  So far, I haven’t been willing to go that route.  The only point to having a ViewModel is if you’re making extensive use of databinding, and databinding happens to be my least favorite Silverlight technology, for reasons that I’ve explained elsewhere.  And if I’m only doing databinding occasionally, it seems like a lot of more-or-less pointless work to recreate a facade for my complete object model on the client, and then keep it synchronized with the classes that the Entity Framework has already helpfully generated for me.  So I’ve been making do with partial classes to add any additional properties or methods that seem appropriate.

But as I said, I ran into a problem today.  One of my classes, SharedFile, has a bindable property called StatusText whose value depends on a complicated graph of other object properties; and to get them all working, I’ve had to string together an unpleasant chain of INotifyPropertyChanged notifications, sorta like so (this is just one of numerous chained handlers):

   1: void uploadCommand_PropertyChanged(object sender, System.ComponentModel.PropertyChangedEventArgs e)

   2: {

   3:     if (e.PropertyName == "State")

   4:     {

   5:         UpdateStatusText();

   6:     }

   7: }

OK, so far so good – but I have to wire up these event handlers somewhere.  Since the definition of SharedFile in reference.cs didn’t define a default constructor, I thought it would be simple to create one in my partial class, and that would be the end of it:

   1: public SharedFile()

   2: {

   3:     this.PropertyChanged += SharedFile_PropertyChanged;

   4: }

But although the objects get created, that constructor never gets called.  WTF?  Well, it turns out that when Silverlight deserializes the XML from my WCF service, it uses FormatterServices.GetUninitializedObject to create the object, which skips calling the constructor.  I guess that makes a certain sort of sense – the WCF service is handing you an object that, in theory, is already constructed, so you shouldn’t need to call the constructor again.  I get that.  But then where do I put this code?

OK, I know, I can put that code in a method that I tag with the [OnDeserialized] attribute.  That’s the normal way of doing it, right?  Oh – except Silverlight doesn’t support serialization callbacks.

Huh?

I get that some features need to be left out of Silverlight.  But this seems like a really odd one.  Serializing and deserializing objects is what you do in Silverlight.  You can write real-world WPF or WinForm or ASP.NET applications all day long and never once have to deal with object serialization and deserialization.  But you can’t use Silverlight for five minutes without needing to touch an object that’s just been deserialized from some web service.  So why choose that particular set of features to cut?  It sure seems like a bizarre design choice.

At any rate, my choices were either to implement a full-blown ViewModel layer, which I really don’t want to do, or write a hack of some sort to initialize the event handlers manually.  Uggh.

What I’ve done for now is to throw an Initialize() method on the containing object (User), which in turn initializes any object that gets added to its SharedFiles ObservableCollection:

   1: // This is necessary because Silverlight doesn't call a constructor when deserializing classes,

   2: // and also doesn't support on the [OnDeserialized] attribute.  Damn annoying.

   3: public void Initialize()

   4: {

   5:     InitializeSharedFileList(this.SharedFiles);

   6:     SharedFiles.CollectionChanged += SharedFiles_CollectionChanged;

   7: }

   8:  

   9: private void SharedFiles_CollectionChanged(object sender, System.Collections.Specialized.NotifyCollectionChangedEventArgs e)

  10: {

  11:     if (e.Action == NotifyCollectionChangedAction.Add)

  12:     {

  13:         InitializeSharedFileList(e.NewItems);

  14:     }

  15: }

  16:  

  17: private void InitializeSharedFileList(IList sharedFileList)

  18: {

  19:     foreach (SharedFile sharedFile in sharedFileList)

  20:     {

  21:         sharedFile.Initialize();

  22:     }

  23: }

And then I call User.Initialize() when I first retrieve it from the web service:

   1: private User user;

   2: public User User

   3: {

   4:     get

   5:     {

   6:         return user;

   7:     }

   8:     set

   9:     {

  10:         if (!object.ReferenceEquals(user, value))

  11:         {

  12:             user = value;

  13:             if (user != null)

  14:             {

  15:                 user.Initialize();

  16:                 UserId = user.UserId;

  17:             }

  18:             else

  19:             {

  20:                 UserId = string.Empty;

  21:             }

  22:         }

  23:     }

  24: }

Like I said, uggh.  But it works.  I just wish that MS had thought through their deserialization scenarios a little better.  I don’t like being forced into creating another abstraction layer if I don’t have to.

Silverlight duplex client limitation

I ran into an interesting (and largely irrelevant) limitation today on Silverlight's implementation of duplex web services.

I'd been using Jeff Wilcox's handy Silverlight Unit Test Framework to test the data access piece of my current Silverlight project. However, I was running into a nasty problem that was driving me nuts. Part way through every test run, my unit tests would start failing. I could usually get through something like 10 or so tests before every new WCF call would return "Not Found" (which is not really the most helpful error message Microsoft ever came up with). It didn't have anything to do with the individual tests themselves, because the error would show up after 10 tests, no matter which 10 tests they were.  I'd been working through this error for some time before I realized that the "10" number was undoubtedly significant, since that seems to be the default number of connections that WCF allows, unless you go in and bump it higher. (Now of course, IMO, that's a pretty dumb default: the obvious purpose for leaving it that low is to prevent DOS attacks -- but the net result is that instead of needing some 10,000 connections to DOS your server, you only need 10. Sigh.)

This made me think that I might be leaking a connection somewhere. In theory, I was closing all my connections in one test before moving on to another, but we all know how well that works :-).  Since I was opening and closing all my connections through the same static class, I wrote up some quick instrumentation, and saw that yeah, one connection was staying open after every test.  Some additional poking around, and I found a method that was opening a new connection and failing to close it.  Easily fixed.  Instead of this final line in my method:

EnqueueTestComplete();

I just made it do this:

EnqueueCallback(() => DataConnectionManager.TryClientClose(client,
            error => EnqueueTestComplete()));

And TryClientClose() looks something like this:

        public static void TryClientClose(RoomServiceClient client, OperationCallback callback)
        {
            if (client != null && client.State == CommunicationState.Opened)
            {
                client.CloseCompleted += (sender, e) =>
                {
                    ClientsClosed++;
                    ClientsOpen--;
                    if (e.Error != null)
                    {
                        client.Abort();
                    }
                    if (callback != null)
                    {
                        callback(e.Error);
                    }
                };
                client.CloseAsync();
            }
            else
            {
                if (callback != null)
                {
                    callback(null);
                }
            }
        }

Close enough. But why was I running into this error in the first place?  Truth be told, I don't completely know.  But my best guess is that the Silverlight client (or maybe the browser that's hosting it) has a limitation on how many duplex callback sessions it can support.  And so far as I'm aware, there's no way to increase this number.  At least, I've poked around in all the relevant blogs, and looked through the appropriate docs, and can't find anything obvious.  But the net result is that you don't want to have more than 10 duplex clients open at the same time on any given Silverlight application.