Mutex-free C# application deadlock

We’ve all heard about & encountered deadlocks. Nasty buggers they are, won’t you agree? After several long-night fights against them you should be well equipped to deal with them. You remember not to call external (delegate) code in synchronized context, you carefully add various logs to monitor behavior of mutexes, but to your surprise there is still an observable deadlock!

First, let’s make sure that we deal with a deadlock, not some funky system behavior, external library bug or endless loop. To test threads behavior you can basically do two things: change their number or add Thread.Yield(). We will focus on the first option.

In order to change the number of actively running threads we use the ThreadPool method. ThreadPool is the default way of using threads so I can safely assume that you will directly or indirectly (Tasks) use them.

We changed the number of threads, and we can observe that indeed the deadlock appears so we eagerly start debugging our poor parallel program, but the problem is not there.

What you should always check when using thread pools is whether the synchronization logic has taken all available threads. Take a look at the following example (assume the ThreadPool can run 10 threads concurrently).

The problem should be visible now.

  1. #10 waits for empty slot.
  2. Old threads won’t finish as they are long-running.
  3. Short-lived thread #9 (or a chain of inter-dependent threads) is waiting on #10
  4. #10 starves to death locking whole application.

You might not encounter this problem every day, but be careful when:

  • you frequently use BeginInvoke() in your code,
  • you port your application to older system on which the default number of threads is lower than you assumed.

While thread pools provide us with efficient creation of threads, they bring their own issues. Remember to enable sufficiently high number of max threads in pool, or reduce the number of threads that have to be run concurrently, in order to avoid synchronization starvation.

Advertisements

Tags: , , , ,

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s


%d bloggers like this: