Monday 30 June 2008

WCF service hangs - the pool of available sessions might have been exhausted

By default WCF accepts only 10 concurrent sessions which is not enough for most applications. If there are 20 clients then 10 of them will be blocked until their requests time out. You can always increase the number of sessions

but this is going to work as long as you control all the clients. If you don't then some of them might not properly close their sessions which next might lead to a resource leak on the service side. This is not an easy problem to solve unless you are ready and able to abandon sessions.
From my perspective it's much more important to know that a service is about to reach it's limit of sessions or it is hung because it already has reached it. Unfortunately as far as I know there is no out of the box way of monitoring the number of active sessions per WCF service using Performance Counters. This leaves us with only one option, namely we have to write a custom performance counter on our own. This can be done as a WCF extension that implements IChannelInitializer and  IInputSessionShutdown interfaces.
When a service seems to be frozen the story is not that simple as there might be dozens of reasons why it is in such a state. The only way I know to prove or disprove that the problem is related to the fact that there are no available sessions is to create a memory snapshot of the process where the service is hosted in and use Debugging Tools for Windows to check the state of all ServiceThrottle objects.
The following steps shows how to carry out such an investigation :) :
0. Install Debugging Tools for Windows and copy C:\Windows\Microsoft.NET\Framework\v2.0.50727\sos.dll (extension for .NET debugging) to the folder where Debugging Tools for Windows are installed.
1. Find the id of the process that is hosting the WCF service we are examining. In my case it is one of the IIS worker processes.

2. Create a memory snapshot using the adplus script which is part of Debugging Tools for Windows. By default adplus creates all snapshots under the folder where the Debugging Tools for Windows is installed.

3. Launch windbg.exe (the same location as adplus) and open the memory snapshot.

4. Type .load sos and press enter to load sos.dll into windbg.

5. Type !dumptheap -type ServiceThrottle -short in the command line to list all objects of type ServiceThrottle that exist on the managed heap. By the list of all objects I mean a list of their addresses in memory.

6. For each address on the output list carry out steps 7 - 8
7. Type !do address of the object to see what's inside of it.

8. The ServiceThrottle object has bunch of fields but only one of them which is called sessions is interesting from our perspective. Type !do address of the sessions field to see what's inside of it.

If you find a sessions field that has count and capacity fields set to the same value then you know that the pool of available sessions has been exhausted. If you can't find it then at least you know that there is something else wrong with your service.
Happy debugging :)

No comments:

Post a Comment