A setting that can boost performance of any heavily network-dependent application

By default .NET allows only 2 connections to a given network address per AppDomain. In most cases this works fine but if your app makes a couple of dozens network calls a second then this value might be too small and it might actually cause a bottleneck that is very hard to diagnose. I decided to increase the value of this setting to the value that is recommend by Microsoft (number_of_cores x 12) and one of my services speeded up significantly. Having said that I have to stress that there is no guarantee this setting will work in your case. Remember, measure, measure and once again measure when you optimize.
And the setting is:

 <system.net>
     <connectionManagement>
         <add address="*" maxconnection="96"/>
     </connectionManagement>
 </system.net>

WCF service hangs - the pool of available sessions might have been exhausted

By default WCF accepts only 10 concurrent sessions which is not enough for most applications. If there are 20 clients then 10 of them will be blocked until their requests time out. You can always increase the number of sessions

but this is going to work as long as you control all the clients. If you don't then some of them might not properly close their sessions which next might lead to a resource leak on the service side. This is not an easy problem to solve unless you are ready and able to abandon sessions.
From my perspective it's much more important to know that a service is about to reach it's limit of sessions or it is hung because it already has reached it. Unfortunately as far as I know there is no out of the box way of monitoring the number of active sessions per WCF service using Performance Counters. This leaves us with only one option, namely we have to write a custom performance counter on our own. This can be done as a WCF extension that implements IChannelInitializer and  IInputSessionShutdown interfaces.
When a service seems to be frozen the story is not that simple as there might be dozens of reasons why it is in such a state. The only way I know to prove or disprove that the problem is related to the fact that there are no available sessions is to create a memory snapshot of the process where the service is hosted in and use Debugging Tools for Windows to check the state of all ServiceThrottle objects.
The following steps shows how to carry out such an investigation :) :
0. Install Debugging Tools for Windows and copy C:\Windows\Microsoft.NET\Framework\v2.0.50727\sos.dll (extension for .NET debugging) to the folder where Debugging Tools for Windows are installed.
1. Find the id of the process that is hosting the WCF service we are examining. In my case it is one of the IIS worker processes.

2. Create a memory snapshot using the adplus script which is part of Debugging Tools for Windows. By default adplus creates all snapshots under the folder where the Debugging Tools for Windows is installed.

3. Launch windbg.exe (the same location as adplus) and open the memory snapshot.

4. Type .load sos and press enter to load sos.dll into windbg.

5. Type !dumptheap -type ServiceThrottle -short in the command line to list all objects of type ServiceThrottle that exist on the managed heap. By the list of all objects I mean a list of their addresses in memory.

6. For each address on the output list carry out steps 7 - 8
7. Type !do address of the object to see what's inside of it.

8. The ServiceThrottle object has bunch of fields but only one of them which is called sessions is interesting from our perspective. Type !do address of the sessions field to see what's inside of it.

If you find a sessions field that has count and capacity fields set to the same value then you know that the pool of available sessions has been exhausted. If you can't find it then at least you know that there is something else wrong with your service.
Happy debugging :)

Windows XP 64 bit - a relatively unknown but great operating system

A few weeks ago I got a new beefy machine and the guy who was setting it up asked me if I wanted to try XP 64 bit. At the beginning I didn't see any compelling reason to move to a new platform but then I realized that XP 32 bit is not going to take full advantage of 4GB of RAM and I decided to give it a try. I'm glad that I've made that decision as it turned out that XP 64 bit is superior to XP 32 bit as a foundation for software development.

• XP 64 bit can leverage all 4GB of RAM which allows me to run 4/5 instances of VS.NET, SQL Server, Outlook, Firefox and a dozen of background applications without any noticeable performance degradation.
• XP 64 bit shares core components with Windows Server 2003 which would explain its great performance and reliability.
• XP 64 bit comes with IIS 6.0. I suppose I don't have to explain why this is big. If my point is not clear to you, just compare IIS 5.1 and IIS 6.0 and you will immediately understand what I'm talking about.

I asked quite a few software developers about XP 64 bit and none of them has ever considered installing it. It looks like the Marketing Department at Microsoft is very busy coming up with even longer names of Microsoft products :) and they have no time to market one of the best systems they've released.

WeakEvent - you wish it was here

At the first glance .NET events are an easy and harmless way to decouple components. The former statement is true but the latter is not. The reason is that whenever an instance of a class subscribes to an event published by another class a strong link between these two is established. By strong link I mean that the subscriber(listener) won't get garbage-collected as long as the publisher is alive. The only way to break that link is to unsubscribe from the event which might be easily omitted as the link is not explicit. Additionally there are cases when explicit(deterministic) cancellation of subscription is impossible. If additionally the publisher is a long living object than we might face a memory leak. In ideal world there would be a way of specifying that a subscription is weak which would mean that if the subscription is the only link to an object then the object can be garbage-collected and the subscription can be deleted. .NET does not provide that facility out of the box but fortunately it provides building blocks that in most cases let us build a good enough solution. The idea is to intercept calls that add and remove subscribers from/to an event and create a weak link between subscribers and publisher instead of the default, strong one.
When you define an event you don't have to write add/remove methods on your own because C# compiler generates them automatically. Basically the following code snippet:

public event EventHandler <EventArgs> MyEvent;

is just "syntactic sugar" that C# compiler transforms to much more verbose form. You can find detailed description of this process in CLR via C#  by Jeffrey Richter. From our perspective the most important thing is that we can overwrite the default behavior of the compiler and inject our own implementation of Add and Remove methods in a way that is completely transparent to subscribers. SomeClass and Subscriber classes show how it can be done. Don't worry about WeakEvent<T> class as it will be explained later.

public class SomeClass
{
    private WeakEvent <EventArgs> myWeakEvent;
    public event EventHandler <EventArgs> MyWeakEvent
    {
        add
        {
            myWeakEvent.Add(value);
        }

        remove
        {
            myWeakEvent.Remove(value);
        }
    }
    private void SomeMethodThatNeedsToRiseMyWeakEvent()
    {
        OnMyWeakEvent(new EventArgs());
    }

    protected void OnMyWeakEvent(EventArgs args)
    {
        myWeakEvent.Invoke(args);
    }
}
public class Subscriber
{
    private SomeClass someClass;
    public Subscriber()
    {
        someClass = new SomeClass();
        someClass.MyWeakEvent += Method;
    }

    private void Method(object sender, EventArgs e)
    {
    }
}

Add and Remove methods take a delegate as the input parameter. Every .NET delegate is an object with 2 properties. One of them is a reference to the target of the delegate(the object the delegate will be called on) and the second one is a description of the method which is provided as an instance of System.Reflection.MethodInfo class. Static delegates have the target property set to null. The target field is the root of all evil as it keeps the subscriber alive(it is a strong reference to the object the delegate will be called on). Fortunately .NET framework provides a class that can act as man in the middle between the method and its target which lets us break the direct link between them.
The class that makes it possible is called (no surprise) System.WeakReference. An instance of System.WeakReference class keeps a weak reference(instead of strong) to the object that is passed to its constructor. The weak reference can be transformed into the strong reference by accessing its Target property and storing its value in an ordinary variable. In this way we resurrect the object. If the object is already garbage-collected then the property returns null. All aforementioned functionality is encapsulated in a custom class that I called WeakDelegate.

internal class WeakDelegate
{
    private WeakReference target;
    private MethodInfo method;

    public object Target
    {
        get
        {
            return target.Target;
        }
        set
        {
            target = new WeakReference(value);
        }
    }

    public MethodInfo Method
    {
        get { return method; }
        set { method = value; }
    }
}

WeakEvent<T> is a class that takes advantage of WeakDelegate class to solve the problem outlined in the first paragraph. Its below implementation is rather straightforward but 2 pieces of code might need some explanation. The first one is inside Invoke method. Internally we store instances of WeakDelegate class which means that we can not invoke them directly and every time one of them needs to be executed we have to assemble an instance of  System.Delegate class. I don't know if the way the code creates delegates is the fastest one but I measured the execution time of that statement and the average time was 0.005384 ms per delegate which is fast enough for me. The second one is related to the fact that the locking is done in a way that prevents threads from waiting forever. If a thread can't enter the critical section within 15 seconds then it throws an exception. The rationale behind that approach is explained here.

public class WeakEvent <T> where T : EventArgs
{
    private readonly List <WeakDelegate> eventHandlers;
    private readonly object eventLock;

    public WeakEvent()
    {
        eventHandlers = new List <WeakDelegate>();
        eventLock = new object();
    }

    public void Invoke(T args)
    {
        ExecuteExclusively(delegate
        {
            for (int i = 0; i  < eventHandlers.Count; i++)
            {
                WeakDelegate weakDelegate = eventHandlers[i];
                // don't move this line to the ELSE block 
                //as the object needs to be resurrected
                Object target = weakDelegate.Target;

                if (IsWeakDelegateInvalid(target, weakDelegate.Method))
                {
                    eventHandlers.RemoveAt(i);
                    i--;
                }
                else
                {
                    Delegate realDelegate = Delegate.CreateDelegate(typeof(EventHandler <T>),
                    target, weakDelegate.Method);
                    EventHandler <T> eventHandler = (EventHandler <T>)realDelegate;
                    eventHandler(this, args);
                }
            }
        });
    }

    public void Remove(EventHandler <T> value)
    {
        ExecuteExclusively(delegate
        {
            for (int i = 0; i  < eventHandlers.Count; i++)
            {
                WeakDelegate weakDelegate = eventHandlers[i];
                Object target = weakDelegate.Target;

                if (IsWeakDelegateInvalid(target, weakDelegate.Method))
                {
                    eventHandlers.RemoveAt(i);
                    i--;
                }
                else
                {
                    if (value.Target == target && value.Method == weakDelegate.Method)
                    {
                        eventHandlers.RemoveAt(i);
                        i--;
                    }
                }
            }
        });
    }

    public void Add(EventHandler <T> value)
    {
        ExecuteExclusively(delegate
        {
            RemoveInvalidDelegates();

            WeakDelegate weakDelegate = new WeakDelegate();
            weakDelegate.Target = value.Target;
            weakDelegate.Method = value.Method;

            eventHandlers.Add(weakDelegate);
        });
    }

    private void RemoveInvalidDelegates()
    {
        for (int i = 0; i  < eventHandlers.Count; i++)
        {
            WeakDelegate weakDelegate = eventHandlers[i];

            if (IsWeakDelegateInvalid(weakDelegate))
            {
                eventHandlers.RemoveAt(i);
                i--;
            }
        }
    }

    private void ExecuteExclusively(Operation operation)
    {
        bool result = Monitor.TryEnter(eventLock, TimeSpan.FromSeconds(15));

        if (!result)
        {
            throw new TimeoutException("Couldn't acquire a lock");
        }

        try
        {
            operation();
        }
        finally
        {
            Monitor.Exit(eventLock);
        }
    }

    private bool IsWeakDelegateInvalid(WeakDelegate weakDelegate)
    {
        return IsWeakDelegateInvalid(weakDelegate.Target, weakDelegate.Method);
    }

    private bool IsWeakDelegateInvalid(object target, MethodInfo method)
    {
        return target == null && !method.IsStatic;
    }
}

You might have noticed that there is some housekeeping going on whenever one of Add, Remove or Invoke methods is called. The reason why we need to do this is that WeakEvent<T> keeps a collection of WeakDelegate objects that might contain methods bound to objects(targets) that have been garbage-collected. In other words we need to take care of getting rid of invalid delegates on our own. Solutions to this problem can vary from very simple to very sophisticated. The one that works in my case basically scans the collection of delegates and removes invalid ones every time a delegate is added, removed or the event is invoked. It might sound like overkill but it works fine for events that have around 1000-5000 subscribers and it's very simple. You might want to have a background thread that checks the collection every X seconds but then you need to figure out what is the value of X in your case. You can go even further and keep the value adaptive but then your solution gets even more complicated. In my case the simplest solutions works perfectly fine.
Hopefully this post will save someone an evening or two :).

Machines are predictable, people are not

 

I suppose we would all agree with that and that's why smart people try to develop processes to make us more predictable. On the other hand nobody likes being constrained by anything and especially a process. Some people call this kind of lack of structure freedom, some call it chaos :). From my experience a bit of process might actually help a lot whereas a complete lack of it leads sooner or later to a disaster. Scrum is one of the approaches that let people develop software in a predictable way and that's the topic of the next MTUG event (29th April) that I'm not going to miss. See you there.

Tags: MTUG, Dublin, Scrum

Never ever synchronize threads without specifying a timeout value

Whenever there is more then one thread and more then one shared resource there must be some synchronization in place to make sure that the overall state of the application is consistent. Synchronization is not easy as it very often involves locking which very easily might lead to all sorts of deadlocks and performance bottlenecks. One of the ways of keeping out of trouble is to follow a set of guidelines. I can list at least a few sources of information worth getting familiar with:

• CLR, Concurrent Programming : Joe Duffy - Microsoft
• CLR : CLR Via C# - Jeffrey Richter - Wintellect
• Debugging: Tess - Microsoft 

And of course :) my two cents or rather lessons I've learnt writing and/or debugging multithreaded code:

1. Minimize locking  -  Basically lock as little as possible and never execute code that is not related to a given shared resource in its critical section. The most problems I've seen were related to the fact that code in a critical section did more then it was absolutely needed.
2. Always use timeout - Surprisingly all synchronization primitives tend to encourage developers to use overloads that never time out. One of the drawbacks of this approach is the fact that if there is a problem with a piece of code then an application hangs and nobody has an idea why. The only way to figure that out is to create a dump of a process (if you are lucky enough and the process is still hanging around) and debug it using  Debugging Tools for Windows. I can tell you that this is not the best way of tackling production issues when every minute matters. But if you use only API that lets you specify a timeout then whenever a thread fails to acquire a critical section within a given period of time it can throw an exception and it's immediately obvious what went wrong.
   
   

   Default	Preferred
   Monitor.Enter(obj)	Monitor.TryEnter(obj, timeout)
   WaitHandle.WaitOne()	WaitHandle.WaitOne(timeout, context)

   
   The same logic applies to all classes that derive from WaitHandle: Semaphore, Mutex, AutoResetEvent, ManualResetEvent.
3. Never call external code when in a critical section - Calling a piece of code that was passed to a critical section handler from outside is a big risk because there is always a good chance that at the time the code was designed nobody even thought that it might be run in a critical section. Such code might try to execute a long running task or to acquire another critical section. If you do something like that you simply ask for trouble :)

I suppose it's easy to figure out which one has bitten me the most :).

MIX summary in Dublin

It looks like there will be a micro MIX like event in Dublin in May - http://visitmix.com/2008/worldwide/. It might be interesting.

Tags: Dublin, MIX