Understanding System.Diagnostics DiagnosticSource and DiagnosticListener (Part 1)

Throughout its history, .NET has evolved various mechanisms to “log” diagnostic information inside applications and libraries, including TraceSource, EventSource, ILogger, and DiagnosticSource, the subject of this post. TraceSource is a legacy option and is rarely used in new code. ILogger is a simple structured logging abstraction that is well suited to many applications, although it does require an extra library dependency in some cases, such as when developing class library projects.

The distinction between EventSource and DiagnosticSource is quite subtle. There is little clear guidance on when to choose between these, although this is a good discussion on the topic, even if it’s not entirely conclusive. In the interest of not bloating this post further, I won’t go deep into the differences in this post.

I’ve recently been working with DiagnosticSource and wanted to dive deeper into the implementation to understand how it functions and identify some of its subtleties. I plan to share what I learned in this new series since the existing documentation could be made easier to follow. Stick with me to learn more about the design of the DiagnosticSource and DiagnosticListener in .NET.

Note: The code I am basing this post on is the .NET 8 branch of the runtime repository. As this is a stable, complete API, there is unlikely to be any significant difference in future releases.

In this first post in the series, we’ll focus on some of the core details of the DiagnosticSource and DiagnosticListener types, touching on the IObservable<T> interface along the way. We won’t yet venture into the recommended use of DiagnosticListener, which will follow in a future post.

Even so, I warn you that this is a relatively long blog post that goes beyond focusing purely on the public API. I believe that discussing the implementation details is helpful in fully appreciating what happens when we call methods on that public API and how things function.

If you’d like to be notified of future posts, you can register for email notifications on the right-hand side of this page.

DiagnosticSource

We’ll begin by studying the code for the DiagnosticSource class.

public abstract partial class DiagnosticSource
{
    internal const string WriteRequiresUnreferencedCode = "The type of object being written " +
        "to DiagnosticSource cannot be discovered statically.";
    internal const string WriteOfTRequiresUnreferencedCode = "Only the properties of the T " +
        "type will be preserved. Properties of referenced types and properties of derived types may be trimmed.";

    [RequiresUnreferencedCode(WriteRequiresUnreferencedCode)]
    public abstract void Write(string name, object? value);

    [RequiresUnreferencedCode(WriteOfTRequiresUnreferencedCode)]
    public void Write<[DynamicallyAccessedMembers(DynamicallyAccessedMemberTypes.PublicProperties)] T>(string name, T value) =>
        Write(name, (object?)value);

    public abstract bool IsEnabled(string name);

    public virtual bool IsEnabled(string name, object? arg1, object? arg2 = null)
    {
        return IsEnabled(name);
    }
}

The basic building block is DiagnosticSource, an abstract class that defines the abstract Write method used to log complex payloads. Each notification is given a name, which identifies it, and an optional object that carries information for the notification. In .NET 7, a new non-abstract, generic-based overload of the Write method was added to assist with AoT trimming, particularly when an anonymous type is used for the payload. We won’t dive too deeply into that rabbit hole, but the relevant issue for the API proposal is here.

An abstract IsEnabled method accepting a name is also defined, which can be used to check if a particular notification name is being observed. Although a best practice, producers are not guaranteed to check this before emitting notifications, so subscribers must account for that themselves. It is available as a performance optimisation that can avoid the allocation cost of creating payload objects when there are no subscribers for a particular notification.

A virtual IsEnabled(string name, object? arg1, object? arg2 = null) overload is also present. This overload defaults to calling IsEnabled(name) but can be overridden in derived types. The arguments allow additional context to be passed, which is then used to evaluate whether the notification is enabled.

To recap, DiagnosticSource defines the API that allows implementations to check if a particular notification is enabled, i.e., if one or more consumers are listening and wish to receive notifications. It also defines methods for writing a new notification that includes the name of the diagnostic notification and an optional payload typed to object.

DiagnosticListener

Of course, DiagnosticSource is abstract, so we need a derived implementation to use it. The framework includes the slightly confusingly named DiagnosticListener type for this. We can create instances of this type in our code to define a “source” of diagnostic notifications that can be observed by zero or more consumers.

DiagnosticListener inherits from DiagnosticSource, implementing its abstract methods. It also implements the IObservable<T> and IDisposable interfaces.

The IObservable<out T> interface is one of a pair of interfaces in C# used to implement the observer design pattern supporting push-based notifications. An IObservable type acts as a provider (referred to as the subject in the general observer pattern). It maintains a list of its observers, referred to as subscribers in the .NET implementation. This design allows the provider to notify any subscribers of new data without knowing in advance who will subscribe to it.

In the case of DiagnosticListener, the generic type used for the IObservable is KeyValuePair<string, object?>. You’ll notice that this matches the signature of the Write method defined on DiagnosticSource. The DiagnosticListener will provide all subscribers with each named notification written using the Write method.

The IObservable interface has been available in .NET for a long time. It defines a single generic method named Subscribe. This is how interested parties can register to receive notifications. Subscribe accepts a single parameter, an instance of an IObserver<out T>, and returns an IDisposable. Subscribers (observers) of a DiagnosticListener must, therefore, implement the IObserver<KeyValuePair<string, object?>> interface. When subscribing, the returned IDisposable allows a subscription to be removed should it no longer need to receive further notifications, even if the provider still has more to send. Any other subscribers will continue to receive notifications.

public DiagnosticListener(string name)
{
    Name = name;
    ...
}

The DiagnosticListener has a single constructor that accepts a string parameter. When creating an instance of DiagnosticListener, the name is used to identify the source of notifications. As we’ll see in a future post, this works in tandem with a mechanism for application code to be notified about all available diagnostic sources, with the ability to choose whether to subscribe based on the name of the source. I’ve omitted most of the code in the constructor for now. We’ll return to the constructor’s internals after discussing this conditional subscription mechanism later in this series.

As we learned earlier, implementations of IObservable must include a Subscribe method for observers to register their interest in receiving notifications. The DiagnosticListener class defines three overloads for the Subscribe method, the simplest of which is the method matching the signature of IObservable.Subscribe. We’ll focus on that for now and revisit the other overloads shortly.

public virtual IDisposable Subscribe(IObserver<KeyValuePair<string, object?>> observer)
{
	return SubscribeInternal(observer, null, null, null, null);
}

Subscribe calls and returns SubscribeInternal(observer, null, null, null, null) and here we begin to get deeper into the implementation details.

Remember that providers that implement IObservable should keep track of their observers so that they can send out notifications. An obvious way to achieve this would be to use a List<IObserver<T>> to store the subscriptions. That is undoubtedly an option, and the documentation for IObservable includes code samples of such an implementation.

DiagnosticListener takes a slightly different approach to achieve the same end goal. It prefers to define a linked list of observers. The code in SubscribeInternal maintains this linked list.

private DiagnosticSubscription SubscribeInternal(IObserver<KeyValuePair<string, object?>> observer,
    Predicate<string>? isEnabled1Arg, Func<string, object?, object?, bool>? isEnabled3Arg,
    Action<Activity, object?>? onActivityImport, Action<Activity, object?>? onActivityExport)
{
    if (_disposed)
    {
        return new DiagnosticSubscription() { Owner = this };
    }

    DiagnosticSubscription newSubscription = new DiagnosticSubscription()
    {
        Observer = observer,
        IsEnabled1Arg = isEnabled1Arg,
        IsEnabled3Arg = isEnabled3Arg,
        OnActivityImport = onActivityImport,
        OnActivityExport = onActivityExport,
        Owner = this,
        Next = _subscriptions
    };

    while (Interlocked.CompareExchange(ref _subscriptions, newSubscription, newSubscription.Next) != newSubscription.Next)
        newSubscription.Next = _subscriptions;

    return newSubscription;
}

Let’s discuss a few implementation details in the preceding code. First, we see that the return type is a DiagnosticSubscription. This type essentially wraps the observer along with a few additional properties to support the design of DiagnosticSource. Each DiagnosticSubscription is a node in a linked list of subscriptions and holds a reference to the next node. It also holds a reference back to its owner, the DiagnosticListener, to which the subscription belongs. It is a private class used internally and is designed to be read-only. An important point is that it implements IDisposable, allowing it to be used as the return type for the Subscribe method.

In the SubscribeInternal code, the subscription attempt is ignored if the DiagnosticListener has been disposed. A placeholder DiagnosticSubscription is returned to satisfy the IDisposable. A new DiagnosticSubscription is created in all other cases, with all fields populated. For the code path we’re describing, when this is called from the Subscribe(IObserver<KeyValuePair<string, object?>> observer) overload, only the first parameter will have been provided, which is the instance of IObserver<KeyValuePair<string, object?>> being subscribed. Therefore, many of the fields will be assigned a null value. The Next field is assigned a reference to the current head of the linked list of subscriptions. The while loop then updates the _subscriptions field on the DiagnosticSubscription using Interlocked.CompareExchange to manage this in a thread-safe, lock-free way. Generally, this should be a simple exchange to replace the reference to the current head with the new subscription. Only in a rare race condition might another subscriber be added at nearly the exact same moment, in which case, the head may have changed. In this case, whoever calls CompareExchange last will be updated to set the Next field to align with the head subscription and then attempt to exchange itself once again. Finally, the new subscription is returned.

So far, we’ve seen the code used to add a new subscription to the linked list. The observer pattern in .NET also supports the possibility that a subscriber may want to unsubscribe from further notifications sent by a provider. This is achieved by disposing of the subscription and is therefore implemented inside the DiagnosticSubscription.Dispose method.

private sealed class DiagnosticSubscription : IDisposable
{
    internal IObserver<KeyValuePair<string, object?>> Observer = null!;
    internal Predicate<string>? IsEnabled1Arg;
    internal Func<string, object?, object?, bool>? IsEnabled3Arg;
    internal Action<Activity, object?>? OnActivityImport;
    internal Action<Activity, object?>? OnActivityExport;

    internal DiagnosticListener Owner = null!;
    internal DiagnosticSubscription? Next;

    public void Dispose()
    {
        while (true)
        {
            DiagnosticSubscription? subscriptions = Owner._subscriptions;
            DiagnosticSubscription? newSubscriptions = Remove(subscriptions, this);    // Make a new list, with myself removed.

            if (Interlocked.CompareExchange(ref Owner._subscriptions, newSubscriptions, subscriptions) == subscriptions)
            {
                break;
            }
        }
    }

    private static DiagnosticSubscription? Remove(DiagnosticSubscription? subscriptions, DiagnosticSubscription subscription)
    {
        if (subscriptions == null)
        {
            return null;
        }

        if (subscriptions.Observer == subscription.Observer &&
            subscriptions.IsEnabled1Arg == subscription.IsEnabled1Arg &&
            subscriptions.IsEnabled3Arg == subscription.IsEnabled3Arg)
            return subscriptions.Next;

        return new DiagnosticSubscription() 
        { 
            Observer = subscriptions.Observer, 
            Owner = subscriptions.Owner, 
            IsEnabled1Arg = subscriptions.IsEnabled1Arg, 
            IsEnabled3Arg = subscriptions.IsEnabled3Arg, 
            Next = Remove(subscriptions.Next, subscription) 
        };
    }
}

The Dispose method is designed to be lock-free and relies on the fact that the linked list is read-only. It first makes a copy of the current subscriptions for the owning DiagnosticListener. It then creates a new copy of the linked list with itself removed. The removal is handled by the Remove method, which first checks for the possibility that the subscriptions are already null, which could occur if the owning DiagnosticListener has also been disposed of.

The simplest scenario is that the subscription being removed could be at the head of the linked list. The second conditional block checks for and handles this situation by simply returning the next subscription, which should be promoted to the new head of the list.

The other scenarios are more complicated as a subscription needs to be removed from somewhere within the list. This is handled by recursively calling Remove. A new DiagnosticSubscription is returned on each invocation, and the Next property is potentially updated with a new reference after the required subscription has been removed. Once the expected subscription is removed, the recursive call ends, and we’ll have an updated linked list of subscriptions.

Back in the Dispose method, the updated list is swapped with the current list using Interlocked.CompareExchange in the same way we saw when a subscription was added. A race may occur here between a subscription being added and one being removed. The while loop with CompareExchange protects against this by ensuring that if the current _subscriptions linked list on the owner no longer matches the copy taken when the loop iteration started, it is not replaced. The process then starts again with an updated copy.

Using DiagnosticSource and DiagnosticListener

We’ve reviewed how to subscribe to and unsubscribe from notifications from a DiagnosticListener. In this first post in this series, we’ll conclude by writing code instrumented with a DiagnosticListener and creating a subscriber to its notifications.

NOTE: The following usage examples are valid but not the recommended way for subscribing to notifications from DiagnosticSources in application code. We’ll visit the recommended pattern once we’ve dug into some of the other implementation details. This approach is suitable if you have a provider (DiagnosticSource) in the same assembly as the observer of its notifications.

We’ll begin by creating a class and a DiagnosticListener.

using System.Diagnostics;

namespace MyNamespace;

public class DoStuff
{
    internal static readonly DiagnosticListener Listener = new("MyNamespace.MySource");
}

In the preceding code, we define a DoStuff method. At this point, we have a static field named Listener, an instance of DiagnosticListener. Remember that DiagnosticListener derives from DiagnosticSource, so this will also be our source of notifications. We have provided a name for this provider in the constructor. Names should be globally unique, so the recommended best practice is to prefix them with the containing namespace.

Two things in the above code are not following the recommended standard pattern. First, the Listener field is typed as DiagnosticListener, when typically, we type this to DiagnosticSource since we only need to call methods on DiagnosticSource to write events. The second deviance from the guidance is that the field is internal when typically it would be private. This recommendation is present as there is a more general, decoupled way to subscribe to notifications where no one outside of the DoStuff class needs direct access to the DiagnosticListener. For this example, I’m ignoring those guidelines so that we can focus on what we’ve explored so far.

Once we have a DiagnosticSource available, we can use it to emit diagnostic notifications.

public class DoStuff
{
    internal static readonly DiagnosticListener Listener = new("MyNamespace.MySource");
    
    private static readonly DiagnosticSource _diagnostics = Listener;

    public static void DoIt(string msg)
    {
        if (_diagnostics.IsEnabled("DoIt"))
            _diagnostics.Write("DoIt", msg);
    }
}

In the above code, I’ve added a static read-only DiagnosticSource field to follow the usage guidelines more closely. Here, I’m assigning it with a reference to the DiagnosticListener instance.

When we have code we want to instrument, we can use the methods defined on DiagnosticSource. The DoIt method can emit a diagnostic notification in the preceding code by calling the Write method. As we learned when exploring the DiagnosticSource code, this method accepts a required string and an optional object. The string is expected to be the name of the event being written. The guidance recommends that these names be short to limit the performance overhead of diagnostics. The name should be unique within all events written by a named DiagnosticListener. Event names can be reused by different DiagnosticListener instances without any issues. This is important because it means runtime code, third-party libraries, and application code can all be instrumented using DiagnosticSource without the risk of conflicting events. This is why it is essential to ensure the name of the DiagnosticListener is unique and prefixed with a unique namespace.

We pass along the string ‘msg’ parameter in the above code as the second argument. An object is generally passed to provide additional data relating to the notification. It is also reasonable to pass null when no such data is required. In this contrived example, we don’t have much data to pass with each notification, so forwarding the string directly is appropriate. When we have additional data to include, we can either send an instance of a type that includes properties containing the data, or we can use an anonymous type, like so.

public static void DoIt(string msg)
{
    if (_diagnostics.IsEnabled("DoIt"))
        _diagnostics.Write("DoIt", new { Message = msg });
}

In the above code, we create an anonymous type with a Message property that we assign from the ‘msg’ parameter. This has no real benefit for this example, but it is enough to describe the usage. We’ll switch back to directly passing the string message object for the remainder of the example.

Anonymous types are the default choice recommended by the guidance for several reasons. First, you can add new properties to them at any time in a compatible way. Second, subscribers don’t necessarily need to depend directly on your assembly to reference the type information required to handle notifications (although this is possible to avoid by using reflection).

That said, this recommendation does bring about some challenges. The biggest is that subscribers will need to use reflection to access data from the anonymous type. This introduces overhead and makes code less AoT trimmer compatible.

The guidance goes on to suggest considering the use of explicitly typed object instances for notification payloads. The principal value is that consuming code can cast the object to a known payload type to work with it more easily. This avoids the need for expensive reflection and reviewing the producer source code to learn about the properties exposed on the payload. The only downside is that the type needs to be publicly exposed by the instrumented assembly, and any changes can break the consuming code. In reality, this is a small price to pay, in my opinion. In some cases, this choice also enables caching and reuse of read-only payloads, which can avoid unnecessary object allocation for each new event being written.

It’s worth also touching on the fact that since the payload is ultimately treated as an object, any ValueTypes used for payloads will be boxed, which brings a small performance penalty. Sadly, even the newer Write<T>(string name, T value) overload doesn’t solve this because it’s only added to support some trimmer attributes. As it is non-abstract, it always calls down to the abstract Write method, casting the T to an object at that point.

The last thing to focus on in the code above is the guard call to IsEnabled before we write the event. This is a recommended performance optimisation. It’s entirely possible that a DiagnosticListener has no subscribers observing its notifications. In that case, there’s no value in writing the event. Furthermore, without the IsEnabled check, the above code would create a new instance of the anonymous type on every invocation of DoIt, even when no one is observing notifications. To avoid unnecessary allocations, we should perform a check beforehand to see if we have at least one observer for the DoIt event. We have yet to see the code (and will save that for a future post in this series), but as well as subscribing to all events, observers can subscribe with a predicate used to identify the notifications they are interested in.

In the interest of completeness, there is one slightly more optimised way of checking for observers before writing an event. It’s not available on DiagnosticSource but is present on DiagnosticListener.

namespace DiagnosticsExample;

public class DoStuff
{
    internal static readonly DiagnosticListener Listener = new("MySource");

    public static void DoIt(string msg)
    {
        if (Listener.IsEnabled && Listener.IsEnabled("DoIt"))
            Listener.Write("DoIt", msg);
    }
}

An advantage of preferring a field typed as DiagnosticListener when adding diagnostics to our code is that we can opt to call its IsEnabled() overload before calling IsEnabled(string). The implementation performs a speedy null check of the _subscriptions field, which is the fastest way to assess if there are any subscribers at all.

public bool IsEnabled()
{
    return _subscriptions != null;
}

This approach is used by code in the Microsoft libraries such as Microsoft.Extensions.Hosting.

Putting It All Together

As we draw to a close on this post, let’s explore the consuming code to create and subscribe an observer to our DiagnosticListener. We’ll begin by defining our observer.

public class Observer : IObserver<KeyValuePair<string, object?>>
{
    public void OnCompleted() => 
        Console.WriteLine("DiagnosticListener was disposed!");

    public void OnError(Exception error) { }

    public void OnNext(KeyValuePair<string, object?> value) => 
        Console.WriteLine($"{value.Key}: {(string)value.Value!}");
}

The preceding code defines a class named MyObserver, which, as we learned earlier, should implement the IObserver<KeyValuePair<string, object?>> interface. An observer must implement three methods that the observable may call on subscribers to push notifications to them.

OnNext is the most important of these callback methods. The observable will call this each time it has new data to provide to subscribers. In the case of DiagnosticListener, this will be called for each diagnostic notification that is written. In the code above, we log a string to the console. We use an interpolated string containing the key of the KeyValuePair, which will be the string name of the notification. Our observable sends a string as the payload for the notification, so we can also cast the Value from the KeyValuePair back to a string and include that in our console message. As we own both the observable and observer code, we can safely use the null-forgiving (aka. dammit) operator to ignore the potential for a null object. In production code, we would need to take more care with null checking and the cast of the value.

OnCompleted is only called if and when the observable DiagnosticListener is disposed. It will, therefore, no longer provide notifications to subscribers. The example code above just logs this situation to the console.

No code invokes the OnError callback in the DiagnosticListener implementation, so we don’t bother to implement it here.

With an IObserver defined, we can now wire things up in the program file for this sample.

using System.Diagnostics;
using DiagnosticsExample;

DoStuff.DoIt("Before subscribing");
var subscription = DoStuff.Listener.Subscribe(new Observer());
DoStuff.DoIt("After subscribing 1");
DoStuff.DoIt("After subscribing 2");
subscription.Dispose();
DoStuff.DoIt("After unsubscribing");

Console.WriteLine("DONE!");

The preceding top-level statements invoke the static DoIt method several times. After the first invocation, we subscribe a new Observer directly to the DiagnosticListener instance from the DoStuff method. This particular approach is less common since it requires a direct dependency on the DiagnosticListener. This is possible in our example since the types are all defined in the same assembly, and we made the DiagnosticListener internal. In a future post, we’ll cover the more common technique to subscribe to one or more DiagnosticListeners without a direct dependency on them.

After subscribing, the code calls DoIt twice before we dispose of the subscription. After disposal, we call DoIt one more time.

When we run this code, we see the following console output:

DoIt: After subscribing 1
DoIt: After subscribing 2
DONE!

Note that we only observe the two notifications that occur after subscribing and before we dispose of our observer. Each notification passes along the string argument of the DoIt call, and we see it included in the console messages.

Before we conclude, let’s look at one final consumer example that may reveal a bug in DiagnosticsListener.

_ = DoStuff.Listener.Subscribe(new Observer());
DoStuff.DoIt("After subscribing 1");
DoStuff.Listener.Dispose();
DoStuff.DoIt("After disposal");

Console.WriteLine("DONE!");

In the preceding code, we subscribe to the listener before invoking the DoIt method. We then dispose of the DiagnositicListener before invoking DoIt one more time.

When we run this code, the following is written to the console:

DoIt: After subscribing 1
DiagnosticListener was disposed!
DoIt: After disposal
DONE!

After disposal of the DiagnosticListener, with the current implementation, when DoIt is next invoked, any previously subscribed observers are still notified. I’m not convinced this is the original intention. Let’s look at the implementation:

public virtual void Dispose()
{
    lock (s_allListenersLock)
    {
        if (_disposed)
        {
            return;
        }
        _disposed = true;
        
		...
    }
	
    // Indicate completion to all subscribers.
    DiagnosticSubscription? subscriber = null;
    Interlocked.Exchange(ref subscriber, _subscriptions);
    while (subscriber != null)
    {
        subscriber.Observer.OnCompleted();
        subscriber = subscriber.Next;
    }
    // The code above also nulled out all subscriptions.
}

We can ignore most of this for now, so I’ve skipped some of the code. The two important points are that the _disposed field is set to true (on the first disposal). Later, the OnCompleted callback is called for each subscriber to notify them that no further notifications should be expected. After disposed is set to true, no further subscriptions are created because SubscribeInternal (which we looked at earlier) checks this field and is essentially no-ops.

The comment suggests that all subscriptions are nulled out, which doesn’t appear to be true. I’ve raised a runtime issue to report this potentially incorrect behaviour and propose a fix.

My suggestion is to amend the last lines of code to the following:

DiagnosticSubscription? subscriber = null;
subscriber = Interlocked.Exchange(ref _subscriptions, subscriber);
while (subscriber != null)
{
    subscriber.Observer.OnCompleted();
    subscriber = subscriber.Next;
}

The difference here is that the code reverses the arguments to Interlocked.Exchange, to replace the _subscriptions field with a null reference. The returned original value is then stored in the local variable in order to notify the subscriber of completion, as before.

With this change, the output logged to the console is:

DoIt: After subscribing 1
DiagnosticListener was disposed!
DONE!

This output is what I would expect since we no longer want to notify subscribers once the listener is disposed of. Perhaps I’m wrong in that assumption, but hopefully, the issue will answer this one way or another.

UPDATE: The above does indeed appear to be a bug, so I’ve submitted a PR to correct this aspect of the DiagnosticListener implementation.

Conclusion

If you’ve stuck with me this far, well done! This has been a lengthy blog post that I hope goes beyond simply demonstrating the public API of DiagnosticSource and DiagnosticListener. It also examines the internal implementation to let us gain a deeper appreciation of the fine details.

Along the way, we explored the abstract DiagnosticSource class and its implementation in DiagnosticListener. We have learned about the IObservable<T> and IObserver<T> interfaces and how they are used to push notifications to one or more subscribers, which do not have to be known by the DiagnosticSource instrumentation.

We concluded by using the APIs to perform a basic subscription and saw how code can be instrumented with diagnostic notifications using an instance of DiagnosticSource.

I plan to continue this series with further posts that dive into another mechanism for subscribers to register their interest in notifications without tightly coupling that code to the sources of those notifications.