Updates to my ASP.NET Core Correlation ID Library Supporting correlation IDs across ASP.NET Core microservices.

Back in May 2017 I blogged about creating a simple library which supports passing correlation IDs between ASP.NET Core micro-services. The library came about because of a basic requirement we had at work to pass an identifier between related services to enable more useful error logging. By passing an identifier from the first service, through to any further services it then calls; if an exception occurs, we can quickly search for the entire history of that request across the distributed environment.

Since I released that first version to NuGet I have been staggered by the download stats. According to NuGet it now has nearly 27,000 downloads at the time of writing this post. I never really expected it to be used that heavily so this is a really pleasant surprise. I’m very pleased that something I’ve created is helping others with a similar requirement. It is a little daunting to think that so many people are dependent on that library in their code!

Three months ago I released version 2.0 of the library which added the concept of a CorrelationContext. This was a something I’d been considering almost immediately after completing version 1.0. An issue with v1 was that I’d chosen to set the TraceIdentifier on the HttpContext to match the correlation ID being passed in via the request headers. In controllers, where the HttpContext is accessible, this was not a major issue since the value of TraceIdentifier could then be read and used in logging. However, to use the correlation ID elsewhere, the only way to access it was via the IHttpContextAccessor. This isn’t registered in ASP.NET Core by default and so for some users of the library meant they would have to register it to make full use of the correlation ID.

I based my version 2 changes on the HttpContext and HttpContextAccessor in ASP.NET Core and this seems to have worked quite nicely so far. This required a breaking change for the library since it needed to register some services to support the new CorrelationContext.

Today I released version 2.1 of the library. This version adds two new configuration options that can be set when registering the middleware. One of these options came as a result of a GitHub issue requesting that it be possible to disable updating the TraceIdentifier with the correlation ID. This is now possible since the ID is passed around in the CorrelationContext and I needn’t rely on the HttpContext. To avoid breaking changes I added the option with it default setting behaving as it did before. I may look to change this default in the next major release.

I took the opportunity to add another new option that determines if the correlation ID should be matched to the TraceIdentifier or whether it should be a GUID in situations where an ID is not present in the header. For some users I can see this being useful and at work I’m considering the move to a GUID for our correlation ID.

A final change I was able to incorporate came as a result of another feature request via the projects GitHub issues. In this case it was to include the configured correlation ID header name on the CorrelationContext. This resulted in the first external PR on the project which I was very happy to receive. Thanks to Julien for his contribution.

I hope that people using the library are happy with these changes. As ever I’m happy to take feedback and ideas via GitHub if there are use cases that it doesn’t currently support.

UPDATE: It seems I wasn’t as careful as I thought about breaking changes. One did slip through in this release as I updated the interface and implementation for the ICorrelationContextFactory to support the new property on the context. If you’re consuming the library this is not something you generally need to access but if you’re mocking for unit testing it’s possible this will break there. Apologies! Turns out it’s harder than you think avoid breaking changes when released publicly!

ASP.NET Core Correlation IDs Writing a basic middleware library to enable correlation IDs on ASP.NET Core

On a recent project we needed to implement the concept of correlation IDs across some ASP.NET Core API services. In our scenario we have an API service called from a front end JavaScript application which will then call one or more secondary back end API services to gather data. This is becoming a pretty common requirement when we have the concept of “microservices” which communicate over HTTP.

Why we need correlation IDs?

A problem that arises with having multiple services separated into distinct units is how we track the flow of a single user request through each of the individual services that might be involved in generating a response. This is particularly crucial for logging and diagnosing faults with the system as a whole. Once the request leaves the first API service and passes control onto the backend API service, we start to struggle to interpret the logs. A failure in the backend API will likely impact the front end API service but those might appear to be two unrelated errors. We need a way to view the entire flow from when the request hit the front end API, through to the backend API and back again.

Solution

This is where the concept of correlation IDs comes into play. A correlation ID is simply a unique identifier that is passed through the entire request flow and crucially is passed between the services. When each service needs to log something, it can include this correlation ID, thereby ensuring that we can track a full user request through from start to finish. We can parse all of the logs from each distinct part of the system, looking for the correlation ID. We can then form a full picture of the lifetime of that request and hopefully identify root causes of any errors.

In this post I’m going to discuss a very simple library that I wrote to help solve this problem. It’s a pretty simple implementation and fits specifically the requirement we needed to achieve. I have some thoughts about extending it to make working with it a bit simpler. I’d like to expose the correlation ID via some kind of context class that allows you to more easily access the ID without the dependency on IHttpContextAccessor in places where the HttpContext is not already exposed. For now, let’s look at how this initial version works.

I hope this will serve as a nice real-world example of how it’s quite simple to build and package a middleware library that you can re-use across your own projects.

Correlation ID Options

As part of the solution I wanted to provide a mechanism to configure the middleware and so I followed the standard pattern used in ASP.NET Core to provide a class representing the configuration properties. My middleware is pretty straightforward so only has two options.

There is an option to control the name of the request header where the correlation ID will be passed between services. Both the sending API and receiving API need to use the same header name to ensure they can locate and pass on the correlation ID. I default this to “X-Correlation-ID” which is a standard name that I found for this type of header.

The second option controls whether the correlation ID is included in the HttpResponse. I included this for our scenario so that the final front end API can pass the ID down to the front end SPA code. This allows us to tie up and client side logging with the backend logs. This feature is enabled by default but can be disabled in the configuration.

Middleware

The main part of the solution involves a piece of middleware and an extension method to provide an easy way to register the middleware into the pipeline. The structure of this code follows the patterns defined as part of ASP.NET Core. The middleware class looks like this:

The constructor accepts our configuration via the IOptions<T> concept which is part of the ASP.NET Core framework. We will use these options values to control the behaviour of the middleware.

The main Invoke method, which will be called by the framework, is where the “magic” happens.

First we check for an existing correlation ID coming in via the request headers. We check for a header matching the name configured in the options and if we find a matching value we set the TraceIdentifier property on the HttpContext. I take advantage of one of the improvements from C# 7 which means we can declare the out variable inline, rather than having to pre-declare it before the TryGetValue line.

If we do not find a correlation ID on the request header we just utilise the existing TraceIdentifier which is generated automatically.

The next block is optional, controlled by the IncludeInResponse configuration property. If true (the default) we use the OnStarting callback to allow us to safely include the correlation ID header in the response.

Extension Method

The final piece of code needed (although not absolutely required) is to include some extension methods to make adding the middleware to the pipeline easier for consumers. The code in that class looks like this:

It provides a UseCorrelationId method (and two overloads) which are extension methods on the IApplicationBuilder. These will register the CorrelationIdMiddleware and if provided handle taking a custom name for the header, or a complete CorrelationIdOptions object which will ensure the middleware behaves as expected.

With this in place, all that a consumer of the library needs to do is include it in their project and then call the appropriate UseCorrelationId method from the Configure method in the startup class.

Summary

Middleware presents a very easy way to build the logic required by the concept of correlation IDs into and ASP.NET Core application. With a few small components I was able to put together a basic solution to solve our immediate problem. This was my first time releasing my own library as open source and publishing a public Nuget package. I’ve decided to summarise the steps I took for that in my next blog post.

If you want to review the full source, you can check it out on GitHub.

If you want to pull in the NuGet package to your work, you can find it up on Nuget.org.