On a recent project we needed to implement the concept of correlation IDs across some ASP.NET Core API services. In our scenario we have an API service called from a front end JavaScript application which will then call one or more secondary back end API services to gather data. This is becoming a pretty common requirement when we have the concept of “microservices” which communicate over HTTP.
Why we need correlation IDs?
A problem that arises with having multiple services separated into distinct units is how we track the flow of a single user request through each of the individual services that might be involved in generating a response. This is particularly crucial for logging and diagnosing faults with the system as a whole. Once the request leaves the first API service and passes control onto the backend API service, we start to struggle to interpret the logs. A failure in the backend API will likely impact the front end API service but those might appear to be two unrelated errors. We need a way to view the entire flow from when the request hit the front end API, through to the backend API and back again.
Solution
This is where the concept of correlation IDs comes into play. A correlation ID is simply a unique identifier that is passed through the entire request flow and crucially is passed between the services. When each service needs to log something, it can include this correlation ID, thereby ensuring that we can track a full user request through from start to finish. We can parse all of the logs from each distinct part of the system, looking for the correlation ID. We can then form a full picture of the lifetime of that request and hopefully identify root causes of any errors.
In this post I’m going to discuss a very simple library that I wrote to help solve this problem. It’s a pretty simple implementation and fits specifically the requirement we needed to achieve. I have some thoughts about extending it to make working with it a bit simpler. I’d like to expose the correlation ID via some kind of context class that allows you to more easily access the ID without the dependency on IHttpContextAccessor in places where the HttpContext is not already exposed. For now, let’s look at how this initial version works.
I hope this will serve as a nice real-world example of how it’s quite simple to build and package a middleware library that you can re-use across your own projects.
Correlation ID Options
As part of the solution I wanted to provide a mechanism to configure the middleware and so I followed the standard pattern used in ASP.NET Core to provide a class representing the configuration properties. My middleware is pretty straightforward so only has two options.
There is an option to control the name of the request header where the correlation ID will be passed between services. Both the sending API and receiving API need to use the same header name to ensure they can locate and pass on the correlation ID. I default this to “X-Correlation-ID” which is a standard name that I found for this type of header.
The second option controls whether the correlation ID is included in the HttpResponse. I included this for our scenario so that the final front end API can pass the ID down to the front end SPA code. This allows us to tie up and client side logging with the backend logs. This feature is enabled by default but can be disabled in the configuration.
Middleware
The main part of the solution involves a piece of middleware and an extension method to provide an easy way to register the middleware into the pipeline. The structure of this code follows the patterns defined as part of ASP.NET Core. The middleware class looks like this:
The constructor accepts our configuration via the IOptions<T> concept which is part of the ASP.NET Core framework. We will use these options values to control the behaviour of the middleware.
The main Invoke method, which will be called by the framework, is where the “magic” happens.
First we check for an existing correlation ID coming in via the request headers. We check for a header matching the name configured in the options and if we find a matching value we set the TraceIdentifier property on the HttpContext. I take advantage of one of the improvements from C# 7 which means we can declare the out variable inline, rather than having to pre-declare it before the TryGetValue line.
If we do not find a correlation ID on the request header we just utilise the existing TraceIdentifier which is generated automatically.
The next block is optional, controlled by the IncludeInResponse configuration property. If true (the default) we use the OnStarting callback to allow us to safely include the correlation ID header in the response.
Extension Method
The final piece of code needed (although not absolutely required) is to include some extension methods to make adding the middleware to the pipeline easier for consumers. The code in that class looks like this:
It provides a UseCorrelationId method (and two overloads) which are extension methods on the IApplicationBuilder. These will register the CorrelationIdMiddleware and if provided handle taking a custom name for the header, or a complete CorrelationIdOptions object which will ensure the middleware behaves as expected.
With this in place, all that a consumer of the library needs to do is include it in their project and then call the appropriate UseCorrelationId method from the Configure method in the startup class.
Summary
Middleware presents a very easy way to build the logic required by the concept of correlation IDs into and ASP.NET Core application. With a few small components I was able to put together a basic solution to solve our immediate problem. This was my first time releasing my own library as open source and publishing a public Nuget package. I’ve decided to summarise the steps I took for that in my next blog post.
If you want to review the full source, you can check it out on GitHub.
If you want to pull in the NuGet package to your work, you can find it up on Nuget.org.
Have you enjoyed this post and found it useful? If so, please consider supporting me:
>I’d like to expose the correlation ID via some kind of context class that allows you to more easily access the ID without the dependency on IHttpContextAccessor in places where the HttpContext is not already exposed.
So what do you do with it, other than assign it to the TraceIdentifier? I still need access to the HttpContext to get at it now if I wanted to stick it into my Redis instance, or into a local scoped dictionary.
Hi Henk,
That’s correct. In this version I’m simply setting the TraceIdentifier to match an incoming correlation ID if one is presented in the header. In our use case this was all we needed since we can use the IHttpContextAccessor to get the HttpContext and the value for logging. I plan to consider options to expose the correlation ID separately from the the HttpContext.
Steve
Hi Steve, I have the same design concern as @Henk mentioned above, where I need to pass Correlation Id to components that do not reference httpContext. Do you have any updates ??
Thanks,
Hi Abhishek,
No update I’m afraid. I’ve not need this so haven’t update the library yet. Still looking to review it at some point. Depending on your dependencies you may have IHttpContextAccessor already available which you can use to get access to the HttpContext. Is that an option in your case?
Steve
Sorry for the delay. If you use Identity it will be registered in DI so you can just ask for it in your constructor of any classes where you want to access the HttpContext. If you’re not using Identity you’ll need to register it. See https://github.com/aspnet/Hosting/issues/793
I’d recommend using some sort of log context. If you’re using Serilog, you can attach an Enricher for the entire context using LogContext.Push. Or you can add the property to all further log messages by using LogContext.PushProperty.
That will get all further downstream messages including it in log responses.
If you want access to it outside of a logger, you’ll probably need to do something similar, but on your own.
Thanks very much for this post! I was able to use it to add the correlation ID to the request as you demonstrate and further add a bit more middleware to push that ID into a SeriLog context that was now usable in different app layers.
Hi Stephen. Thanks for the feedback. I’m very pleased to hear that this has been useful.