Concurrent Hosted Service Start and Stop in .NET 8

In this post, I will describe a new feature of the Microsoft.Extensions.Hosting library coming in .NET 8 (available since preview 4) affecting hosted services. Let’s first begin with a brief recap of hosted services. The hosting library for .NET, used in both the ASP.NET Core project template and the Worker Service template, provides the capability to start a long-running console application.

In the case of ASP.NET Core, the application starts the Kestrel webserver and runs until the process is shut down. The Worker Service template is perfect for creating lightweight processing applications which run continually. For example, this can include microservices which poll for messages from a queue and process them.

Hosted services are essentially background tasks started by the host which perform the work of the application. Applications may define one or more hosted services by registering an implementation of the IHostedService interface with the Microsoft dependency injection container. Microsoft provides a general implementation of this interface with the BackgroundService abstract class, which provides the core machinery required to start some long-running work.

To use BackgroundService, developers derive from this class and provide an implementation for the abstract ExecuteAsync method. When the hosting framework starts, it will trigger the Task defined in the ExecuteAsync method.

Before .NET 8, the code that starts and stops hosted services does so sequentially. Each IHostedService registered with the DI container is started in sequence by calling the StartAsync method on the instance. Crucially, this included awaiting the completion of the StartAsync method, meaning that each StartAsync Task was required to complete before the next service was triggered. The effect of this design does not significantly impact most applications, but it is still possible for this default behaviour to cause issues. Even though it’s recommended that there be little work inside StartAsync, it’s possible for a slow hosted service to delay the start of the remainder of the application.

When stopping applications, the same sequential behaviour occurred, this time with services being stopped in reverse order. Again, each StopAsync method for the registered services would be called with an await, causing them to run one after another. This behaviour can be more problematic at shutdown as there is a timeout configured that limits how long the graceful shutdown can take. This means that your timeout must consider the length of time needed to shut down each hosted service gracefully. In some situations, when draining currently in-process work, this could cause some services to consume most of the timeout window.

For those developers who have run into issues with the existing behaviour, good news! In .NET 8, we are gaining two new options which allow us to switch to a concurrent start and/or stop behaviour. To achieve this, we can configure the HostOptions and set one or both of the new properties to true to enable concurrent logic for starting or stopping any registered IHostedServices.

To configure concurrent start logic, we must set the ServicesStartConcurrently property to true. To configure concurrent shutdown, we can use ServicesStopConcurrently.

using WorkerService;

var host = Host.CreateDefaultBuilder(args)
    .ConfigureServices(services =>
    {
        services.Configure<HostOptions>(options =>
        {
            options.ServicesStartConcurrently = true;
            options.ServicesStopConcurrently = true;
        });
        services.AddHostedService<WorkerOne>();
        services.AddHostedService<WorkerTwo>();
    })
    .Build();

host.Run();

In the preceding code, we use the Configure method to configure the HostOptions, setting start and stop behaviour to run concurrently. In the above example, which registers two hosted services, they will start and stop concurrently without delaying the other.

Let’s explore the new code for the Host implementation:

if (_options.ServicesStartConcurrently)
{
	List<Task> tasks = new List<Task>();

	foreach (IHostedService hostedService in _hostedServices)
	{
		tasks.Add(Task.Run(() => StartAndTryToExecuteAsync(hostedService, combinedCancellationToken), combinedCancellationToken));
	}

	Task groupedTasks = Task.WhenAll(tasks);

	try
	{
		await groupedTasks.ConfigureAwait(false);
	}
	catch (Exception ex)
	{
		exceptions.AddRange(groupedTasks.Exception?.InnerExceptions ?? new[] { ex }.AsEnumerable());
	}
}

private async Task StartAndTryToExecuteAsync(IHostedService service, CancellationToken combinedCancellationToken)
{
	await service.StartAsync(combinedCancellationToken).ConfigureAwait(false);

	if (service is BackgroundService backgroundService)
	{
		_ = TryExecuteBackgroundServiceAsync(backgroundService);
	}
}

When ServicesStartConcurrently is set to true, internally, the code for the Host start logic will use Task.Run to ultimately invoke the StartAsync method on each registered hosted service. These Tasks are still triggered in the order they are registered, but crucially, they are not awaited at this point. The tasks are added to a list, and only once each Task has been triggered is WhenAll used to await the completion of StartAsync for each hosted service. In this mode, the startup will take only as long as the slowest StartAsync call, allowing the application as a whole to get going more quickly. It’s a similar story with the shutdown code:

if (_options.ServicesStopConcurrently)
{
	List<Task> tasks = new List<Task>();

	foreach (IHostedService hostedService in hostedServices)
	{
		tasks.Add(Task.Run(() => hostedService.StopAsync(token), token));
	}

	Task groupedTasks = Task.WhenAll(tasks);

	try
	{
		await groupedTasks.ConfigureAwait(false);
	}
	catch (Exception ex)
	{
		exceptions.AddRange(groupedTasks.Exception?.InnerExceptions ?? new[] { ex }.AsEnumerable());
	}
}

When it comes to shutdown, if ServicesStopConcurrently is set to true, then each StopAsync method will be called without awaiting the result. Each Task will be added to a List of tasks that can later be awaited together with WhenAll. This has the crucial effect of allowing each hosted service the entire length of the configured ShutdownTimeout to complete its work. A single slow StopAsync will no longer eat up the timeout remaining for subsequent services.

This is a small but welcome addition to the options for the hosting library. The existing sequential behaviour remains the default, but we can now easily configure the concurrent modes when suitable for our applications.

Generally, switching to the concurrent mode is expected to be safe for most applications. One thing to consider is any situations where your hosted services depend directly on one another in some crucial way. In such situations, starting or stopping them concurrently may cause bugs in your application, as you cannot rely on services registered before your service being completely started.

.NET 8 is still in preview at the time of writing, and you can try out this new feature by downloading preview 4 or newer of the .NET SDK.