In this post, we will explore a couple of options to apply health checks to your gRPC server running on ASP.NET Core 3.0. Health checks are a common requirement. In particular, in containerised environments, they are required so that the container orchestrator and load balancers know which services are actually functioning.
This post is part of my gRPC and ASP.NET Core series.
Option 1: Using the ASP.NET Core Health Check Middleware
The first quite simple option is to leverage the exiting health check middleware which is documented within ASP.NET Core docs. This middleware supports various configurations to suit many requirements. It adds at least one health check endpoint to your application, reachable via a regular HTTP request.
For a basic health check, we can add the feature to an ASP.NET Core application by first registering the health check services within the
We then map a health check endpoint under the /health path using the
On line 4 of the above code, we use the
MapHealthChecks method to register a simple health check endpoint.
When our application is running, we can make a get request to the configured endpoint. On my local machine, I can request “https://localhost:5005/health” which returns a 200-OK response with “Heathy” as the response body.
I won’t go into more advanced configurations for the health check feature in this post.
Option 2: Using the Health Checks Library
The gRPC framework specifies a gRPC health checking protocol. It uses the same approach of defining a service definition as for regular gRPC services. Health checking is just another set of RPC methods.
A common service definition is available and defined using a proto file called health.proto, in the proto package grpc.heath.v1.
The service definition includes two RPC methods currently:
‘Check‘ is a unary service method which returns a HealthCheckResponse when a HealthCheckRequest is received.
The HealthCheckRequest is defined as follows:
The request includes a single string field called ‘service‘. A gRPC server may be handling multiple gRPC service definitions. gRPC health checks support the concept of requesting a health check status for each specific service. The string is used to identify the service for which the health check status should be returned.
The ‘Check‘ method returns the appropriate HealthCheckResponse if the provided service name is known. In cases where it is not known, the response from the gRPC server will be NOT_FOUND.
The response message has the following definition:
The ‘ServingStatus‘ enum is used to define some possible statuses that may be returned. Serving indicates that the server is running and ready to handle calls for the service. NOT_SERVING identifies that the opposite i strue and that the server is not currently healthy.
The ‘Watch‘ method defines a way for the server to stream status changes for a given service. When called from a client, an immediate response will be returned indicating the current status, which may be NOT_SERVING or SERVICE_UNKNOWN, for example. The call will remain open so that if the status changes on the server, an updated response will be sent to the client.
A server can provide an implementation for this service definition to support health check RPC calls from clients and infrastructure such as load balancers. We could, of course, implement this ourselves but for basic requirements the Grpc.HealthCheck NuGet package can be included in your server project. This package contains an implementation of the ‘Check‘ method that we can utilise.
To get started, add a package reference to your server project using your preferred approach. Once complete, you should have a PackageReference included in your project file (csproj).
<PackageReference Include="Grpc.HealthCheck" Version="2.24.0" />
This package includes a partial implementation of the health.proto service definition in the class
HealthServiceImpl. This class includes an implementation for only the
Check method currently. The class defines three other methods.
public void SetStatus(string service, HealthCheckResponse.Types.ServingStatus status);
public void ClearStatus(string service);
public void ClearAll();
These methods are used to register and update the state of the various gRPC services for the server. Internally, a
Dictionary is used to maintain the values for each registered service. Our job is to ensure we set the appropriate status(es) and maintain them.
It’s not immediately obvious how best to achieve this. In the end, I settled on using a BackgroundService, a feature of the .NET Core hosting model. We’ll use a basic approach in this post to demonstrate a way to get started.
This class extends the
BackgroundService provided by the .NET Core hosting libraries. It supports dependency injection, and we expect two services to be injected into the constructor. The first is the
HealthServiceImpl which comes from the Grpc.HealthCheck NuGet package we included earlier. The second is the
HealthCheckService from the Microsoft.Extensions.Diagnostics.HealthChecks package.
In our override for the
ExecuteAsync method, we include some logic which sets the statuses for our services. This occurs in a while loop which will exit only if the application is shutting down, signalled by the
We use the
CheckHealthAsync method on the
HealthCheckService to request the general health status for the application. In more complex, real-world applications, there may be many other things which have to be checked to establish a healthy state; such as the ability to connect to a database or reach a downstream dependency. It is possible to include those checks within the broader Microsoft.Extensions.Diagnostics health checks or as additional, gRPC specific code in this method.
Based on the returned
HealthStatus from the
CheckHealthAsync method, we then set the appropriate
ServingStatus on our gRPC service.
The gRPC Health Checking Protocol specifies that the server should also return the overall server status as a response to a service name that is an empty string. We, therefore, ensure we include a status for both the “Weather” service and, for an empty string, the overall server.
ExecuteAsync code then delays (asynchronously) for 15 seconds, before the loop continues and the statuses may update. This is basic logic which ensures that the statuses are refreshed periodically.
Registering and Mapping the gRPC Health Check Implementation
Finally, we must register and map an endpoint to the health service. To support our approach here, we have to do some extra work. Since we need the statuses, set through the
HealthServiceImpl class to be globally available and consistent within the application, we need to manually register this class as a singleton service with the dependency injection container. Additionally, we must register our
IHostedService so that it is started when the application hosting begins.
We achieve this in the
ConfigureServices method of the
The first line registers the
HealthServiceImpl as a singleton. This ensures that the same instance, with the same dictionary of statuses, is used throughout the application. Internally, the gRPC libraries, which activate the appropriate service to handle calls for an endpoint, will use an instance from the dependency injection container if one is available.
The second line uses the
AddHostedService extension method to register our
We can now update our endpoint registration:
On line 5, we include a call to
MapGrpcService for the
HealthServiceImpl. This ensures that it will be serving for calls to the health check RPC methods.
Calling Health Checks From a Client
Now that our server is configured for gRPC health checks, we can test it by creating a basic console client. We’ll use a new .NET Core 3.0 console application for simplicity.
After creating the console application, we can reference the same Grpc.HealthCheck NuGet library that we added to our server. This includes a client implementation also.
<PackageReference Include="Grpc.HealthCheck" Version="2.24.0" />
We can now use its client to call the health check method.
This simple client implementation establishes a
HealthClient using a gRPC channel. The
HealthClient has been generated based upon the health.proto service definition. We can, therefore, call the
CheckAsync method, providing a
HealthCheckRequest, as seen on line 7 in the preceding code. We may supply a specific service name or pass an empty string to request the overall server status.
In this post, we’ve explored a couple of approaches to including health checks for an ASP.NET Core 3.0 application which acts as a gRPC server.
The first option leverages the existing Health Checks middleware for ASP.NET Core which responds to a GET request made to an HTTP endpoint.
The second option is slightly more complicated but supports the gRPC Health Checking Protocol. Most of this is available as part of the Grpc.HealthCheck NuGet package, although a little work is needed on our end to set the statuses and register things into dependency injection.
Both approaches may be valid, and they can also be combined within a single application if you desire.
You can find a working example of a gRPC with health checks included in my gRPC Demos GitHub Repository.
For more gRPC content, you can find all of my posts that are part of my gRPC and ASP.NET Core series.