Playing with C#9 Top-level Programs, Records and Elasticsearch.NET

This post is for C# Advent Calendar 2020 organized by Matthew Groves. I recommend that you check out some of the other posts being published throughout December!

In this post, I thought it was long overdue that I spent a little time playing around with some new C# 9 language features. C# 9 introduced quite a few new concepts, as well as expanding on existing features such as pattern matching. There is a bit of debate in the community about the speed at which C# is evolving and moving towards functional programming paradigms. I personally lean toward the “keep it up” camp, enjoying much of what the team has introduced in recent years. However, I also reserve judgment on features until I have the opportunity to try them out and assess their usefulness in my own code.

I’ve decided to try out a few of the newest features in a small hello world application, combining them with a small sample that shows the use of the Elasticsearch.NET low-level client for controlling Elasticsearch. I recently joined Elastic as a senior engineer, so this was a good chance for me to explore the current API surface while at the same time playing with C# 9.

I’ll introduce code through this post. If you’d like to find the final source, I have provided it in a GitHub Gist.

Top-level Programs

One of the C# 9 features which I wanted to explore is top-level programs. The concept is quite simple and is intended to reduce the boilerplate code necessary for simple applications. Before C# 9, a simple console application requires developers to include a Program class, with a static void Main (or static async Task Main) method. This is the expected entry point that the runtime looks to invoke when the application executes.

Top-level programs remove this requirement, allowing applications to be written directly into a *.cs file without declaring a namespace, class or method. Initially, I wasn’t too sure about the usefulness of this feature. The templates create the Program.Main method for use automatically, so I didn’t see much benefit for top-level programs. However, a few people on Twitter had highlighted the lower barrier to entry that top-level programs provide for developers when they first start to learn C#. Rather than needing to understand why there is a class and what a method is, developers can instead focus on core coding concepts first. I do like this argument, and I can see some benefit when used in teaching.

A second use case presented itself when I started building some small feature samples around Elasticsearch.NET. My goal was to expand my experience with using the APIs in the Elasticsearch.NET low-level client library. In this case, I wanted these samples to be simple to follow. By removing the standard boilerplate, I think it does help to focus immediately on the code being demonstrated.

This benefit and the top-level program feature in general, can be more easily demonstrated with a walkthrough. To follow along, you’ll need to install the .NET 5 SDK and if you use an IDE, the latest version supporting C# 9.0.

Using the dotnet command line or an editor/IDE, create a new .NET Core solution with a single console application project. The first point to highlight here is that by using the templates for a console application, we receive a project which includes a Program class, with a Main method. To demonstrate top-level programs, we need to remove these. This feels a little in conflict with the benefit of top-level programs as choosing to use them requires a deletion step. Personally, I’d like a switch in the command line or a checkbox in the UI to opt-out of the default Program.cs file. We’ll remove the Program.cs file entirely, leaving us an empty project.

In the empty project, we’ll add a new, empty class file named MyApp.cs. In a few moments, we’ll be writing our top-level program in this file. This is valid, and the compiler will locate the code, generating the necessary Program.Main method for us behind the scenes. .NET executables must still have this expected entry point. The top-level program feature simply removes the need for us to include it explicitly in our code. You’ll notice, we can name our top-level program file whatever we like. The main rule to follow is to only include one such file in your project so that the compiler can locate it and generate the entry point code.

The final preparation step we need to complete is to add a package reference to the latest Elasticsearch.Net client from NuGet. At the time of writing, version 7.10.1 is the newest release, so we’ll use that here. You may add the reference via the IDE UI, via the command line, or by directly including the package reference in the HelloWorld.csproj file.

If you’re following along using the command line on Windows, you can run these six commands to create the initial project state all from the CLI.

dotnet new sln --name HelloWorld
dotnet new console --name HelloWorld --output src/HelloWorld
dotnet sln add src/HelloWorld
del src\HelloWorld\Program.cs
copy NUL src\HelloWorld\MyApp.cs
dotnet add src\HelloWorld package Elasticsearch.Net --version 7.10.1

Your project file should contain the following code and must target .NET 5.0 using the net5.0 target framework moniker.

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>net5.0</TargetFramework>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="Elasticsearch.Net" Version="7.10.1" />
  </ItemGroup>

</Project>

Writing Our Top-Level Program

At this point, we’re ready to write our top-level program. This can, of course, perform any work you wish and it can call into other types and methods in your codebase. For this sample, I’ll include everything inside the MyApp.cs file, and our sample will communicate with Elasticsearch, performing some basic operations.

IMPORTANT: To follow along, you require an instance of Elasticsearch running locally (or remote if you prefer). I’m running a single node using Docker, and I have exposed it via the host port 9200. If you’d like to set up an instance, here is a basic Docker Compose file that you can use to start your own local instance.

Inside the empty MyApp.cs, we’ll start writing our application code. Just as in standard class files, we can include using directives to other namespaces that contain types we wish to use. We’ll need a using statement for System, System.Threading.Tasks and Elasticsearch.Net.

using System;
using System.Threading.Tasks;
using Elasticsearch.Net;

Next, we’ll define an instance of the low-level client for Elasticsearch. The low-level client exposes APIs for working with Elasticsearch. Generally, you’ll use this in situations where you have existing requests that you wish to dispatch to Elasticsearch directly. It has no dependencies and leaves much of the work for formatting requests up to the developer. In many cases, developers will prefer the NEST client, a higher-level library that supports using strongly typed requests, often defined using its fluent DSL. I want to focus on the low-level client in this sample as I believe it’s useful to understand the base upon which NEST is built.

var client = new ElasticLowLevelClient();

The default, parameterless constructor for the client, configures the application to attempt to communicate with Elasticsearch over http://localhost:9200. You can, of course, configure this by passing a ConnectionConfiguration instance to the client, but we don’t need that for this sample.

Elasticsearch exposes REST endpoints which may be used by applications to configure and run Elasticsearch features. The low-level Elasticsearch.NET client provides an initial abstraction for working with requests and responses to these REST APIs.

As an example, we’ll begin by indexing a document into our Elasticsearch node. First, we’ll need some JSON representing the content of our document. We’ll use an escaped JSON string for this first document.

var json = "{\"Id\":1,\"Title\":\"Pro .NET Memory Management\",\"ISBN\":\"978-1-4842-4026-7\"}";

Next, we’ll use the client to index this by calling its IndexAsync method. This is a generic method where the generic argument specifies the type of response we require. A few options are available to us here. For example, we could ask for a BytesResponse, which gives us access to the returned bytes of the response from Elasticsearch. In high-performance situations, this may be used to manually parse the bytes and to extract any necessary information. For this sample, we’ll prefer a string response, where the body of the response is converted to a string.

For our index request, we’ll use the following line of code.

var indexResponse = await client.IndexAsync<StringResponse>("books", "1", PostData.String(json));

The arguments here state that we want to index our document into an index named “books” and we want to explicitly provide an ID of “1” for the document. The final argument accepts PostData, which will be used to form the body of the request to Elasticsearch. We can define string-based post data which includes our JSON string content.

This is an async method, so we must await it. Asynchronous code is supporting in top-level programs. After this method executes and returns, we’ll have an instance of StringResponse, representing the response from Elasticsearch, along with request/response metadata.

Following our request to index a document, we’ll test that we can query for it using a search request. Here is the code we’ll use…

var indexResponse = await client.IndexAsync<StringResponse>("books", "1", PostData.String(json));

if (indexResponse.Success)
{
    await client.Indices.RefreshAsync<VoidResponse>("books");
    
    // search for a book with a specific ISBN
    var searchResponse = await client.SearchAsync<DynamicResponse>("books", PostData.Serializable(new
    {
        query = new
        {
            match = new
            {
                ISBN = new
                {
                    query = "978-1-4842-4026-7"
                }
            }
        }
    }));

    if (searchResponse.Success)
    {
        // access the title by path notation from the dynamic response
        Console.WriteLine($"Title: {searchResponse.Get<string>("hits.hits.0._source.Title")}");
        Console.WriteLine();
    }
}

We should not assume that our request succeeded, so first, we’ll check the Success property to ensure that Elasticsearch was able to successfully accept our request to index a document. It may take a few moments for documents to appear in the index for searching since the index must refresh first. By default, this occurs every second for indexes that have received at least one search request in the last 30 seconds. For this sample, it’s possible that our index has not been queried already, so for demo purposes, we send a Refresh request to force a refresh of the “books” index. Since I’m not interested in the response, I’ve chosen to use the special VoidResponse type for this request.

It’s important to highlight that generally, it’s best to avoid manually refreshing an index. The default Elasticsearch behaviour should be sufficient, and additional refreshes are resource-intensive. Regularly refreshing via the REST API can impact cluster performance. We’re using it here since it’s a demo!

After the refresh, we perform a search via the Search API on the low-level client. This time, we specify that we want a DynamicResponse, which allows us to traverse the data for specific values. In this example, we attempt to access the first hit in the results and then access the title string from the _source document using dot notation. Our search is made to the “books” index and can include our search query in the body. We could have used a JSON string again here, but I want to demonstrate another way that can be used to form post data. This time, we’ll have an object serialised for the body of the request. We could provide a strongly typed instance of an object here, but another option for rarely used ad-hoc queries is to use an anonymous object instead. This anonymous object specifies properties that align to the JSON we need to send to perform a match query on the ISBN number of a book. We ask for any documents where the ISBN matches that of our newly indexed book.

After this returns, we again check for a success status and then write the response body string out to the console.

Using Local Functions in Top-level Programs

We cannot define regular methods as part of a top-level program since these must exist within a type such as a class. However, we can use a C# 7.0 feature called Local Functions. Local functions are private methods of a type that are nested inside another member. In the case of top-level programs, a local function can be defined at the bottom of our file and called from without the top-level program code.

Here is the definition for a local function we’ll add to the bottom of the MyApp.cs file.

async Task IndexAndRetrieveAsync()
{
    ...
}

We’ll come back to this code shortly and provide the implementation. Before we do, let’s introduce two other C# 9 features which we’ll use in our code.

Records

Records are one of the headline features of C# 9 and have taken a long time to reach a stable state for inclusion in the language. I’ll summarise them here, but you may want to check out the “C# 9.0 on the record” blog post for more thorough information.

We can use a new record keyword to define a record type. Records are still classes (reference types), but the compiler can include default behaviour for us, removing boilerplate code and providing functionality more akin to value types. A fundamental behaviour is value-based equality, where two instances of a record type are considered equal as long as their values are the same. This differs from the standard reference equality of a traditional class, where two different instances are not equal.

A good use case for records is to use them to define DTOs (Data Transfer Objects) used to pass data between different layers of the application, to transfer data into a database, or for serialisation of objects to send them across the wire. The record syntax supports shortcuts to define simple DTO objects without the need to manually auto initialised properties.

In the sample, we’ll define a DTO to represent a Book type which we’ll be using to index book data into Elasticsearch. At the bottom of our MyApp.cs file, we can include a record declaration.

internal record Book(int Id, string Title, string ISBN);

In this single line, we define an internal record type named Book. This is an example of a positional record where its contents may be provided only via constructor arguments. For small types, this is pretty convenient. We’ve used the short syntax to define a record with three properties, each of which must be initialised via the constructor. Our Book is therefore an immutable type.

Target-typed New Expressions

We’re now able to provide the implementation for our IndexAndRetrieveAsync method. We start by creating a book instance, making use of our Book record type and another C# 9 feature, target-typed new expressions.

async Task IndexAndRetrieveAsync()
{
    Book bookToIndex = new(2, "Pro .NET Benchmarking", "978-1-4842-4940-6");

    ...
}

With target-typed new expressions, we can omit the type of a new object in situations where the type can be inferred from the preceding code. In this example, we have declared a type of the bookToIndex variable already, so rather than repeat the type before the constructor arguments, we simply omit it.

This feature is likely to open up a whole new battleground in the var vs. explicit type arguments which exist today. Some developers would prefer to use “var bookToIndex =” here, in which case, we would have to specify the type after the new keyword. Personally, in normal code, I’d probably stick to using the var keyword here. In cases where the type for a variable is defined earlier though, the simplified new syntax is something I might consider.

As a short aside, places where I think this feature will be most useful, is when defining a list of objects. For example, we could define an initial list of books as follows.

var books = new List<Book>
{
    new (1, "Title 1", "ISBN-1"),
    new (2, "Title 2", "ISBN-2"),
    new (3, "Title 3", "ISBN-3")
};

Here, the type of each object in the list can be clearly inferred and we save a few keystrokes and repetition by avoiding the type for each entry.

We can now complete the local function as follows…

async Task IndexAndRetrieveAsync()
{
    Book bookToIndex = new(2, "Pro .NET Benchmarking", "978-1-4842-4940-6");

    var indexResponse = await client.IndexAsync<StringResponse>("books", bookToIndex.Id.ToString(), PostData.Serializable(bookToIndex));

    if (indexResponse.Success)
    {
        // get the book back by its ID
        var searchResponse = await client.GetAsync<StringResponse>("books", bookToIndex.Id.ToString());

        if (searchResponse.Success)
        {
            Console.WriteLine($"Response body: {searchResponse.Body}");
        }
    }
}

This code is similar to our earlier code since it indexes a document and then accessed it. For variety, it uses slightly different techniques for each operation. After creating a book instance, we index it into Elasticsearch, this time, using the PostData.Serializable factory method. Our book record will be serialised to JSON in order to index it into Elasticsearch.

Assuming the response indicates indexing succeeded, we’ll attempt to get our document by its ID. This is different from our previous search since we explicitly know which single document we want back. This time, we’ll return a StringResponse and print the body to the console.

To complete the sample, we can now call this method from our top-level program. To clean up afterward, we’ll also delete the “books” index.

await IndexAndRetrieveAsync();

await client.Indices.DeleteAsync<VoidResponse>("books");

Summary

In this post, we explored the use of some of the new C# 9 language features in a console application. We interacted with Elasticsearch using the low-level Elasticsearch.NET library within a top-level program. We also touched on using a Record to define our object used for serialisation. Utilising a record here meant we only required a single line of code to define our type. We also used target-typed new expressions to further simplify the code in our application.

I can certainly see myself using top-level programs for simple console applications, particularly sample code since it allows consumers to focus on the essential code blocks and less on infrastructure. It’s a shame that the default templates don’t provide a simple way to produce a top-level program without having to delete code ourselves. Perhaps that will come in future versions.

If you’re following along and feel that there’s a lot of code needed here to index and retrieve documents with Elasticsearch.NET, you’d be correct. This is a base, no-dependency, low-level library which provides a simple abstraction over working with Elasticsearch. For more complex applications, you’ll likely want to use NEST, a high-level library that sits on top of the low-level client. This supports strongly typed requests and responses along with a syntax for defining more complex requests. The low-level client is still accessible when you want to use it.

As a reminder, if you’d like to view the final source, I have provided it in this GitHub Gist.

Thanks for reading this post, part of the C# Advent Calendar. Happy holidays!