In this post, I want to cover something which I find extremely interesting. How are .NET APIs Designed? Sounds exciting… right?
Perhaps you’re thinking no, it’s not and you are planning to hit that back button. Hold fire for one second to think about the challenge from the perspective of the .NET team.
You’re designing a set of libraries, used by millions of developers every day, running crucial applications around the world. You want to improve it and add a new feature or enhanced functionality. You must do this, all the while not breaking the millions of existing applications.
Does that sound fun? For me, the answer is kind of yes and no. From a curiosity point of view, I love writing C# code and figuring out how to makes things work. That’s the yes side of my answer. The no part of my answer comes from my experience of building small libraries in the past. Most of these are internal libraries and my consumers are less than thirty people. Even so, I’ve made mistakes and included a breaking change that I didn’t realise was breaking or produced a public API which was not fit for all of the use cases. In my experience, designing public APIs is difficult.
Since the team started on .NET Core, things are a little easier. The version of .NET is no longer tied to the operating system, which means that developers can choose when to upgrade. In theory, this makes introducing breaking changes a little more practical between major versions.
For the remainder of this post, I will explain the .NET API design process as I understand it. I don’t work for Microsoft, so these are my own interpretations based on watching this process take place for a few years. Much of what the team does, is released publicly, and it’s therefore possible to learn a lot from how they organise API design for .NET Core (and .NET 5).
To make the explanation more concrete, I’ll follow the design of a recent new library which will be included as part of the .NET BCL (base class library) from .NET 5. Its available today as a NuGet package, and I discuss its functionality in my post – Sending and Receiving JSON using HttpClient with System.Net.Http.Json. For now, it’s not too important to understand the details of what this API does. We’ll focus on how it was brought into existence.
The Design Phase
The original design detail for the System.Net.Http.Json library was added to the .NET Designs repository on GitHub on the 8th of February by Immo Landwerth. This design includes details of the complexity which developers encounter today, which the design sets out to improve.
It sets out in clear terms the scenario the design aims to make better and what the user (developer) experience should be for the feature. This includes sample code illustrating how the API would be consumed by a developer for several potential situations.
With the scenarios clear, it then goes on to introduce the requirements for the new API. What must it achieve and under what timescales? This section also makes it clear any non-goals, things that would potentially allow the design scope to slip if not expressly defined.
Then comes the design itself. The design includes proposed public APIs without any implementation details. This consists of all public methods and types which the design introduces.
.NET Design Review Phase
The next phase in the .NET process is for an API design review to take place. A GitHub issue was opened in the .NET Runtime repository on GitHub which allows this process to take place publicly. Sometimes an API review will happen before any code is written. This is intended to agree on the shape of the public API surface that will later be implemented. In this case, from reading the comments, I understand that an initial proof of concept implementation was developed first. This allowed issues to be worked through, which would shape the API being proposed. Because the proposal and discussion are all public on GitHub, this makes it possible for the community to step in with feedback and suggestions. I really like how open .NET is these days!
The API gets reviewed in a process called the .NET Design Review. This is a meeting where core experts of the .NET team come together to assess proposals and to make sure that the public API is fit for purpose. This is a crucial step since changing a public API after a release would constitute a breaking change. For backwards compatibility, this is avoided as far as is reasonably practical. It means that API decisions need to be thorough; otherwise, mistakes or oversights in the design could live for an extremely long time. Goals of the review also include consistency of the API surface. The team want the API to be easy and obvious to consume. Ensuring that it follows standard naming conventions, method shapes and method parameter patterns.
You can view a recording of the design review for the HttpClient JSON extension methods on YouTube.
During the API Review, someone will represent the proposal and explain the goals and reason for the proposed design. The team will then discuss it and determine if the proposal needs further work before approving it. An API may be presented at multiple design reviews before being considered acceptable.
Something I really admire about the team is that they hold this meeting live on YouTube so that anyone can watch. This is mostly a view-only approach, although sometimes comments and feedback left in the chat during the meeting may be considered as part of the discussion. All of the past recorded meetings are available on YouTube as a playlist under the .NET Foundation channel.
I often watch these when the API(s) being discussed are of interest to me. I find it extremely interesting to hear the discussion and watch how the .NET team think about designing a framework. There are many nuanced considerations which have to be made along the way. The sheer amount of .NET knowledge in the room is quite staggering. Minor behaviours of subtle implementation details are often brought up, as are historical aspects of existing APIs and their behaviour. Watching a one to two, hour-long meeting as a hobby may not be everyone’s cup of tea, but I highly recommend catching a few of these to really appreciate the design of the .NET framework.
Standard practice is for the GitHub issue to be used during the review. Immo Landwerth, the program manager for .NET, usually chairs the meeting and takes notes during the discussion. Any concerns, feedback and changes are recorded as the output from the design review. Due to an admin error, the main issue (previously linked above) was not used for the design review feedback. A duplicate was accidentally opened. This includes the summary of the discussion around the API that were agreed during the design review.
Once approved, a developer will start work to implement the approved API. As with this example, some work may already have been done experimentally and will require changes to incorporate the feedback from the design review.
Most of the work for this feature was completed by David Cantu and can be seen in his pull request (PR) “Add new System.Net.Http.Json project/namespace“ on GitHub. Again, the work is completed in the open, and anyone can subscribe to notifications and even leave comments.
This phase will hopefully be reasonably familiar to many of you who use pull requests for your work today. The idea is quite simple. A developer completes some work on a git branch. Once that work is complete and ready for consideration to be merged, they open a pull request for it. At this point, the work could be merged as-is, but for quality purposes, it’s common for one or more code reviews to take place by other developers. In the Microsoft .NET world, this is a comprehensive process, since bugs, inconsistencies and performance issues can be a huge problem to address later on.
In this example, the review was extensive and involved multiple experienced reviewers. If you take a few minutes to scroll down the PR, you will see detailed feedback about the intricacies of the code. This is another point where I learn a lot from seeing small items being raised and discussed. Watching the PR over time, you can even view the newer commits, which address the feedback and resolve any problems. You’ll also appreciate again the knowledge that the experts on the team retain, regarding the wider base class library.
Merge and Release
Once the PR has been approved by all required reviewers, it can continue its journey and be merged into the master branch. The .NET runtime is a complicated repository with advanced build processes triggering for newly merged code. That’s beyond the scope of this post and frankly my knowledge!
Eventually the new code will appear in nightly builds of the relevant library and may be pushed to a MyGet or NuGet feed for preview use and testing. For this example, a new package was produced and released as a prerelease preview on NuGet.
I hope this post was interesting to you. Personally, I find this stuff very interesting and a great learning experience. By understanding how the .NET team take an idea from proposal, through to final code, I can apply some of the techniques in my work also. I’ll never be as smart or as experienced as the developers on the .NET team, but I come away from each viewing of a design review, with a little more knowledge. I appreciate this is geek level 100 stuff and not for everyone, but much like foods from around the world, you should try it once before deciding it’s not for you!