ASP.NET Core Gotchas – No. 2 Some settings from launchSettings.json are now loaded by dotnet run

Following on from my first gotcha that we hit yesterday, here is another one which caught us out at the same time.

TL:DR; When using dotnet run (.NET Core 2.0), expect it to set some properties (including the port and environment) from launchSettings.json if it’s included in the Properties folder!

The issue we faced was that when running a new API using ASP.NET Core 2.0 inside a container, it was starting on the wrong port; not port 80 as we’d expected. The scenario where this can show itself is unlikely to be a common one, so hopefully only a few people will run into this exact gotcha.

In our case, prototyping the new API needed some front end involvement so we added a quick and dirty dockerfile and docker-compose.yml file to enable it to be spun up inside a container. Normally our production containers are all built on images which are based on the ASP.NET Core runtime image. We then copy in our published dlls and run the application inside the container. However, in this particular case we’d cheated a little and were using the SDK based image to allow us to build and then run the source inside a container.

In our case our dockerfile looked like this (note: do not use this in production!)

FROM microsoft/aspnetcore-build:2.0

ENV DOTNET_SKIP_FIRST_TIME_EXPERIENCE true

WORKDIR /app

COPY . .

RUN dotnet restore --verbosity quiet && dotnet build -c Release --verbosity quiet

EXPOSE 80

WORKDIR /app/src/OurNewApi

ENTRYPOINT dotnet run

The details of what it does are not too important. If you’re interested your can read my Docker for .NET Developers series to learn more. What is important here is that we are copying the entire solution folder contents into the container and later using dotnet run to start it. Seems safe enough – right?

As stated earlier, we noticed that for some reason, instead of using the default ASP.NET Core environment variable (ASPNETCORE_URLS=”https://*:80) which defines the default port of 80 it was using a different port. We also noticed that the environment was showing as “Development” and not “Production”.

We then examined the console output and noticed the following information:

Using launch settings from C:\Projects\OurApi\OurApi\Properties\launchSettings.json…

We checked and indeed, the port being used was the one defined in launchSettings.json. This was a bit of a surprise since in the past that file has only been used by Visual Studio. We scratched our heads as we’d previously done something similar with an ASP.NET Core 1.0 project without hitting any issues. I started investigating and soon found a closed GitHub issue titled “Add support for launchSettings.json to dotnet run”. Reading through it, it seems that since 2.0, dotnet run will load up some settings from launchSettings.json if it finds one. This makes some sense from a developer tooling point of view as I guess there are cases where it could be useful. In our case, the fact we were blindly copying in our entire solution folder (including the launchSettings.json file) as well as starting our API using dotnet run, meant that we experienced this behaviour inside the container. It’s not something we normally face, but in this quick prototype, it showed itself.

The quick solution in our case was to include the Properties folder in our dockerignore file which specifies any folders/files that you do not want included in the build context. This then avoids them being copied into your image using the COPY command.

For the curious among you, this functionality is implemented inside ProjectLaunchSettingsProvider.cs in the .NET CLI repository. If the file is found then the properties from the environmentVariables section are used as well as the value of the applicationUrl property.

Long story short, be aware of the fact that since .NET 2.0 the “dotnet run” CLI command will look for and use a launchSettings.json file if it’s available. If you suspect this may be happening in your case you can check the console output to see if it has loaded from launchSettings.json.

ASP.NET Core Gotchas – No. 1 Using Environment Variables with ASP.NET Core 2.0 and Linux

We’ve been using ASP.NET Core 1.0 for some time as part of a microservices architecture we’ve developed. We run the services in production as Docker containers on Amazon ECS. We recently created a new API based on ASP.NET Core 2.0 and ran into some issues with configuration.

The first of the two issues we encountered is in cases where we use environment variables in our Docker containers that we expect to override ASP.NET Core configuration values. The ASP.NET Core configuration system allows many sources for configuration values to be defined. This includes loading from json files and environment variables. When loading configuration, each of the providers is checked in turn for configuration values. Any which define the same key as a previous configuration item are overridden with the new value. This works nicely as we can define common configuration in JSON files and optionally override this in production using environment variables.

This is exactly how we run our APIs currently. In ASP.NET Core 1.0 we could pass in environment variables to containers (in our case, using docker-compose files locally and AWS ECS TaskDefinitions in production). Configuration in ASP.NET Core supports a hierarchy of settings which allows us to define “sets” of values. For example, in our case we have a top level section called DistributedCacheConfig and within that there are three settings to control various things all related to caching.

When overriding these settings using environment variables we previously used the colon separator to define the layer of the hierarchy the value targets. One such environment variable would look like this…

DistributedCacheConfig:Enabled=true

When read and mapped to the ASP.NET Core configuration system this would override the enabled value for the DistributedCacheConfig even if a previous JSON file had set it to false. This even worked when deployed to production where we could configure AWS to start our containers with the necessary environment variables when launching new instances.

When setting up a new API using the latest ASP.NET Core 2.0 version we noticed an issue when deploying to AWS. The settings we had defined in the TaskDefinition (which controls the environment variables containers start with) were not being applied and the settings from the JSON files were still being used. We then tested this locally by starting up a container from the ASP.NET Core 2.0 docker image and again noted that the environment variables were not overriding the JSON values as expected.

I spent some time investigating this and one item I was able to find this GitHub issue for the Configuration repository which mentions the possibility to use a double underscore(__) as the separator between the layers on Linux. I made the change to our environment definition and immediately it was working again. I’d personally never been aware of the option to use this separator.

With the problem hopefully solved I set about investigating what had changed. Certainly the colon separator was working fine on our older projects. Finally I noticed that by default the ASP.NET Core 2.0 Docker image returned when asking for the “2.0” tag is based on Debian Stretch. With ASP.NET Core 1.x it was based on the older Debian Jessie version. I started to wonder if this might explain the change in behaviour. I quickly modified the dockerfile we were using to target the “2.0-jessie” tag, changing the environment variable back to the colon separated version as well. When I ran that as a container, the value was once again set using the environment variable as expected.

My guess (although I’ve not dug any deeper) is that between the two Debian versions, something has changed in how the colon separator is handled for environment variables. To validate this assumption I modified my application to spit out the environment variables at startup.

When running on Stretch – Environment.GetEnvironmentVariables() returns the following console output:

web_1 | key = HOME - value = /root
web_1 | key = TestSetting101 - value = Something
web_1 | key = ASPNETCORE_PKG_VERSION - value = 2.0.3
web_1 | key = NODE_VERSION - value = 6.11.3
web_1 | key = DOTNET_SDK_DOWNLOAD_SHA - value = 74A0741D4261D6769F29A5F1BA3E8FF44C79F17BBFED5E240C59C0AA104F92E93F5E76B1A262BDFAB3769F3366E33EA47603D9D725617A75CAD839274EBC5F2B
web_1 | key = NUGET_XMLDOC_MODE - value = skip
web_1 | key = PWD - value = /app/TestingConfiguration
web_1 | key = DOTNET_SKIP_FIRST_TIME_EXPERIENCE - value = true
web_1 | key = ASPNETCORE_URLS - value = http://+:80
web_1 | key = HOSTNAME - value = 44d9a86fba25
web_1 | key = DOTNET_SDK_DOWNLOAD_URL - value = https://dotnetcli.blob.core.windows.net/dotnet/Sdk/2.0.3/dotnet-sdk-2.0.3-linux-x64.tar.gz
web_1 | key = PATH - value = /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
web_1 | key = DOTNET_SDK_VERSION - value = 2.0.3

When running on Jessie – Environment.GetEnvironmentVariables() returns

web_1 | key = PWD - value = /app/TestingConfiguration
web_1 | key = TestSetting101 - value = Something
web_1 | key = DOTNET_SDK_VERSION - value = 2.0.3
web_1 | key = PATH - value = /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
web_1 | key = NUGET_XMLDOC_MODE - value = skip
web_1 | key = DOTNET_SDK_DOWNLOAD_SHA - value = 74A0741D4261D6769F29A5F1BA3E8FF44C79F17BBFED5E240C59C0AA104F92E93F5E76B1A262BDFAB3769F3366E33EA47603D9D725617A75CAD839274EBC5F2B
web_1 | key = MySettings:Setting1 - value = From_DockerCompose
web_1 | key = HOME - value = /root
web_1 | key = ASPNETCORE_URLS - value = http://+:80
web_1 | key = HOSTNAME - value = 0262ce3069ae
web_1 | key = ASPNETCORE_PKG_VERSION - value = 2.0.3
web_1 | key = DOTNET_SDK_DOWNLOAD_URL - value = https://dotnetcli.blob.core.windows.net/dotnet/Sdk/2.0.3/dotnet-sdk-2.0.3-linux-x64.tar.gz
web_1 | key = DOTNET_SKIP_FIRST_TIME_EXPERIENCE - value = true
web_1 | key = NODE_VERSION - value = 6.11.3

You can see here that on Stretch the variable with the key MySettings:Setting1 is not even returned. So this explains why it’s not available in the configuration.

While I’d like to know what actually changed that affected the behaviour between the two OS versions, I’ll have to leave that as a mystery. However, the advice here is that if you plan to run on Linux, it’s probably safest to use the double underscore separator when defining any environment variables which works on either OS. Perhaps we were simply lucky in 1.0 that it worked and we should have been using double underscore all along.