We’ve been using ASP.NET Core 1.0 for some time as part of a microservices architecture we’ve developed. We run the services in production as Docker containers on Amazon ECS. We recently created a new API based on ASP.NET Core 2.0 and ran into some issues with configuration.
The first of the two issues we encountered is in cases where we use environment variables in our Docker containers that we expect to override ASP.NET Core configuration values. The ASP.NET Core configuration system allows many sources for configuration values to be defined. This includes loading from json files and environment variables. When loading configuration, each of the providers is checked in turn for configuration values. Any which define the same key as a previous configuration item are overridden with the new value. This works nicely as we can define common configuration in JSON files and optionally override this in production using environment variables.
This is exactly how we run our APIs currently. In ASP.NET Core 1.0 we could pass in environment variables to containers (in our case, using docker-compose files locally and AWS ECS TaskDefinitions in production). Configuration in ASP.NET Core supports a hierarchy of settings which allows us to define “sets” of values. For example, in our case we have a top level section called DistributedCacheConfig and within that there are three settings to control various things all related to caching.
When overriding these settings using environment variables we previously used the colon separator to define the layer of the hierarchy the value targets. One such environment variable would look like this…
DistributedCacheConfig:Enabled=true
When read and mapped to the ASP.NET Core configuration system this would override the enabled value for the DistributedCacheConfig even if a previous JSON file had set it to false. This even worked when deployed to production where we could configure AWS to start our containers with the necessary environment variables when launching new instances.
When setting up a new API using the latest ASP.NET Core 2.0 version we noticed an issue when deploying to AWS. The settings we had defined in the TaskDefinition (which controls the environment variables containers start with) were not being applied and the settings from the JSON files were still being used. We then tested this locally by starting up a container from the ASP.NET Core 2.0 docker image and again noted that the environment variables were not overriding the JSON values as expected.
I spent some time investigating this and one item I was able to find this GitHub issue for the Configuration repository which mentions the possibility to use a double underscore(__) as the separator between the layers on Linux. I made the change to our environment definition and immediately it was working again. I’d personally never been aware of the option to use this separator.
With the problem hopefully solved I set about investigating what had changed. Certainly the colon separator was working fine on our older projects. Finally I noticed that by default the ASP.NET Core 2.0 Docker image returned when asking for the “2.0” tag is based on Debian Stretch. With ASP.NET Core 1.x it was based on the older Debian Jessie version. I started to wonder if this might explain the change in behaviour. I quickly modified the dockerfile we were using to target the “2.0-jessie” tag, changing the environment variable back to the colon separated version as well. When I ran that as a container, the value was once again set using the environment variable as expected.
My guess (although I’ve not dug any deeper) is that between the two Debian versions, something has changed in how the colon separator is handled for environment variables. To validate this assumption I modified my application to spit out the environment variables at startup.
When running on Stretch – Environment.GetEnvironmentVariables() returns the following console output:
web_1 | key = HOME - value = /root
web_1 | key = TestSetting101 - value = Something
web_1 | key = ASPNETCORE_PKG_VERSION - value = 2.0.3
web_1 | key = NODE_VERSION - value = 6.11.3
web_1 | key = DOTNET_SDK_DOWNLOAD_SHA - value = 74A0741D4261D6769F29A5F1BA3E8FF44C79F17BBFED5E240C59C0AA104F92E93F5E76B1A262BDFAB3769F3366E33EA47603D9D725617A75CAD839274EBC5F2B
web_1 | key = NUGET_XMLDOC_MODE - value = skip
web_1 | key = PWD - value = /app/TestingConfiguration
web_1 | key = DOTNET_SKIP_FIRST_TIME_EXPERIENCE - value = true
web_1 | key = ASPNETCORE_URLS - value = http://+:80
web_1 | key = HOSTNAME - value = 44d9a86fba25
web_1 | key = DOTNET_SDK_DOWNLOAD_URL - value = https://dotnetcli.blob.core.windows.net/dotnet/Sdk/2.0.3/dotnet-sdk-2.0.3-linux-x64.tar.gz
web_1 | key = PATH - value = /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
web_1 | key = DOTNET_SDK_VERSION - value = 2.0.3
When running on Jessie – Environment.GetEnvironmentVariables() returns
web_1 | key = PWD - value = /app/TestingConfiguration
web_1 | key = TestSetting101 - value = Something
web_1 | key = DOTNET_SDK_VERSION - value = 2.0.3
web_1 | key = PATH - value = /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
web_1 | key = NUGET_XMLDOC_MODE - value = skip
web_1 | key = DOTNET_SDK_DOWNLOAD_SHA - value = 74A0741D4261D6769F29A5F1BA3E8FF44C79F17BBFED5E240C59C0AA104F92E93F5E76B1A262BDFAB3769F3366E33EA47603D9D725617A75CAD839274EBC5F2B
web_1 | key = MySettings:Setting1 - value = From_DockerCompose
web_1 | key = HOME - value = /root
web_1 | key = ASPNETCORE_URLS - value = http://+:80
web_1 | key = HOSTNAME - value = 0262ce3069ae
web_1 | key = ASPNETCORE_PKG_VERSION - value = 2.0.3
web_1 | key = DOTNET_SDK_DOWNLOAD_URL - value = https://dotnetcli.blob.core.windows.net/dotnet/Sdk/2.0.3/dotnet-sdk-2.0.3-linux-x64.tar.gz
web_1 | key = DOTNET_SKIP_FIRST_TIME_EXPERIENCE - value = true
web_1 | key = NODE_VERSION - value = 6.11.3
You can see here that on Stretch the variable with the key MySettings:Setting1 is not even returned. So this explains why it’s not available in the configuration.
While I’d like to know what actually changed that affected the behaviour between the two OS versions, I’ll have to leave that as a mystery. However, the advice here is that if you plan to run on Linux, it’s probably safest to use the double underscore separator when defining any environment variables which works on either OS. Perhaps we were simply lucky in 1.0 that it worked and we should have been using double underscore all along.
Other posts in the ASP.NET Core Gotchas Series:
No. 2 – Why are settings from launchSettings.json now picked up when using dotnet run?
Have you enjoyed this post and found it useful? If so, please consider supporting me:
Ya, it’s pretty vague. Note https://docs.microsoft.com/en-us/aspnet/core/fundamentals/configuration/index?tabs=basicconfiguration#simple-configuration and https://stackoverflow.com/a/40094999
Thanks Scott! It’s one of those things that once you know it, it’s fine. Until then, it’s not too obvious. I opened a docs issue to discuss this – https://github.com/aspnet/Docs/issues/4927