In this post, I want to dive into how the AWS SDK for .NET attempts to load credentials for any service clients which you use in your applications. I’m going to focus specifically on .NET Core applications, where the SDK client(s) are resolved from the dependency injection (DI) container. However, the steps are relatively consistent, even if you are creating the client instances directly (not from the DI container) and even on .NET Framework.
From where and in what order does the AWS .NET SDK load credentials?
- From the Credentials property of AWSOptions, if an instance of AWSOptions is provided during registration.
- Shared Credentials File (Custom Location). When an instance of AWSOptions is provided, and both the profile and profile location are specified.
- SDK Store (Windows Only). When an instance of AWSOptions is provided and only the profile is set (profile location is null).
- Shared Credentials File (Default Location). When an instance of AWSOptions is provided and only the profile is set (profile location is null).
- AWS Web Identity Federation Credentials. When an OIDC token file exists and is set in the environment variables.
- SDK Store (if on Windows) encrypted using Windows Data Protection API.
- Shared Credentials File in the default location.
- Environment variables. When the Access Key ID and Secret Access Key environment variables are set.
- ECS Task Credentials or EC2 Instance Credentials. When using IAM roles with ECS tasks and ECS instances.
The above is correct as of version 3.3.101 of the AWS SDK for .NET. Merely knowing this order is usually enough when consuming the library in your applications. If you’re like me though, you may be curious to understand a little more about the credential loading internals.
How AWS Credentials Are Loaded
The AWSSDK.Extensions.NETCore.Setup NuGet package supports the integration of the AWS SDK for .NET with the .NET Core configuration and dependency injection frameworks. This package allows us to register the AWS service clients we need to use in our application so that they may be resolved from the dependency injection container. It’s also possible to create an instance of the SDK service clients directly, in which case, many of the same steps for credential loading are also applied.
Internally, the extensions library uses a ClientFactory to create new instances of the service clients when they are required. This type, combined with some core internal mechanisms, will configure the service client instance by following several steps.
When registering a service client with the DI container, we call the AddAWSService<T> extension method on the IServiceCollection. This method has a few overloads, one of which accepts an AWSOptions instance used to configure the service client.
In the preceding example, we set the Credentials on the AWSOptions using an instance of BasicAWSCredentials. This is really bad idea since we’re directly exposing our credentials in the source code and likely our source control history. Don’t use it in this way for real applications!
AddAWSService adds a ServiceDescriptor to the IServiceCollection as follows:
This method provides a factory registration, Func<IServiceProvider, object>, which gets called anytime a new instance of the service client is required. Note that by default, service clients are registered with the singleton lifetime, so only a single instance is created when it is first needed to fulfil a dependency. The Func registered here, creates a new ClientFactory which accepts an AWSOptions parameter. AWSOptions may be null or may be an instance provided in the registration as I showed above. The service client is then created by calling the CreateServiceClient method. Don’t worry about this detail too much for this post; we’ll focus on what comes next.
When creating the service client, the first step the ClientFactory completes is to load the AWS credentials, which it will provide to the service client. This takes place inside a CreateCredentials method.
If an AWSOptions instance were provided when registering the AWS service client, it would be not null at the point that this method is called. The Credentials property on the AWSOptions class may hold a reference to a manually created AWSCredentials instance, which will be used by the service client if it is available. This is, therefore, the first choice for credentials which may be applied to the service client.
2: Shared Credentials File (Custom profile name and location)
The next conditional occurs if the Profile property on the AWSOptions has a value. This is expected to be the name of a profile from which to load the credentials. AWS supports declaring multiple named profiles in some of the possible credential files.
We could, for example, register our service with AWSOptions specifying the use of a profile named custom.
In this scenario, an instance of the CredentialProfileStoreChain class is created within the CreateCredentials method on the ClientFactory. As a reminder, here’s the relevant code again.
The CredentialProfileStoreChain is created passing in the ProfilesLocation (which may be null) from the AWSOptions. The TryGetAWSCredentials method is called passing in the specified profile name. This, in turn, calls down to a method named TryGetProfile.
When the ProfilesLocation is not null, then this will used to try to access a shared credentials file at that location. The shared credentials file stores credentials in plain text and can be accessed by various AWS tools such as any of the AWS SDKs, the AWS CLI and AWS Tools for PowerShell. It includes credentials for one or more profiles.
The credentials file from the supplied profiles location will be loaded and searched for a profile matching the Profile property from the AWSOptions. It’s possible that a matching section for the profile will not be found in the shared credentials file.
3: SDK Store (.NET SDK Credentials File) – Windows Only (Custom profile name)
When the TryGetProfile method (above) is called on a CredentialProfileStoreChain that was created with a null profile location, its preference, when the platform supports it, is to attempt to load credentials from the .NET SDK credentials file (SDK Store). Credentials in the SDK Store are encrypted and reside in the current user’s home directory. This helps to limit the risk of accidental exposure of the credentials. This functionality depends on the Windows Crypt32.dll module being available. Credentials contained within the SDK Store can be used by the AWS SDK for .NET, AWS Tools for Windows PowerShell and the AWS Toolkit for Visual Studio.
If cryptography is available (on Windows), a new instance of NetSDKCredentialsFile is created. This supports loading credentials which have been stored under the current users AppData folder encrypted using Windows Data Protection API. A profile with a matching name (or default) will be located if it exists in the store and be returned. The SDK Store is located in the C:\Users\<username>\AppData\Local\AWSToolkit folder in the RegisteredAccounts.json file.
4: Shared Credentials File (Custom profile name and default location)
In cases, where the ProfilesLocation is null, and the platform does not support the SDK Store, then the shared credentials file in the default location will be searched for a matching profile. The default location for the credentials file is within a directory named “.aws” in the home directory of the current user. For example:
When present, the file from this default location will be loaded and parsed to see if it contains a matching profile name. If the profile is located, the SDK attempts to create the credentials instance from the loaded values.
If no profile name was supplied to the CreateCredentials method, then the process continues and uses a class named FallbackCredentialsFactory to attempt to load credentials from several fall back options.
FallbackCredentialsFactory is a static class, which includes a static ctor which calls a static Reset() method.
Here is some of the relevant code inside FallbackCredentialsFactory.cs
During reset, any cached credentials are cleared.
FallbackCredentialsFactory includes a delegate member “CredentialsGenerator” which defines a method which accepts no arguments and returns an instance of AWSCredentials. A list of these delegates is populated by the reset method.
In the NetStandard case (which we’ll focus on here) four delegates are added to the list in a specific order (we’ll come to that very shortly). After creating an instance of the FallbackCredentialsFactory, the ClientFactory.CreateCredentials code, calls its GetCredentials method. Its code is as follows.
This code loops over each registered CredentialsGenerator delegate and invokes it. The delegates will either return an instance of AWSCredentials or will throw an exception. If and when one of the generators successfully provides the AWSCredentials, the instance is cached (stored in the cachedCredentials field) and the foreach loop breaks, returning the credentials.
5: AWS Web Identity Federation Credentials
In AWS, it’s possible to allow login through an OpenID Connect (OIDC)-compatible identity provider. In such cases, you will be issued a token by the OIDC IdP which is expected to be stored in a file.
The first delegate, which is added, calls the AssumeRoleWithWebIdentityCredentials.FromEnvironmentVariables method. This expects to load values from the environment variables which define the user of an OIDC provider for temporary, token-based access by assuming a role.
When any of the required environment variables are missing, an exception will is thrown, most probably an ArgumentNullException because the “AWS_WEB_IDENTITY_TOKEN_FILE” variable will not contain a value. Should all valid values be in place, an instance of AssumeRoleWithWebIdentityCredentials will be properly constructed. This provides refreshing credentials which will be refreshed every 5 minutes.
The second delegate in the list will attempt to load a profile using the CredentialProfileStoreChain. The registered delegate calls into the FallbackCredentialsFactory.GetAWSCredentials method, passing in a CredentialProfileStoreChain. A shared, static instance of the CredentialProfileStoreChain is stored in a private field of the FallbackCredentialsFactory. You’ll recall that we saw the CredentialProfileStoreChain used earlier as well. In that case, it was only called if a custom profile name had been provided on the AWSOptions. At this stage, the profile name will either by the value of the “AWS_PROFILE” environment variable, if present, or will be “default”.
GetAWSCredentials will attempt to load credentials from various sources by providing the profile name. The code will then try to load a profile from the CredentialProfileStoreChain. On Windows, this will first search the SDK Store (as above) and after that, the Shared Credentials File. On Linux, only the Shared Credentials File will be searched. If a profile is found, the code will return the credentials for that profile. If a profile could not be found in the chain, an exception is thrown.
7: Environment Variables
The third delegate that is added attempts to create an instance of EnvironmentVariablesAWSCredentials which derives from the AWSCredentials class. The constructor for this type calls a FetchCredentials method which searches for configured environment variables.
The access key is expected to be stored in an environment variable “AWS_ACCESS_KEY_ID”. The secret access key is expected in the environment variable “AWS_SECRET_ACCESS_KEY” or the legacy “AWS_SECRET_KEY” environment variable. This code also looks for an “AWS_SESSION_TOKEN” environment variable which may be set if you are using temporary credentials. This can be the case if you use the AWS Security Token Service to provide short-lived credentials. AWS uses the session token to validate the temporary security credentials.
At a minimum, the access key ID and the secret key must be located. An instance of ImmutableCredentials is then created and returned.
8: ECS Task Credentials or EC2 Instance Credentials
The last generator attempts to load credentials from locations which may be available if you have deployed your service on AWS using either ECS (Elastic Container Service) or an EC2 instance. When running your services in production, a best practice is not to manually provide credentials but to rely on IAM roles which can be assigned to EC2 instances and ECS tasks. This allows AWS to manage the credentials for the instance or task by providing credentials that are granted access permissions from an IAM role. This gets into some deeper security territory about how these features work which I will gloss over here.
The code which loads the ECS/EC2 credentials is as follows:
In short, when your service is running as a container on ECS, and a task role is applied, the Amazon ECS agent populates an environment variable “AWS_CONTAINER_CREDENTIALS_RELATIVE_URI” for all containers that belong to the task with a relative URI. The code above checks to see if this environment variable is set with a relative URI and if so uses a URIBasedRefreshingCredentialHelper to load the credentials.
When running directly on an EC2 instance, the instance role will be used to fetch credentials from the ECS instance metadata. The DefaultInstanceProfileAWSCredentials is used to access a cached instance of the credentials which refreshes every two minutes based on the EC2 instance metadata.
IAM roles for EC2 instances and ECS tasks are the recommended way to supply credentials. In both of these cases, you should not need to load the credentials manually. Instead, allow the SDK to load them for you. This will occur automatically unless you have provided credentials using any of the methods which are checked first.
That’s way more information than you probably needed. If you’ve reached this far, well done indeed! While the internal implementation for credential loading is not something you need to know to this depth, I find this useful background knowledge to understand the sources which may supply credentials. If your service is failing to load credentials or is using credentials which do not grant the expected access, understanding how these resolve can be useful.
Here are the steps, one more time.