Protecting Developer Secrets
- February 9, 2021
Team Solvo
Congratulations! You just joined an exciting new project, and you’re getting started setting up your environment. You’ve got your IDE set up just right, cloned the source code repository, and now you’re downloading all the dependencies. “Let’s go!” you think excitedly – and then realize you still need to provision access to the application server and the various cloud services. No big deal though, you’ll just grab a few API keys for your AWS account, and stick those in your code everywhere.
Everywhere, hmm… Let’s take a step back and consider the ramifications before you commit to a questionable course of action – always a good idea. Clearly, hardcoding secrets such as passwords and keys is a terrible idea, and sticking it in a plain text configuration file is not much better – especially if you need to commit that to the source code repository, where all the rest of the team have access to that as well. Not to mention that they should be using their own credentials instead of yours anyway, and if it’s an open source project then obviously the entire world now has your secrets.
Sure, using an API key feels different than a password, but these are effectively the same from an exposure point of view, and should be protected in a similar way. API keys can support running application-level scripts on a system-level account, or many kinds of automation such as Terraform. Likewise, when you run and debug your application locally, you’re probably using an API key so your local copy of the application can authenticate to AWS and other cloud services. And of course, any time you want to use the CLI you will need to provide an access key to authenticate to the service. But you do need to properly encrypt these keys when you store them.
Friction with API Keys
However, simply storing API keys is no more secure than leaving account passwords in plain text – and yet this is most often how many organizations do it. These API keys are typically stored in non-encrypted plain text, either hardcoded in the application’s source code or in a shared configuration file. JSON or YAML doesn’t provide much protection, especially if they are checked into the codebase and pushed to a public GitHub repo! Moreover, these keys will usually never expire and remain valid indefinitely. This means that once they are stolen from the plaintext configuration file, the attacker has a large window of opportunity to exploit them, until someone manually revokes them. And just to make matters worse – AWS access keys will have a common signature pattern, starting with `AKIA…` . This makes them very easy to identify, so there are plenty of tools to search public repos and find these API keys.
So, what can we do? Using API keys is definitely a better solution than directly typing in user passwords, but we still need to properly protect them. At least when you’re using the AWS CLI, you can configure your CLI profile with `aws configure` to store your access keys internally in its own `credentials` file, so that you do not need to worry about where and how to store these. However, the AWS CLI profile actually does NOT protect these credentials, and it stores them in its configuration file in plain text. While that’s not quite as bad as committing plaintext keys into the repo, it’s still not much better! At that point, you’re just hoping no one will get access to your developers’ accounts – including via malware – or steal their laptops (if you don’t have full disk encryption)… This doesn’t seem like a promising path either.
A Better Solution
Well, as it turns out, there are in fact some better solutions. What if we could automatically and transparently create constrained, limited lifetime, granularly permissioned, device authorized, access tokens only as we need them, with just the right amount of privileges, and get rid of them right away as soon as we’re done? Fortunately, 99designs, a digital design marketplace platform, actually created an opensource tool called AWS-Vault to do precisely this. Of course, there are actually several other tools like this out there, but we really liked the implementation of this one from 99designs (though we have no direct relationship with them), and we decided to use this one in our own development teams here at Solvo to great effect. Let’s see how!
The first thing to know about is AWS’s Security Token Service (STS), which lets you generate short-lived, limited-privilege secure access tokens to use for an account, instead of a password or generic API key. Don’t worry, you don’t really need to know too much more about this STS or its specific parameters, since AWS-Vault will call GetSessionToken or AssumeRole STS APIs for you behind the scenes. But at a high level, you should understand that AWS-Vault gets this token from STS and uses it to create a constrained “session” of sorts, which authenticates the rest of the current workflow and then expires very shortly after.
The other thing worth realizing is that any modern operating system will have facilities for protecting user secrets, such as OSX’s Keychain, Linux’s Kwallet, or Windows’ DPAPI and CredManager. AWS-Vault will leverage this to strongly protect the root credential, and then automatically provision session tokens with STS into the current process’s environment variables. In this way, your actual credentials are protected with strong encryption and key management instead of dropping them in a plaintext profile file, while at the same time other apps and scripts will transparently get an authenticated session for AWS on demand. And even if some malicious software or attacker is able to surreptitiously misuse that authenticated session, they will still be limited to that session only – which you’ve hopefully configured to have both limited privileges and a very short lifetime.
How To Use AWS-Vault
Let’s start by setting up AWS-Vault. You can download the latest binary for your operating system here https://github.com/99designs/aws-vault/releases/latest, or install it directly with your package manager, e.g. `brew install aws-vault` for Homebrew (or even `brew install –cask aws-vault` on MacOS), `port install aws-vault` with MacPorts, or `choco install aws-vault` if you’re using Chocolatey on Windows (and a high-five to you if you are 😉 ). You’ll find a full list of package installers at https://github.com/99designs/aws-vault/blob/master/README.md#installing .
There is not a whole lot of configuration you have to do to get started aside from adding profile keys, since AWS-Vault reuses the AWS settings from ~/.aws/config. However, there are a few environment variables you can tweak. In particular, you can set the AWS_VAULT_BACKEND environment variable (or –backend command line flag) to control how the credential secrets are stored, for example `Keychain`.
Next, you should create at least one profile by calling `aws-vault add <profile_name> `:
You’ll need to provide the access key id, as well as the secret key – and these will be encrypted and stored securely in your Keychain. Now anytime you want to use this account, you’ll create an authenticated session with `aws-vault exec <profile_name>`, and you can run any command with those credentials. For example:
Of course you’d need to provide your Keychain password so it can decrypt the account secrets. You can also have multiple profiles with different settings and privileges, including the duration of the session lifetime. And by the way, you can share settings between profiles with the include_profile setting, to “inherit” settings from another profile. You can view a list of all configured profiles and active sessions with `aws-vault list`.
There is one more step that you should definitely perform to protect the IAM roles you are using – and you should definitely be using roles to delegate permissions, instead of granting resource access directly to user accounts. You should configure each of these roles to enforce MFA – multi factor authentication. This will require a user to additionally authenticate with a secondary, one-time key generated from an MFA device (such as an Authenticator app on a smartphone, a U2F hardware token, or a Yubikey), without which they will not be allowed to assume the role.
For example, this powerful IAM policy will require any AWS user to provide their configured MFA code before assuming the role this policy is attached to:
Finally, we need to update the ~/.aws/config file to specify an identifier for the account’s MFA device (you can find this in the AWS web console) to your profile, with the mfa_serial setting:
This is incredibly powerful, as this prevents even an attacker that somehow did succeed in stealing the credential secret from using it or impersonating the user, since they do not have access to the user’s physical device.
Summary
Using API keys to authenticate your programs to AWS provides a lot of control and flexibility, however we do still need to treat these keys as credential secrets – no different from passwords – and protect them accordingly. AWS-Vault can apply strong encryption and automatically generate limited use session tokens – and you can configure these to have reduced privileges and a very short window of use. These get pushed into the AWS CLI environment when they are needed, and will expire shortly after – so even if they get misused or stolen, there is a very limited window of opportunity for an attacker to exploit them, and even then they should have restricted access according to the principle of least privilege.
Once you’ve set it up right, AWS-Vault can transparently provide these limited sessions, and adding MFA to these accounts makes them exceedingly hard for malware or an attacker to abuse. This is a very flexible and secure approach to protecting developer secrets, and especially AWS API keys.
Action items:
- Install and configure AWS-Vault
- Set up AWS IAM roles
- Configure roles to require MFA
- Add a profile to AWS-Vault with `aws-vault add`
- Update the AWS CLI configuration file with mfa_serial and the account’s MFA device serial for the profile
- Create an authenticated session and run commands with `aws-vault exec`
- Be safe!
- Stay home, wear a mask!
