Applications often require access to small pieces of sensitive data at build or run time. These pieces of data are often referred to as secrets. Secrets are similar in concept to configuration files, but are generally more sensitive, as they may grant access to additional data, such as user data.
This topic describes some of the main concepts of secret management. It also provides guidance on how you can use Cloud Key Management Service for secret management.
Overview of secret management
Several options exist for managing secrets. Some common ways of storing secrets include using:
- Code or binaries
- A deployment manager
- A secret volume in a container
- Metadata of a VM
- A storage system
Choosing from among these options generally requires striking a balance between security and functionality. Typical security concerns include:
- Authorization: Access management for secrets or where they are stored, including tight authorization scopes.
- Verification of usage: The ability to audit at a low level of granularity (for example, on a per-secret basis), of access to and use of secrets.
- Encryption at rest: The encryption of secrets in case of data theft or loss.
- Rotation: The ability to rotate or refresh secrets, either regularly or as needed in reaction to an incident.
- Isolation: Separation between where secrets are managed versus where they are used. Isolation also applies to separation of duties between users who have the ability to manage secrets versus use secrets.
Typical functionality concerns include:
- Consistency: Synchronization of secrets across multiple locations and in multiple applications.
- Version management: An understanding of when and how keys are updated, to support secret rotation.
Choosing a secret management solution
Choosing the best secret management solution depends on your unique existing environment, secrets, and security needs. Some common approaches include:
Storing secrets in code, encrypted with a key from Cloud KMS. This solution is typically implemented by encrypting secrets at the application layer. Using this solution helps provide an additional layer of protection against insider threats by limiting the scope of access to the secret. Access to the secret is restricted from all developers with access to the code to only those who have both access to the code and the corresponding key. Even in cases where all developers have access to both the code and the key, implementing this option has benefits as it provides the ability to audit access to the secret, which might not be possible in a code repository.
Storing secrets in a storage bucket in Cloud Storage, encrypted at rest. This solution has similar benefits as the previous solution, in that it limits access to secrets to a smaller set of developers and provides the ability to audit that access. In addition, by storing secrets in a separate location, you can more easily rotate secrets when necessary; for example, if a security breach is detected. Also, this solution allows for separation of systems; if the code repository using the secrets is breached, the secrets themselves might still be protected.
Using a third-party secret management solution. A dedicated secret management tool builds on the first two options in this list. In addition, these tools might allow you to not only rotate secrets more easily, but, in some cases, either perform that rotation on your behalf, or simplify regular rotation.
Another important consideration when considering a secret management solution is how easy it is to change secrets. For example, hardcoding a secret is often a tempting solution, but this solution makes changing secrets at a later time-consuming and difficult.
When looking at a solution for secret management, consider the following design requirements and how relevant they are to your application:
- Rotating secrets. You may want to rotate secrets regularly, especially for security. Ideally, you can store multiple versions of each secret, and have your code try them one at time. By storing many versions of a secret, and rotating to newer secrets as needed, you can better maintain consistency with an external system that may need that secret. This also allows you to roll back to earlier secrets when needed. This solution can be quite complicated to implement, but considering these needs in advance can make it easier to manage secrets over time.
- Cache secrets locally. Depending on where you store your secrets with respect to your application, you might need to cache secrets locally. These secrets can then be refreshed frequently, such as several times per hour. The advantage to this solution is that, the more frequently you refresh, the faster you can respond to an outage. However, the disadvantage is that, if the secret becomes misconfigured somehow, a faster refresh allows that error to spread faster throughout your fleet.
- Using a separate solution or platform. When it comes to secret management, you might want to avoid lock-in by taking your secrets to a platform-agnostic secret management solution. This way, you have options should a more flexible solution becomes available.
Secret management using the Google Cloud Platform
The GCP offers several ways to help you address how you want to implement your secret management solution. This section describes an approach that uses Cloud Storage for secret storage, Cloud KMS for encryption keys, Cloud Identity and Access Management for access control and Cloud Audit Logs for auditing.
Here's one way to implement using secret management on the GCP. For more information, see How to store secrets encrypted with Cloud KMS.
- Create two projects. The first project uses Cloud Storage to store secrets. The second project uses Cloud KMS to manage encryption keys.
cloudkms.cryptoKeyEncrypterDecrypterroles to any user that needs to access secrets. Alternatively, you can use a service account that accesses Cloud Storage on a user's behalf. Ensure that users that do not need access to secrets have management, but not access, permissions.
- In Cloud Storage, store each secret as an encrypted object, and group those secrets into buckets as needed. In general, you could group secrets if they share the same usage, access, and protection needs.
- Protect each bucket, by using a unique key in Cloud KMS at the application layer. Another option is to rely on Google's default encryption.
- Rotate secrets regularly whenever possible.
- Monitor activity using Cloud Audit Logs. By default, administrative activity logs, such as key rotation or Cloud IAM permission changes, are recorded by default. An additional option to consider is to enable logging for data access logs on Cloud Storage objects. These data access logs are helpful when monitoring secrets that are particularly critical.
This solution addresses the majority of the secret management requirements described in Choosing a secret management solution. One item that is not addressed is version management. That's because addressing version management varies from application to application.
On the GCP, you have two options for encrypting secrets:
Use application layer encryption using a key in Cloud KMS. With this option, you implement encryption on objects or buckets in Cloud Storage on top of existing Google encryption, using a key stored in Cloud KMS. This is the recommended option.
Use the default encryption built into the Cloud Storage bucket. GCP encrypts customer content stored at rest, using one or more encryption mechanisms. As the name implies, this encryption is available by default and requires no additional action on your part.
To learn about these and other encryption options, see Encryption at Rest.
Application layer encryption using a key in Cloud KMS
The recommended way to store secrets is to use application layer encryption using a key in Cloud KMS. This method is particularly useful if you are looking for an additional layer of control, or have a compliance requirement to manage your own keys.
To implement this type of encryption, you send the secrets to be encrypted to
Cloud KMS using an
Encrypt request. Cloud KMS then
returns the encrypted secrets, which you can then write to storage.
Cloud KMS can handle secrets up to 64 KiB in size. If you need to encrypt larger secrets, it is recommended that you use a key hierarchy, with a locally-generated data encryption key (DEK) to encrypt the secret, and a key encryption key (KEK) in Cloud KMS to encrypt the DEK. To learn more about DEKs, see Envelope Encryption.
If using application layer encryption is not an option for your application, another common solution is to use Cloud Storage's default encryption. This option is often used when you are primarily looking for a cloud solution to protect secrets in storage.
This encryption is enabled automatically and requires no additional effort on your part to implement.
Managing access to secrets
There are two main options to restrict and enforce access:
- Access controls on the bucket in which the secret is stored. This option can support multiple secrets (objects) per bucket, or only a single secret per bucket. This is the recommended option.
- Access controls on the key which encrypts the bucket in which the secret is stored. This option supports multiple secrets per key, or only a single secret per key.
You should segregate secrets only when it improves security and ease of use. A common best practice is to limit the amount of data that any one encryption key protects, for cryptographic isolation; or that any one access control list protects. This practice allows you to have greater granular control over secret access, helps prevent accidental permissions, and supports more granular auditing. When you group secrets together, do so when it logically makes sense. For example, you might group some secrets together to simplify control, such as when a single application needs access to a particular collection of secrets at runtime.
When deciding how to store your secrets, it is recommended you use these guidelines:
- Each secret as its own object
- Secrets in a single bucket where they have multiple traits in common
- A single encryption key is used to encrypt each bucket, which includes these logically grouped secrets.
Some scenarios in which you might want to group secrets together include:
- The same application requires secret access
- The same human administrators manage secret versions and access
- The same environment, such as production, development, or test
- The same use time, such as build time or deploy time
- The same desired level of protection
A good practice is to regularly rotate keys to encrypt secrets. Rotating keys helps to limit the amount of data encrypted with a single key, and helps limit the key's lifecycle in the event it becomes compromised. In addition to automatic key rotation, you can also manually rotate a key. For example, you might manually rotate a key when a new version of the secret is updated. To learn more, see Key rotation.
In addition to rotating keys, you can also rotate secrets. A secret is often rotated (or updated) when a new version is created. For example, generating a new password for a database credential. You might also want to rotate keys regularly to limit the secret’s lifecycle.
Secret rotation results in a secret having multiple versions that you must manage. There might be a single version of a secret that is valid at any one time, or multiple versions of secrets. It is recommended that you consider preserving older versions of a secret for a period of time, because they may be needed if the application is rolled back to an earlier version.
One way to manage multiple versions of secrets is to create an object for each version, and store those objects in the same bucket associated with that specific secret. You can then employ a naming convention that you can use to track which version is in use. In addition, you can use a central set of variables to help you determine which secret should be in use at any given time.
Permission management using Cloud IAM
As part of secret management on GCP, it is recommended that you use Cloud IAM. Cloud IAM enables you to create and manage permissions for GCP resources. It also unifies access control for GCP services into a single system and presents a consistent set of operations. To learn more about Cloud IAM, see Cloud IAM Documentation.
A critical concept in role and access management is the separation of duties. You want to avoid having a single individual who is able to both encrypt and decrypt data, as well as manage or create new keys. For more information, see Separation of Duties.
The two ways of managing permissions are:
- Without a service account. This is the recommended option.
- With a service account
Permission management without a service account
For secret management using Cloud KMS, the ideal configuration should minimize unnecessary access and enforce separation of duties. This type of configuration requires several users:
- An organizational-level administrator. This is a user with the
resourcemanager.organizationAdminrole. The organizational-level administrator is typically the business owner of the account. Note that if you used the
resourcemanager.projectCreatorrole instead, that user is granted
owneraccess on those projects. This level of access is usually unnecessary. It is recommended that you do not use this role for secret management.
- A second user that has the
storage.objectAdminrole. This user is responsible for managing secrets. This user also uses a service account that has the
storage.adminrole for the product that contains the Cloud Storage bucket. This service account limits this user's ability to edit the bucket metadata, or delete the bucket.
- A third user with the
cloudkms.adminrole. This user manages the keys used to encrypt secrets.
- A fourth user that has both the
cloudkms.cryptoKeyEncrypterDecrypterroles. This is the end user that needs to access secrets.
Permission management with a service account
An alternative implementation requires that the end user only have permissions on the storage bucket, and have the storage service use a service account to access the key on the user’s behalf. This configuration is similar to the one described in Permission management without a service account, with the following changes:
- The end user accessing the secrets, has the
- A fifth user, which is for the Cloud Storage service account, has the
This configuration allows for a situation where no human has access to encrypt and decrypt with a key, which can be preferable in some cases. However, this configuration allows Cloud Storage to encrypt and decrypt with the key on its own authority.
Auditing using Cloud Audit Logging
A final consideration when managing secrets on GCP is using Cloud Audit Logs. This service consists of two log streams, Admin Activity and Data Access, which are generated by GCP services. These streams help you answer the question of "who did what, where, and when?" within your GCP projects.
Admin Activity logs contain log entries for API calls or administrative actions that modify the configuration or metadata of a service or project. This log is always enabled and is visible by all project members.
Data Access logs contain log entries for API calls that create, modify, or read user-provided data managed by a service, such as data stored in a database service. Data Access logs are visible only by project owners and users with the Private Logs Viewer role.
In both Cloud Storage and Cloud KMS, Admin Activity logs are on by default; these include actions like creating a new bucket, or rotating a key. Admin activity logs are on by default, and require no action from the user to turn on.
Neither Cloud Storage nor Cloud KMS have data access logs turned on by default, as these could potentially be a significant volume of data. These logs track actions that occur frequently; especially when implementing the solution described here. Examples of these actions include reading the bucket, or encrypting or decrypting with a key. In addition, any secret access requires the use of both Cloud Storage and Cloud KMS, so interactions with secrets could potentially be recorded in either location (although doing both would be redundant).
If you opt to use logging, it is recommended you turn on Data Access logs on objects in Cloud Storage, rather than for keys in Cloud KMS. These logs provide data that is more granular and easier to audit than the data access logs on keys in Cloud KMS. You might also want to turn on Data Access logs on Cloud Storage objects for secrets that are particularly critical.