Secrets in your organization encompass cryptographic keys, credential sets, API keys, OAuth tokens, and other sensitive data that's needed in order to run your applications. Google Cloud offers the Cloud Key Management Service (Cloud KMS) for key management and Secret Manager for managing secrets. You can use these two services to help secure your Google Cloud environment and help you avoid insecure practices such as storing secrets in code.
Cloud Key Management Service
Cloud KMS is a unified API for performing cryptographic operations and managing cryptographic keys. The keys can be backed by software, hardware (HSM), or external systems (EKM). Cloud KMS is designed so that key material never leaves the service. Cloud KMS also offers customer-managed encryption key (CMEK) integrations that give you control over how Google Cloud services encrypt data at rest. Cloud KMS resources include key rings, keys, and key versions as defined in the following list:
- Key ring: Organizes keys in a specific Google Cloud location.
- Key: Contains zero or more key versions. This named resource exists on exactly one key ring. Keys specify parameters such as type and purpose.
- Key version: Contains key material that's used for performing cryptographic information. A key version can be assigned as the primary version for a key.
Cloud KMS is not a general-purpose secret store where you can expect to securely store secret material and retrieve it later. For that capability, you should use Secret Manager, as explained in Secret Manager.
Cloud KMS resource organization and access control
Key access and key ring access is managed by organizing keys into key rings and projects, and by granting IAM roles on the keys, key rings, and projects. As you build out your cloud environment, follow the guidance in the following list for how to design your key resource hierarchy to reduce risk:
- Create a dedicated project for Cloud KMS that's separate from workload projects.
- Add key rings into the dedicated Cloud KMS project. Create key rings as needed to impose separation of duties.
The following list provides guidance for how to apply IAM roles to support your security and administrative needs:
- Avoid the basic project-wide roles like
vieweron projects that host keys, or on enclosing folders or organizations. Instead, designate an organization administrator that's granted at the organization level, as noted on the Cloud KMS separation of duties page. The Organization Admin functions as the administrator for all the organization's cryptographic keys. Grant other IAM roles at the project level. If you have further concerns about separation of duties, grant IAM roles at the key ring level or key level.
Cloud KMS predefined roles
separation of duties.
- Separate administration roles (
and roles/cloudkms.importer) and usage roles for keys.
- Limit the use of
roles/cloudkms.adminto members of the security administration team and to service accounts that are responsible for creating key rings and key objects through tools such as Terraform.
- For asymmetric keys, grant roles that need private key access
roles/cloudkms.signer) separately from roles that don't need private key access (
- In general, grant the most limited set of permissions to the lowest object in the resource hierarchy.
- Separate administration roles (
As shown in Figure 2.4.1, the example.com reference architecture provides you with projects for storing keys and secrets at the organization level and at the environment (folder) level. The intent of having a separate secret repository per environment is to create a clear separation between organization-level and environment-level secrets.
Cloud KMS infrastructure decisions
When you're setting up Cloud KMS for a project, you must make several decisions regarding your keys. Table 2.8.1 provides you with guidance on what factors to consider when you create keys and key rings.
|Key attribute||Key attribute guidance|
|Key location||Choose the location that's geographically closest to the data on which you will be performing cryptographic operations. Use the same key ring location for CMEK keys that's used for the data that they are encrypting. For more information, see choosing the best type of location in the Cloud KMS documentation.|
|Key protection level||Use protection level
|Use protection level
|Use protection level
|Choose appropriate protection levels for development, staging, and production environments. Because the Cloud Key Management Service API is the same regardless of protection level, you can use different protection levels in different environments, and you can relax protection levels where there is no production data.|
|Key source||Allow Cloud KMS to generate keys unless you have workload requirements that require keys to be generated in a specific manner or environment. For externally generated keys, use Cloud KMS key import to import them as described in Importing keys into Cloud KMS.|
|Key rotation||For symmetric encryption, configure automation key rotation by setting a key rotation period and starting time when you create the key|
|For asymmetric encryption, you must always manually rotate keys, because the new public key must be distributed before the key pair can be used. Cloud KMS doesn't support automatic key rotation for asymmetric keys.|
|If you have indications that keys have been compromised, manually rotate the keys and re-encrypt data that was encrypted by the compromised keys as soon as possible. To re-encrypt data, you typically download the old data, decrypt it with the old key, encrypt the old data using the new key, and then re-upload the re-encrypted data.|
|Key destruction||Destroy old keys when there is no data encrypted by those keys.|
Table 2.8.1 Cloud KMS key attribute guidance
Application data encryption
Your application might use Cloud KMS by calling the API directly to encrypt, decrypt, sign, and verify data. Applications that handle data encryption directly should use the envelope encryption approach, which provides better application availability and scaling behavior.
Note that to perform application data encryption in this way, your application must have IAM access to both the key and the data.
Integrated Google Cloud encryption
By default, Google Cloud encrypts all your data at rest and in transit without requiring any explicit setup by you. This default encryption for data at rest, which is transparent to you, uses Cloud KMS behind the scenes and manages IAM access to the keys on your behalf.
Customer-managed encryption keys (CMEK)
For more control over the keys for encrypting data at rest in a Cloud project, you can use several Google Cloud services that offer the ability to protect data related to those services by using encryption keys managed by the customer within Cloud KMS. These encryption keys are called customer-managed encryption keys (CMEK).
Google Cloud products that offer
might require the keys to be hosted in the same location as the data used that's
with the key. Cloud KMS might use different names for some locations
than other services use. For example, the
Cloud KMS multi-regional location
europe corresponds to the
Cloud Storage multi-region location
EU. Cloud KMS also has some locations that are not available in all
other services. For example, the
Cloud KMS dual-regional location
eur5 has no counterpart in Cloud Storage. You need to identify these
requirements before you create the Cloud KMS key ring so that the key
ring is created in the correct location. Keep in mind that you cannot delete a
The security foundation provides you with an example of CMEK with
Cloud Storage. In the
4-projects folder of the foundation, there's an
example project where a key is created in the key ring that's in the environment
secrets project. That key is set as the default key for a bucket
that's created in the
Importing keys into Cloud KMS
Your workloads might require you to generate the keys outside of Cloud KMS. In this case, you can import key material into Cloud KMS. Furthermore, you might need to provide assurance to relying parties on the key generation and import processes. These additional steps are referred to as a key ceremony.
You use a key ceremony to help people trust that the key is being stored and used securely. Two examples of key ceremonies are the DNSSEC root key signing key (KSK) ceremony and the ceremony used by Google to create new root CA keys. Both of these ceremonies support high transparency and high-assurance requirements because the resulting keys must be trusted by the entire internet community.
The people involved in a key ceremony might include the following (using the role names in the US NIST publication A Profile for U.S. Federal Cryptographic Key Management Systems):
- System authority
- System administrator
- Cryptographic officer
- Key custodian
- Key owner
- Audit administrator
- Key-recovery agent
- Cryptographic key management system operator
- Internal or external witnesses
Before the ceremony, the participants and the audience must all agree to a script for the ceremony. During the ceremony, the participants follow the script and document any exceptions that occur.
Depending on the trust required for the key, you might need to perform the ceremony in a sensitive compartmented information facility (SCIF) or other secure facility that you should identify and reserve as part of preparing for the ceremony.
You need to document the performance of the key ceremony, and you might also need to record a video of the ceremony.
During the key ceremony, you generate the key material and encrypt a known plaintext into ciphertext. You then import the key material into Cloud KMS and use the ciphertext to verify the imported key. After you've successfully completed the key ceremony, you can enable the key in Cloud KMS and use it for cryptographic operations.
Because key ceremonies require a lot of setup and staffing, you should carefully choose which keys require ceremonies.
Note that this is a high-level description of the key ceremony. Depending on the key's trust requirements, you might need more steps.
During the course of operating your workloads, you need to manage the lifecycle of your keys. The US National Information Standards Institute (NIST) special publication (SP) 800-57, Part 1 describes a key management lifecycle that's divided into four phases: pre-operational, operational, post-operational, and destroyed. Table 2.8.2 provides a mapping of the NIST lifecycle functions from the publication to Cloud KMS lifecycle functions.
|Lifecycle phase||NIST operations||Cloud KMS operations|
|Pre-operational||8.1.4 Keying-Material Installation Function||Key import|
|8.1.5 Key Establishment Function||Key creation (symmetric, asymmetric)|
|Operational||8.2.1 Normal Operational Storage Function||Key creation in
|8.2.3 Key Change Function||Key rotation|
|Post-operational||8.3.4 Key Destruction Function||Key destruction (and recovery)|
Table 2.8.2 Cloud KMS key lifecycle
In addition, Cloud KMS offers the capability to disable a key. Typically, you do this to the old key after rotating to a new key. This gives you a period to confirm that no additional data needs to be re-encrypted before the key is destroyed. You can re-enable a disabled key if you discover data that needs to be re-encrypted. When you're sure that there's no more data that was encrypted by using the old key, the key can be scheduled for destruction.
Secret Manager is a secure and convenient storage system for API keys, passwords, certificates, and other sensitive data. Secret Manager provides a central place and single source of truth to manage, access, and audit secrets across Google Cloud. Secret Manager lets you set access policies for each secret and to configure audit logs for each secret access.
Secret Manager resources include secrets and secret versions as defined in the following list:
- Secret: A named resource that contains zero or more secret versions. Secrets let you manage access control and replication policy for your secret versions.
- Secret versions: The actual secret material being stored. Secret Manager allows multiple secret versions to be active at one time and lets you control the lifecycle of the stored secret material (video).
Secret Manager infrastructure decisions
When you're setting up Secret Manager for a project, you need to make several decisions regarding your secrets. Table 2.8.3 provides you with guidance on what factors you should consider.
|Infrastructure||Recommended best practice|
|Resource hierarchy location||For secrets that are used by workloads in a single project, create secrets in the project where the workload resides.|
|For secrets that are used by workloads in multiple projects, use a separate project for secrets that's distinct from the workload projects.|
|Access control||Grant IAM roles directly on secrets, rather than at the organization, folder, or project level. Using an infrastructure as code (IaC) approach can help with managing secret-level IAM role grants.|
|Avoid using the basic project-wide roles like
|Use Secret Manager predefined roles in order to enforce least privilege (video) and separation of duties.|
|Create separate administration and access roles for keys:
|Replication policy||Choose the
|Audit logging||Secret Manager writes Admin Activity audit logs, which record operations that modify the configuration or metadata of a resource. You can't disable Admin Activity audit logs. You should enable data access logs at the folder or organization level so you can obtain and analyze AccessSecretVersion requests. For more information, see the Secret Manager audit logging documentation.|
|Secret rotation||Rotate secrets automatically and have emergency rotation procedures available in case of a compromise.|
|Accessing secret versions||Reference secrets by their version number
rather than by using the
|Service account keys||Don't store Google Cloud service account keys in Secret Manager. Instead, use alternatives that use short-lived credentials rather than stored keys, such as service account impersonation or GKE workload identity. This reduces the risk of an attacker capturing credentials from storage.|
|Secrets for high-availability workloads||For high-availability processes that use secrets, prevent flag-day events by storing multiple sets of credentials (identifier and secret), not just the secret. This helps prevent downtime when a secret is rotated. For example, a workload uses Secret Manager to store a database password. When the password is rotated, process restarts might be required across the workload. If there's only one set of credentials, this can cause an outage. To help mitigate this type of outage, have multiple database user accounts with the same access, and rotate one account's password at a time.|
Table 2.8.3 Secret Manager infrastructure guidance
Secret Manager content decisions
Secret material that's stored in secret versions must be no larger than the size indicated in the Secret Manager quotas and limits documentation. Other than this specification, Secret Manager doesn't restrict the format of the secret material.
For secrets like API keys, SSH keys, and cryptographic keys that can't reside in Cloud KMS, the secrets are generated in external systems that impose formatting constraints. For secrets like passwords where the format and length don't have such constraints, use randomly generated, long, complex values, and use cryptographically strong random number generators.
Secret Manager lifecycle
Table 2.8.4 describes the lifecycle of secrets.
|(Secret does not exist)||Creation||Created|
|Created||Deletion||(Secret does not exist)|
Table 2.8.4 Secret Manager lifecycle for secrets
Table 2.8.5 describes the lifecycle of secret versions.
|(Secret version does not exist)||Creation||Enabled|
|Enabled||Destroy||(Secret version does not exist)|
|Disabled||Destroy||(Secret version does not exist)|
Table 2.8.5 Secret Manager lifecycle for secret versions
Follow the guidance in Table 2.8.6 for best practices that you can use for different secret and secret version lifecycle states.
|Secret lifecycle stage||Secret lifecycle guidance|
|Enabled||Access Secret Manager using the API or a client library unless programmatic access isn't feasible, because that's the most secure method.|
|Use the API or client library to access secrets. You might be unable to use the API because your workload involves legacy apps or closed source tools, or because it requires multi-cloud portability. If you can't use the API or client library directly, use environment variables or file paths, but be aware that those methods increase the risk that attackers can gain unauthorized access to your secrets.|
|Disabled||If you're going to disable a secret version after rotating it, then wait until no other processes are using the old secret version. Secret Manager allows multiple secrets to be enabled at once, but you should disable old secret versions as soon as your workloads aren't using them. You can use Data Access audit logs to help identify unused secret versions.|
|Destroyed||Destroy secret versions after a reasonable period following disablement. Because destroying secret versions is irreversible, you shouldn't destroy them until you're sure the secret version is no longer in use.|
Table 2.8.6 Secret Manager lifecycle best practices