Encrypt secrets at the application layer

This page explains how to encrypt Kubernetes Secrets at the application layer using a key you manage in Cloud Key Management Service (Cloud KMS). Since this feature relies on functionality from Cloud KMS, you should familiarize yourself with key rotation and envelope encryption.

Overview

By default, Google Kubernetes Engine (GKE) encrypts customer content stored at rest, including Secrets. GKE handles and manages this default encryption for you without any additional action on your part.

Application-layer secrets encryption provides an additional layer of security for sensitive data, such as Secrets, stored in etcd. Using this functionality, you can use a key managed with Cloud KMS to encrypt data at the application layer. This encryption protects against attackers who gain access to an offline copy of etcd.

To use application-layer secrets encryption, you must first create a Cloud KMS key and give the GKE service account access to the key. You can use a key that has any of the protection levels supported by Cloud KMS.

The key must be in the same location as the cluster to decrease latency and to prevent cases where resources depend on services spread across multiple failure domains. After creating a key, you can enable the feature on a new or existing cluster by specifying the key you want to use.

Envelope encryption

Kubernetes offers envelope encryption of Secrets with a KMS provider, meaning that a local key, commonly called a data encryption key (DEK), is used to encrypt the Secrets. The DEK itself is encrypted with another key called the key encryption key (KEK). Kubernetes does not store the KEK.

Envelope encryption has the following benefits:

  • The KEK can be rotated without requiring re-encryption of all the Secrets. This means that you can more easily follow the best practice of regular key rotation, without a significant impact on performance.

  • Secrets that are stored in Kubernetes can rely on an external root of trust. This means that you can use a central root of trust, for example a Hardware Security Module, for all your Secrets, and that an adversary accessing your containers offline can't obtain your Secrets.

With application-layer secrets encryption in GKE, your Secrets are encrypted locally, using the AES-CBC provider, with local DEKs, and the DEKs are encrypted with a KEK that you manage in Cloud KMS.

To learn more about envelope encryption, see Envelope encryption.

What happens when you create a Secret

When you create a new Secret, here's what happens:

  1. The Kubernetes API server generates a unique DEK for the Secret using a random number generator.

  2. The Kubernetes API server uses the DEK locally to encrypt the Secret.

  3. The KMS plugin sends the DEK to Cloud KMS for encryption. The KMS plugin uses your project's GKE service account to authenticate to Cloud KMS.

  4. Cloud KMS encrypts the DEK using the KEK, and sends it back to the KMS plugin.

  5. The Kubernetes API server saves the encrypted Secret and the encrypted DEK. The plaintext DEK is not saved to disk.

  6. The Kubernetes API server creates a cache entry mapping the encrypted DEK to the plaintext DEK. This lets the API server decrypt the Secret without using Cloud KMS.

When a client requests a Secret from the Kubernetes API server, here's what happens:

  1. The Kubernetes API server retrieves the encrypted Secret and the encrypted DEK.

  2. The Kubernetes API server checks the cache for an existing mapping entry and decrypts the Secret without using Cloud KMS.

  3. If a cache entry is not found, the KMS plugin sends the DEK to Cloud KMS for decryption using the KEK. The decrypted DEK is then used to decrypt the Secret.

  4. The Kubernetes API server returns the decrypted Secret to the client.

What happens when you destroy a key

When you destroy a KEK in Cloud KMS used to encrypt a Secret in GKE, the Secret is no longer available unless you update the cluster to use a new KEK first.

If you plan to destroy an old KEK version after a key rotation, use the new KEK version to re-encrypt the Secret first.

Unless you use a Service Account Token Volume Projection, service accounts used by your workloads on GKE also use Secrets, and if a key is destroyed these become unavailable. The inability to access these means that the workloads will fail.

The following exceptions apply:

  • Pods with existing access to Secrets as mounted volumes or environment variables retain access.

  • The Kubernetes API server can still use cached DEK mapping entries to decrypt a Secret after you destroy the KEK. This lets restarted or rescheduled Pods access the Secret unless:

    • The cluster control plane is restarted.
    • The Kubernetes API server Pod is restarted.
    • The DEK mapping entry for the Secret is not in the Kubernetes API server cache.

Before you destroy a KEK, check if it is being used by your cluster. You can also create an alerting policy for key destruction in Cloud KMS.

Before you begin

  • To do the exercises in this topic, you need two Google Cloud projects:

    • Key project: This is where you create a KEK.

    • Cluster project: This is where you create a cluster that enables application-layer secrets encryption.

  • In your key project, ensure that you have enabled the Cloud KMS API.

    Enable Cloud KMS API

  • In your key project, the user who creates the key ring and key needs the following IAM permissions:

    • cloudkms.keyRings.getIamPolicy
    • cloudkms.keyRings.setIamPolicy

    These permissions (and many more) are granted to the pre-defined roles/cloudkms.admin Identity and Access Management role. You can learn more about granting permissions to manage keys in the Cloud KMS documentation.

  • In your cluster project, ensure that you have enabled the Google Kubernetes Engine API.

    Enable Google Kubernetes Engine API

  • Ensure that you have installed the Google Cloud CLI.

  • Update gcloud to the latest version:

    gcloud components update

Create a Cloud KMS key

To create a Cloud KMS key, you must first create a key ring. Keys and key rings are regional resources. When you create a key ring, specify a location that matches the location of your GKE cluster:

  • A zonal cluster should use a key ring from a superset region. For example, a cluster in the us-central1-a zone can only use a key in the us-central1 region.

  • A regional cluster should use a key ring from the same location. For example, a cluster in the asia-northeast1 region should be protected with a key ring from the asia-northeast1 region.

You can use the gcloud CLI or the Google Cloud console.

Console

In your key project, create a key ring:

  1. Go to the Key Management page in Cloud console.

    Go to Key Management

  2. Click Create key ring.

  3. In the Key ring name field, enter the name for your key ring.

  4. From the Location dropdown, select the location of your Kubernetes cluster.

  5. Click Create.

Next, create a key:

  1. Go to the Key Management page in Cloud console.

    Go to Key Management

  2. Click the name of the key ring for which you will create a key.

  3. Click Create key.

  4. In the Key name field, enter the name for your key.

  5. Accept the default values for Rotation period and Starting on, or set a key rotation period and starting time if you want to use different values.

  6. [Optional] In the Labels field, click Add label if you want to add labels to your key.

  7. Click Create.

gcloud

In your key project, create a key ring:

gcloud kms keyrings create RING_NAME \
    --location=LOCATION \
    --project=KEY_PROJECT_ID

Replace the following:

  • RING_NAME: the name of your new key ring.
  • LOCATION: the location where you want to create the key ring.
  • KEY_PROJECT_ID: your key project ID.

Create a key:

gcloud kms keys create KEY_NAME \
    --location=LOCATION \
    --keyring=RING_NAME \
    --purpose=encryption \
    --project=KEY_PROJECT_ID

Replace the following:

  • KEY_NAME: the name of your new key.
  • LOCATION: the Cloud KMS location where you created your key ring.
  • RING_NAME: the name of your key ring.
  • KEY_PROJECT_ID: your key project ID.

Grant permission to use the key

The GKE service account in your cluster project has the following name:

service-CLUSTER_PROJECT_NUMBER@container-engine-robot.iam.gserviceaccount.com

Replace CLUSTER_PROJECT_NUMBER with your cluster project number. To find your project number using the gcloud CLI, run the following command:

gcloud projects describe CLUSTER_PROJECT_ID \
    --format="value(projectNumber)"

To grant access to the service account, you can use the Google Cloud console or the gcloud CLI.

Console

Grant your GKE service account the Cloud KMS CryptoKey Encrypter/Decrypter role:

  1. Open the Cloud Key Management Service Keys browser in the Google Cloud console.
    Open the Cloud KMS Keys browser
  2. Click on the name of the key ring that contains the desired key.

  3. Select the checkbox for the desired key.

    The Permissions tab in the right window pane becomes available.

  4. In the Add members dialog, specify the email address of the GKE service account you are granting access.

  5. In the Select a role drop down, select Cloud KMS CryptoKey Encrypter/Decrypter.

  6. Click Save.

gcloud

Grant your GKE service account the Cloud KMS CryptoKey Encrypter/Decrypter role:

gcloud kms keys add-iam-policy-binding KEY_NAME \
    --location=LOCATION \
    --keyring=RING_NAME \
    --member=serviceAccount:SERVICE_ACCOUNT_NAME \
    --role=roles/cloudkms.cryptoKeyEncrypterDecrypter \
    --project=KEY_PROJECT_ID

Replace the following:

  • KEY_NAME: the name of your key.
  • LOCATION: the Cloud KMS location where you created your key ring.
  • RING_NAME: the name of your key ring.
  • SERVICE_ACCOUNT_NAME: the name of your GKE service account.
  • KEY_PROJECT_ID: your key project ID.

Ensure that the key has enough quota if it is a Cloud HSM key

If you use a Cloud HSM key, the Google Cloud project that contains the key is limited by your key quota. Ensure that you have enough quota to use your Cloud HSM keys with application-layer secrets encryption. If your quota is exhausted, your nodes might lose connectivity to your cluster control plane.

Enable application-layer secrets encryption

You can enable application-layer secrets encryption on new or existing GKE Standard and GKE Autopilot clusters using the gcloud CLI or the Cloud console.

After you enable application-layer secrets encryption, we recommend that you perform a key rotation. You can configure automatic key rotation in Cloud KMS. For instructions, see Configuring automatic rotation.

On a new cluster

You can create a new cluster with application-layer secrets encryption enabled by using the Cloud console or the gcloud CLI.

Console

Create a Standard cluster

To create a Standard cluster with application-layer secrets encryption enabled, perform the following steps:

  1. Go to the Google Kubernetes Engine page in the Cloud console.

    Go to Google Kubernetes Engine

  2. Click Create.

  3. In the Standard section, click Configure.

  4. Configure your cluster as desired.

  5. From the navigation pane, under Cluster, click Security.

  6. Select the Enable Application-layer secrets encryption checkbox and choose the key that you created in Create a Cloud KMS key.

  7. Click Create.

Create an Autopilot cluster

To create a Autopilot cluster with application-layer secrets encryption enabled, perform the following steps:

  1. Go to the Google Kubernetes Engine page in the Cloud console.

    Go to Google Kubernetes Engine

  2. Click Create.

  3. In the Autopilot section, click Configure.

  4. Configure your cluster as desired.

  5. Expand the Advanced Options section and locate the Security options.

  6. Select the Enable Application-layer secrets encryption checkbox and choose the key that you created in Create a Cloud KMS key.

  7. Click Create.

gcloud

To create a cluster that supports application-layer secrets encryption, specify a value for the --database-encryption-key parameter in your creation command.

Create a Standard cluster

To create a Standard cluster that supports application-layer secrets encryption, use the following command:

gcloud container clusters create CLUSTER_NAME \
    --cluster-version=latest \
    --region=COMPUTE_REGION \
    --database-encryption-key=projects/KEY_PROJECT_ID/locations/LOCATION/keyRings/RING_NAME/cryptoKeys/KEY_NAME \
    --project=CLUSTER_PROJECT_ID

Create an Autopilot cluster

To create an Autopilot cluster that supports application-layer secrets encryption, use the following command:

gcloud container clusters create-auto CLUSTER_NAME \
    --cluster-version=latest \
    --region=COMPUTE_REGION \
    --database-encryption-key=KEY_PROJECT_ID/locations/LOCATION/keyRings/RING_NAME/cryptoKeys/KEY_NAME \
    --project=CLUSTER_PROJECT_ID

Replace the following:

  • CLUSTER_NAME: the name you choose for your new cluster.
  • COMPUTE_REGION: the Compute Engine region where you want to create the cluster.
  • KEY_PROJECT_ID: your key project ID.
  • LOCATION: the Cloud KMS location where you created your key ring.
  • RING_NAME: the name of your key ring.
  • KEY_NAME: the name of your key.
  • CLUSTER_PROJECT_ID: your cluster's project ID.

On an existing cluster

You can use the gcloud CLI or the Google Cloud console to update an existing cluster to use application-layer secrets encryption.

Console

To update a cluster to support application-layer secrets encryption:

  1. Go to the Google Kubernetes Engine page in Cloud console.

    Go to Google Kubernetes Engine

  2. Click the name of the cluster you want to modify.

  3. Under Security, in the Application-layer secrets encryption field, click Edit Application-layer secrets encryption.

  4. Select the Enable Application-layer secrets encryption checkbox and choose the key you created in Create a Cloud KMS key.

  5. Click Save Changes.

gcloud

To enable application-layer secrets encryptions on an existing cluster, run the following command:

gcloud container clusters update CLUSTER_NAME \
    --region=COMPUTE_REGION \
    --database-encryption-key=KEY_PROJECT_ID/locations/LOCATION/keyRings/RING_NAME/cryptoKeys/KEY_NAME \
    --project=CLUSTER_PROJECT_ID

Replace the following:

  • CLUSTER_NAME: the name of your cluster.
  • COMPUTE_REGION: the Compute Engine region of the cluster.
  • KEY_PROJECT_ID: your key project ID.
  • LOCATION: the Cloud KMS location where you created your key ring.
  • RING_NAME: the name of your key ring.
  • KEY_NAME: the name of your key.
  • CLUSTER_PROJECT_ID: your cluster's project ID.

Update a Cloud KMS key

You can use the gcloud CLI or the Google Cloud console to update an existing cluster to use a new Cloud KMS key.

Console

To update a cluster to use a new Cloud KMS key:

  1. Go to the Google Kubernetes Engine page in Cloud console.

    Go to Google Kubernetes Engine

  2. Click the name of the cluster you want to modify.

  3. Under Security, in the Application-layer secrets encryption field, click Edit Application-layer secrets encryption.

  4. Select the new encryption key you want to use.

  5. Click Save Changes.

gcloud

Update your existing cluster to use a new Cloud KMS key:

gcloud container clusters update CLUSTER_NAME \
    --region=COMPUTE_REGION \
    --database-encryption-key=KEY_PROJECT_ID/locations/LOCATION/keyRings/RING_NAME/cryptoKeys/KEY_NAME \
    --project=CLUSTER_PROJECT_ID

Replace the following:

  • CLUSTER_NAME: the name of your cluster.
  • COMPUTE_REGION: the Compute Engine region of the cluster.
  • KEY_PROJECT_ID: your key project ID.
  • LOCATION: the Cloud KMS location where you created your key ring.
  • RING_NAME: the name of your key ring.
  • KEY_NAME: the name of your key.
  • CLUSTER_PROJECT_ID: your cluster's project ID.

Disable application-layer secrets encryption

To disable application-layer secrets encryption, you can use the gcloud CLI or the Google Cloud console.

Console

  1. Go to the Google Kubernetes Engine page in Cloud console.

    Go to Google Kubernetes Engine

  2. Click the name of the cluster you want to modify.

  3. Under Security, in the Application-layer secrets encryption field, click Edit application-layer secrets encryption.

  4. Clear the Enable Application-layer secrets encryption checkbox.

  5. Click Save Changes.

gcloud

To disable application-layer secrets encryption, run the following command:

gcloud container clusters update CLUSTER_NAME \
    --region=COMPUTE_REGION \
    --disable-database-encryption \
    --project=CLUSTER_PROJECT_ID

Replace the following:

Verify that application-layer secrets encryption is enabled

You can check to see whether a cluster is using application-layer secrets encryption using the Cloud console or the gcloud CLI.

Console

  1. Go to the Google Kubernetes Engine page in Cloud console.

    Go to Google Kubernetes Engine

  2. Click the name of the cluster you want to modify.

  3. Under Security, verify that the Application-layer secrets encryption field displays Enabled and lists the correct key.

gcloud

Check to see whether a cluster is using application-layer secrets encryption:

gcloud container clusters describe CLUSTER_NAME \
    --region=COMPUTE_REGION \
    --format='value(databaseEncryption)' \
    --project=CLUSTER_PROJECT_ID

Replace the following:

If the cluster uses application-layer secrets encryption, the output is similar to the following:

keyName=projects/KEY_PROJECT_ID/locations/LOCATION/keyRings/RING_NAME/cryptoKeys/KEY_NAME;state=ENCRYPTED

Rotate your keys

We recommend that you rotate your keys on a regular schedule, including after you enable application-layer secrets encryption. For instructions to configure automatic key rotation or to manually rotate your keys, see Rotating keys.

When you perform a key rotation, your existing secrets remain encrypted with the previous key encryption key (KEK) version. To ensure a newer KEK version wraps a Secret, re-encrypt the Secret after Key rotation.

For example, you create and store a Secret, Secret1. It is encrypted with DEK1, which itself is wrapped with KEKv1.

After the KEK rotates, you re-encrypt Secret1 so that it is wrapped by DEK2, which in turn is wrapped with KEKv2, the rotated KEK.

Re-encrypt your Secrets

After performing a key rotation, you should re-encrypt your Secrets to wrap them with the new version of the KEK. While you cannot configure automatic re-encryption using the gcloud CLI, you could, for example, use a CronJob to run the re-encryption command at regular intervals.

To manually re-encrypt your Secrets after a key rotation, wait at least three hours for the new version to become consistent. Then, touch every Secret to force the re-encryption using a command like the following:

kubectl get secrets --all-namespaces -o json | kubectl annotate --overwrite -f - encryption-key-rotation-time="TIME"

Replace TIME with a string indicating when the rotation happens (for example, 20200909-090909).

Limitations

  • GKE supports up to 30,000 secrets per cluster for application-layer secrets encryption. If you store more than 30,000 secrets, your cluster might become unstable at upgrade time, causing a potential outage for your workloads.
  • Make sure the average size of metadata of a secret in every namespace is lower than 5KiB. If the average size of metadata is above 5KiB, your cluster might enter a bad state where some secrets are encrypted while others are decrypted after enabling the feature or disabling the feature.
  • You must select a key in the same region as the cluster. For example, a zonal cluster in us-central1-a can only use a key in the region us-central1. For regional clusters, keys must be in the same location to decrease latency and to prevent cases where resources depend on services spread across multiple failure domains.

  • GKE only supports keys from Cloud KMS. You cannot use another Kubernetes KMS provider or another encryption provider.

Troubleshooting

Cloud KMS key is disabled.

GKE's default service account cannot use a disabled Cloud KMS key for application-layer secrets encryption.

To re-enable a disabled key, see Enable a disabled key version.

What's next