Encrypting secrets at the application layer

This page explains how to encrypt Kubernetes Secrets at the application layer using a key you manage in Cloud Key Management Service (Cloud KMS). Since this feature relies on functionality from Cloud KMS, you should familiarize yourself with key rotation and envelope encryption.

Overview

By default, Google Kubernetes Engine (GKE) encrypts customer content stored at rest, including Secrets. GKE handles and manages this default encryption for you without any additional action on your part.

Application-layer secrets encryption provides an additional layer of security for sensitive data, such as Secrets, stored in etcd. Using this functionality, you can use a key managed with Cloud KMS to encrypt data at the application layer. This encryption protects against attackers who gain access to an offline copy of etcd.

To use application-layer secrets encryption, you must first create a Cloud KMS key and give the GKE service account access to the key. You can use a key that has any of the protection levels supported by Cloud KMS.

The key must be in the same location as the cluster to decrease latency and to prevent cases where resources depend on services spread across multiple failure domains. After creating a key, you can enable the feature on a new or existing cluster by specifying the key you would like to use.

Envelope encryption

Kubernetes offers envelope encryption of Secrets with a KMS provider, meaning that a local key, commonly called a data encryption key (DEK), is used to encrypt the Secrets. The DEK itself is encrypted with another key called the key encryption key (KEK). The KEK is not stored by Kubernetes.

Envelope encryption has two main benefits:

  • The KEK can be rotated without requiring re-encryption of all the Secrets. This means that you can more easily follow the best practice of regular key rotation, without a significant impact on performance.

  • Secrets that are stored in Kubernetes can rely on an external root of trust. This means that you can use a central root of trust, for example a Hardware Security Module, for all your Secrets, and that an adversary accessing your containers off line isn't able to obtain your Secrets.

With application-layer secrets encryption in GKE, your Secrets are encrypted locally, using the AES-CBC provider, with local DEKs, and the DEKs are encrypted with a KEK that you manage in Cloud KMS.

To learn more about envelope encryption, see Envelope encryption.

What happens when you create a Secret

When you create a new Secret, here's what happens:

  1. The Kubernetes API server generates a unique DEK for the Secret using a random number generator.

  2. The Kubernetes API server uses the DEK locally to encrypt the Secret.

  3. The KMS plugin sends the DEK to Cloud KMS for encryption. The KMS plugin uses your project's GKE service account to authenticate to Cloud KMS.

  4. Cloud KMS encrypts the DEK using the KEK, and sends it back to the KMS plugin.

  5. The Kubernetes API server saves the encrypted Secret and the encrypted DEK. The plaintext DEK is not saved to disk.

  6. The Kubernetes API server creates a cache entry mapping the encrypted DEK to the plaintext DEK. This lets it decrypt the Secret without using Cloud KMS.

When a client requests a Secret from the Kubernetes API server, here's what happens:

  1. The Kubernetes API server retrieves the encrypted Secret and the encrypted DEK.

  2. The Kubernetes API server checks the cache for an existing mapping entry and decrypts the Secret without using Cloud KMS.

  3. If a cache entry is not found, the KMS plugin sends the DEK to Cloud KMS for decryption using the KEK. The decrypted DEK is then used to decrypt the Secret.

  4. The Kubernetes API server returns the decrypted Secret to the client.

What happens when you destroy a Key

When you destroy a KEK in Cloud KMS used to encrypt a Secret in GKE, the Secret is no longer available unless you update the cluster to use a new KEK first.

If you plan to destroy an old KEK version after a key rotation, use the new KEK version to re-encrypt the Secret first.

Unless using a Service Account Token Volume Projection, Service Accounts used by your workloads on GKE also use Secrets, and if a key is destroyed these become unavailable. The inability to access these means that the workloads will fail.

The following exceptions apply:

  • Pods with existing access to Secrets as mounted volumes or environment variables retain access.

  • The Kubernetes API server can still use cached DEK mapping entries to decrypt a Secret after you destroy the KEK. This lets restarted or rescheduled Pods access the Secret unless:

    • The cluster control plane is restarted.
    • The Kubernetes API server Pod is restarted.
    • The DEK mapping entry for the Secret is not in the Kubernetes API server cache.

Before you destroy a KEK, check if it is being used by your cluster. You can also create an alerting policy for key destruction in Cloud KMS.

Before you begin

  • To do the exercises in this topic, you need two Google Cloud projects:

    • Key project: This is where you create a KEK.

    • Cluster project: This is where you create a cluster that enables application-layer secrets encryption.

  • In your key project, ensure that you have enabled the Cloud KMS API.

    Enable Cloud KMS API

  • In your key project, the user who creates the key ring and key needs the following IAM permissions:

    • cloudkms.keyRings.getIamPolicy
    • cloudkms.keyRings.setIamPolicy

    These permissions (and many more) are granted to the pre-defined roles/cloudkms.admin Identity and Access Management role. You can learn more about granting permissions to manage keys in the Cloud KMS documentation.

  • In your cluster project, ensure that you have enabled the Google Kubernetes Engine API.

    Enable Google Kubernetes Engine API

  • Ensure that you have installed the Cloud SDK.

  • Update gcloud to the latest version:

    gcloud components update

Creating a Cloud KMS key

When you create key ring, specify a location that matches the location of your GKE cluster:

  • A zonal cluster should use a key ring from a superset location. For example, a cluster in the us-central1-a zone can only use a key in the us-central1 region.

  • A regional cluster should use a key ring from the same location. For example, a cluster in the asia-northeast1 region should be protected with a key ring from the asia-northeast1 region.

  • The Cloud KMS global region is not supported for use with GKE.

You can use the gcloud tool or the Google Cloud Console.

Console

In your key project, create a key ring:

  1. Go to the Cryptographic Key Rings page in Cloud Console.

    Go to Cryptographic Key Rings

  2. Click Create key ring.

  3. In the Key ring name field, enter the name for your key ring.

  4. From the Location dropdown, select the location of your Kubernetes cluster.

  5. Click Create.

Next, create a key:

  1. Go to the Cryptographic Key Rings page in Cloud Console.

    Go to Cryptographic Key Rings

  2. Click the name of the key ring for which you will create a key.

  3. Click Create key.

  4. In the Key name field, enter the name for your key.

  5. Accept the default values for Rotation period and Starting on, or set a key rotation period and starting time if you want to use different values.

  6. [Optional] In the Labels field, click Add label if you want to add labels to your key.

  7. Click Create.

gcloud

In your key project, create a key ring:

gcloud kms keyrings create RING_NAME \
    --location LOCATION \
    --project KEY_PROJECT_ID

Replace the following:

  • RING_NAME: the name that you choose for your key ring.
  • LOCATION: the location where you want to create the key ring.
  • KEY_PROJECT_ID: your key project ID.

Create a key:

gcloud kms keys create KEY_NAME \
    --location LOCATION \
    --keyring RING_NAME \
    --purpose encryption \
    --project KEY_PROJECT_ID

Replace the following:

  • KEY_NAME: the name that you choose for your key.
  • LOCATION: the Cloud KMS location where you created your key ring.
  • RING_NAME: the name of your key ring.
  • KEY_PROJECT_ID: your key project ID.

Grant permission to use the key

The GKE service account in your cluster project has the following name:

service-CLUSTER_PROJECT_NUMBER@container-engine-robot.iam.gserviceaccount.com

Replace CLUSTER_PROJECT_NUMBER with your cluster project number. To find your project number using the gcloud tool, run the following command:

gcloud projects describe CLUSTER_PROJECT_ID \
    --format="value(projectNumber)"

To grant access to the service account, you can use the Google Cloud Console or the gcloud tool.

Console

Grant your GKE service account the Cloud KMS CryptoKey Encrypter/Decrypter role:

  1. Open the Cloud Key Management Service Keys browser in the Google Cloud Console.
    Open the Cloud KMS Keys browser
  2. Click on the name of the key ring that contains the desired key.

  3. Select the checkbox for the desired key.

    The Permissions tab in the right window pane becomes available.

  4. In the Add members dialog, specify the email address of the GKE service account you are granting access.

  5. In the Select a role drop down, select Cloud KMS CryptoKey Encrypter/Decrypter.

  6. Click Save.

gcloud

Grant your GKE service account the Cloud KMS CryptoKey Encrypter/Decrypter role:

gcloud kms keys add-iam-policy-binding KEY_NAME \
  --location LOCATION \
  --keyring RING_NAME \
  --member serviceAccount:SERVICE_ACCOUNT_NAME \
  --role roles/cloudkms.cryptoKeyEncrypterDecrypter \
  --project KEY_PROJECT_ID

Replace the following:

  • KEY_NAME: the name of your key.
  • LOCATION: the Cloud KMS location where you created your key ring.
  • RING_NAME: the name of your key ring.
  • SERVICE_ACCOUNT_NAME: the name of your GKE service account.
  • KEY_PROJECT_ID: your key project ID.

Enabling application-layer secrets encryption

On a new cluster

You can create a new cluster by using the Google Cloud Console or the gcloud tool.

Console

  1. Go to the Google Kubernetes Engine page in the Cloud Console.

    Go to Google Kubernetes Engine

  2. Click Create.

  3. Configure your cluster as desired.

  4. From the navigation pane, under Cluster, click Security.

  5. Select Enable Application-layer secrets encryption and choose the database encryption key you created in Creating a Cloud KMS key.

  6. Click Create.

gcloud

To create a cluster that supports application-layer secrets encryption, specify a value for the --database-encryption-key parameter in your creation command.

gcloud container clusters create CLUSTER_NAME \
  --cluster-version=latest \
  --region COMPUTE_REGION \
  --database-encryption-key projects/KEY_PROJECT_ID/locations/LOCATION/keyRings/RING_NAME/cryptoKeys/KEY_NAME \
  --project CLUSTER_PROJECT_ID

Replace the following:

  • CLUSTER_NAME: the name you choose for your new cluster.
  • COMPUTE_REGION: the compute region where you want to create the cluster.
  • KEY_PROJECT_ID: your key project ID.
  • LOCATION: the Cloud KMS location where you created your key ring.
  • RING_NAME: the name of your key ring.
  • KEY_NAME: the name of your key.
  • CLUSTER_PROJECT_ID: your cluster's project ID.

On an existing cluster

You can use the gcloud tool or the Google Cloud Console to update an existing cluster to use application-layer secrets encryption.

Console

To update a cluster to support application-layer secrets encryption:

  1. Go to the Google Kubernetes Engine page in Cloud Console.

    Go to Google Kubernetes Engine

  2. Click the name of the cluster you want to modify.

  3. Under Security, in the Application-layer secrets encryption field, click Edit Application-layer secrets encryption.

  4. Select the Enable Application-layer secrets encryption checkbox and choose the database encryption key you created in Creating a Cloud KMS key.

  5. Click Save Changes.

gcloud

To enable application-layer secrets encryptions on an existing cluster, run the following command:

gcloud container clusters update CLUSTER_NAME \
  --region COMPUTE_REGION \
  --database-encryption-key projects/KEY_PROJECT_ID/locations/LOCATION/keyRings/RING_NAME/cryptoKeys/KEY_NAME \
  --project CLUSTER_PROJECT_ID

Replace the following:

  • CLUSTER_NAME: the name of your cluster.
  • COMPUTE_REGION: the compute region of the cluster.
  • KEY_PROJECT_ID: your key project ID.
  • LOCATION: the Cloud KMS location where you created your key ring.
  • RING_NAME: the name of your key ring.
  • KEY_NAME: the name of your key.
  • CLUSTER_PROJECT_ID: your cluster's project ID.

Updating a Cloud KMS key

Console

To update a cluster to use a new Cloud KMS key:

  1. Go to the Google Kubernetes Engine page in Cloud Console.

    Go to Google Kubernetes Engine

  2. Click the name of the cluster you want to modify.

  3. Under Security, in the Application-layer secrets encryption field, click Edit Application-layer secrets encryption.

  4. Select the new encryption key you want to use.

  5. Click Save Changes.

gcloud

Update your existing cluster to use a new Cloud KMS key:

gcloud container clusters update CLUSTER_NAME \
  --region COMPUTE_REGION \
  --database-encryption-key projects/KEY_PROJECT_ID/locations/LOCATION/keyRings/RING_NAME/cryptoKeys/KEY_NAME \
  --project CLUSTER_PROJECT_ID

Replace the following:

  • CLUSTER_NAME: the name of your cluster.
  • COMPUTE_REGION: the compute region of the cluster.
  • KEY_PROJECT_ID: your key project ID.
  • LOCATION: the Cloud KMS location where you created your key ring.
  • RING_NAME: the name of your key ring.
  • KEY_NAME: the name of your key.
  • CLUSTER_PROJECT_ID: your cluster's project ID.

Disabling application-layer secrets encryption

You can use the gcloud tool or the Google Cloud Console.

Console

  1. Go to the Google Kubernetes Engine page in Cloud Console.

    Go to Google Kubernetes Engine

  2. Click the name of the cluster you want to modify.

  3. Under Security, in the Application-layer secrets encryption field, click Edit application-layer secrets encryption.

  4. Clear the Enable application-layer secrets encryption checkbox.

  5. Click Save Changes.

gcloud

To disable application-layer secrets encryption, run the following command:

gcloud container clusters update CLUSTER_NAME \
  --region COMPUTE_REGION \
  --disable-database-encryption \
  --project CLUSTER_PROJECT_ID

Replace the following:

  • CLUSTER_NAME: the name of your cluster.
  • COMPUTE_REGION: the compute region of the cluster.
  • CLUSTER_PROJECT_ID: your cluster's project ID.

Verifying that application-layer secrets encryption is enabled

You can verify whether a cluster is using application-layer secrets encryption in the Google Cloud Console or the gcloud command-line tool.

Console

  1. Go to the Google Kubernetes Engine page in Cloud Console.

    Go to Google Kubernetes Engine

  2. Click the name of the cluster you want to modify.

  3. Under Security, verify that the Application-layer secrets encryption field displays Enabled and lists the correct key.

gcloud

Check to see whether a cluster is using application-layer secrets encryption:

gcloud container clusters describe CLUSTER_NAME \
  --region COMPUTE_REGION \
  --format 'value(databaseEncryption)' \
  --project CLUSTER_PROJECT_ID

Replace the following:

  • CLUSTER_NAME: the name of your cluster.
  • COMPUTE_REGION: the compute region of the cluster.
  • CLUSTER_PROJECT_ID: your cluster's project ID.

If the cluster is using application-layer secrets encryption, the response contains EncryptionConfig:

keyName=projects/project/locations/LOCATION/keyRings/RING_NAME/cryptoKeys/KEY_NAME;state=ENCRYPTED

Limitations

Number of secrets

GKE supports up to 10,000 secrets per cluster for application-layer secrets encryption. If you store more secrets than 10,000, your cluster might become unstable at upgrade time, causing a potential outage for your workloads.

Key location

You must select a key in the same region as the cluster that it is being used in. For example, a zonal cluster in us-central1-a can only use a key in the region us-central1. For regional clusters, keys must be in the same location to decrease latency and to prevent cases where resources depend on services spread across multiple failure domains.

Key rotation

When you perform a key rotation, your existing secrets remain encrypted with the previous key encryption key (KEK) version. To ensure a newer KEK version wraps a Secret, re-encrypt the Secret after Key rotation.

For example, you create and store a Secret, Secret1. It is encrypted with DEK1, which itself is wrapped with KEKv1.

After the KEK rotates, you re-encrypt Secret1 so that it is wrapped by DEK2, which in turn is wrapped with KEKv2, the rotated KEK.

Re-encrypting your Secrets

There is no way to force automatic re-encryption of your Secrets. If desired, you can manually rotate your KEK by creating a new key version:

gcloud kms keys versions create --location LOCATION \
   --keyring RING_NAME \
   --key KEY_NAME \
   --primary \
   --project KEY_PROJECT_ID

Replace the following:

  • LOCATION: the Cloud KMS location where you created your key ring.
  • RING_NAME: the name of your key ring.
  • KEY_NAME: the name of your key.
  • KEY_PROJECT_ID: your key project ID.

Then force GKE to re-encrypt by touching every Secret:

kubectl get secrets --all-namespaces -o json | kubectl annotate --overwrite -f - encryption-key-rotation-time="TIME"

Replace TIME with a string indicating when the rotation happens (for example, 20200909-090909).

EncryptionConfig

At this time, only keys from Cloud KMS are supported in GKE. You cannot use another Kubernetes KMS provider or another encryption provider.

Troubleshooting

Cloud KMS key is disabled.

GKE's default service account cannot use a disabled Cloud KMS key for application-layer secrets encryption.

To re-enable a disabled key, see Enable a disabled key version.

What's next