Encrypting secrets at the application layer

This page explains how to encrypt Kubernetes Secrets at the application layer using a key you manage in Cloud Key Management Service (Cloud KMS). Since this feature relies on functionality from Cloud KMS, you should familiarize yourself with key rotation and envelope encryption.

Overview

By default, GKE encrypts customer content stored at rest, including Secrets. GKE handles and manages this default encryption for you without any additional action on your part.

Application-layer Secrets Encryption provides an additional layer of security for sensitive data, such as Secrets, stored in etcd. Using this functionality, you can use a key managed with Cloud KMS to encrypt data at the application layer. This protects against attackers who gain access to an offline copy of etcd.

To use Application-layer Secrets Encryption, you must first create a Cloud KMS key and give the GKE service account access to the key. The key must be in the same location as the cluster to decrease latency and to prevent cases where resources depend on services spread across multiple failure domains. Then, you can enable the feature on a new or existing cluster by specifying the key you would like to use.

Note: Cloud KMS usage is subject to a quota. GKE calls the Cloud KMS API every time it performs a cryptographic operation. Confirm that you have enough quota to enable application-layer secrets encryption for your application.

Envelope encryption

Kubernetes offers envelope encryption of Secrets with a KMS provider, meaning that a local key, commonly called a data encryption key (DEK), is used to encrypt the Secrets. The DEK itself is encrypted with another key called the key encryption key. The key encryption key is not stored by Kubernetes.

Envelope encryption has two main benefits:

• The key encryption key can be rotated without requiring re-encryption of all the Secrets. This means that you can more easily follow the best practice of regular key rotation, without a significant impact on performance.

• Secrets that are stored in Kubernetes can rely on an external root of trust. This means that you can use a central root of trust, for example a Hardware Security Module, for all your Secrets, and that an adversary accessing your containers off line isn't able to obtain your Secrets.

With Application-layer Secrets Encryption in GKE, your Secrets are encrypted locally, using the AES-CBC provider, with local DEKs, and the DEKs are encrypted with a key encryption key that you manage in Cloud KMS.

To learn more about envelope encryption, see Envelope encryption.

What happens when you create a Secret

When you create a new Secret, here's what happens:

• The Kubernetes API server generates a unique DEK for the Secret by using a random number generator.

• The Kubernetes API server uses the DEK locally to encrypt the Secret.

• The KMS plugin sends the DEK to Cloud KMS for encryption. The KMS plugin uses your project's GKE service account to authenticate to Cloud KMS.

• Cloud KMS encrypts the DEK, and sends it back to the KMS plugin.

• The Kubernetes API server saves the encrypted Secret and the encrypted DEK. The plaintext DEK is not saved to disk.

When a client requests a Secret from the Kubernetes API server, the process described above is reversed.

What happens when you destroy a Key

Warning: Destroying a key is NOT reversible - any data encrypted with the destroyed version will be unrecoverable.

When you destroy a key in Cloud KMS used to encrypt a Secret in GKE, that Secret is no longer available. Unless using a Service Account Token Volume Projection, Service Accounts used by GKE also use Secrets, and if a key is destroyed these become unavailable. The inability to access these means that the cluster will fail to start.

Prior to destroying a key, it is recommended that you verify if it is being used by your cluster. You can also create an alerting policy for key destruction in Cloud KMS.

Before you begin

• To do the exercises in this topic, you need two Google Cloud projects:

  • Key project: This is where you create a key encryption key.

  • Cluster project: This is where you create a cluster that enables Application-layer Secrets Encryption.

  Note: You can use the same project for your key project and cluster project. But the recommended practice is to use separate projects.

• In your key project, ensure that you have enabled the Cloud KMS API.

  Enable Cloud KMS API

• In your key project, the user who creates the key ring and key needs the following IAM permissions:

  • cloudkms.keyRings.getIamPolicy
  • cloudkms.keyRings.setIamPolicy

  These permissions (and many more) are granted to the pre-defined roles/cloudkms.admin Identity and Access Management role. You can learn more about granting permissions to manage keys in the Cloud KMS documentation.

• In your cluster project, ensure that you have enabled the Google Kubernetes Engine API.

  Enable Google Kubernetes Engine API

• Ensure that you have installed the Cloud SDK.

• Update gcloud to the latest version:

  gcloud components update

Creating a Cloud KMS key

When you create key ring, specify a location that matches the location of your GKE cluster:

• A zonal cluster should use a key ring from a superset location. For example, a cluster in the zone us-central1-a can only use a key in the region us-central1.

• A regional cluster should use a key ring from the same location. For example, a cluster in the region asia-northeast1 should be protected with a key ring from region asia-northeast1.

• The Cloud KMS global region is not supported for use with GKE.

You can use the gcloud tool or the Google Cloud Console.

Console

In your key project, create a key ring:

1. Go to the Cryptographic Keys page in the Cloud Console.

   Go to the Cryptographic Keys page

2. Click Create key ring.

3. In the Key ring name field, enter the name for your key ring.

4. From the Location dropdown, select the location of your Kubernetes cluster.

5. Click Create.

Next, create a key:

1. Go to the Cryptographic Keys page in the Cloud Console.

   Go to the Cryptographic Keys page

2. Click the name of the key ring for which you will create a key.

3. Click Create key.

4. In the Key name field, enter the name for your key.

5. Accept the default values for Rotation period and Starting on, or set a key rotation period and starting time if you want to use different values.

6. [Optional] In the Labels field, click Add label if you want to add labels to your key.

7. Click Create.

gcloud

In your key project, create a key ring:

gcloud kms keyrings create RING_NAME \
    --location LOCATION \
    --project KEY_PROJECT_ID

Replace the following:

• RING_NAME: the name that you choose for your key ring.
• LOCATION: the location where you want to create the key ring.
• KEY_PROJECT_ID: your key project ID.

Create a key:

gcloud kms keys create KEY_NAME \
    --location LOCATION \
    --keyring RING_NAME \
    --purpose encryption \
    --project KEY_PROJECT_ID

Replace the following:

• KEY_NAME: the name that you choose for your key.
• LOCATION: the Cloud KMS location where you created your key ring.
• RING_NAME: the name of your key ring.
• KEY_PROJECT_ID: your key project ID.

Grant permission to use the key

The GKE service account in your cluster project has this name:

[e-mail]

Replace CLUSTER_PROJECT_ID with your cluster project ID.

To grant access to the service account, you can use the Google Cloud Console or the gcloud command.

Console

Grant your GKE service account the Cloud KMS CryptoKey Encrypter/Decrypter role:

1. Open the Cloud Key Management Service Keys browser in the Google Cloud Console.
   Open the Cloud KMS Keys browser

2. Click on the name of the key ring that contains the desired key.

3. Select the checkbox for the desired key.

   The Permissions tab in the right window pane becomes available.

4. In the Add members dialog, specify the email address of the GKE service account you are granting access.

5. In the Select a role drop down, select Cloud KMS CryptoKey Encrypter/Decrypter.

6. Click Save.

gcloud

Grant your GKE service account the Cloud KMS CryptoKey Encrypter/Decrypter role:

gcloud kms keys add-iam-policy-binding KEY_NAME \
  --location LOCATION \
  --keyring RING_NAME \
  --member serviceAccount:SERVICE_ACCOUNT_NAME \
  --role roles/cloudkms.cryptoKeyEncrypterDecrypter \
  --project KEY_PROJECT_ID

Replace the following:

• KEY_NAME: the name of your key.
• LOCATION: the Cloud KMS location where you created your key ring.
• RING_NAME: the name of your key ring.
• SERVICE_ACCOUNT_NAME: the name of your GKE service account.
• KEY_PROJECT_ID: your key project ID.

Enabling Application-layer Secrets Encryption

On a new cluster

You can create a new cluster by using the Google Cloud Console or the gcloud tool.

Console

1. Visit the Google Kubernetes Engine menu in Cloud Console.

   Visit the Google Kubernetes Engine menu

2. Click Create.

3. Configure your cluster as desired.

4. From the navigation pane, under Cluster, click Security.

5. Select Enable Application-layer Secrets Encryption and choose the database encryption key you created in Creating a Cloud KMS key.

6. Click Create.

gcloud

To create a cluster that supports Application-layer Secrets Encryption, specify a value for the --database-encryption-key parameter in your creation command.

gcloud container clusters create CLUSTER_NAME \
  --cluster-version=latest \
  --zone ZONE \
  --database-encryption-key projects/KEY_PROJECT_ID/locations/LOCATION/keyRings/RING_NAME/cryptoKeys/KEY_NAME \
  --project CLUSTER_PROJECT_ID

Replace the following:

• CLUSTER_NAME: the name you choose for your new cluster.
• ZONE: the compute zone where you want to create the cluster.
• KEY_PROJECT_ID: your key project ID.
• LOCATION: the Cloud KMS location where you created your key ring.
• RING_NAME: the name of your key ring.
• KEY_NAME: the name of your key.
• CLUSTER_PROJECT_ID: your cluster's project ID.

On an existing cluster

You can use the gcloud tool or the Google Cloud Console to update an existing cluster to use Application-layer Secrets Encryption.

Console

To update a cluster to support Application-layer Secrets Encryption:

1. Visit the Google Kubernetes Engine menu in Cloud Console.

   Visit the Google Kubernetes Engine menu

2. Click the name of the cluster you want to modify.

3. Under Security, in the Application-layer secrets encryption field, click Edit Application-layer Secrets Encryption.

4. Select the Enable Application-layer Secrets Encryption checkbox and choose the database encryption key you created in Creating a Cloud KMS key.

5. Click Save Changes.

gcloud

To enable Application-layer Secrets Encryptions on an existing cluster, run the following command:

gcloud container clusters update CLUSTER_NAME \
  --zone ZONE \
  --database-encryption-key projects/KEY_PROJECT_ID/locations/LOCATION/keyRings/RING_NAME/cryptoKeys/KEY_NAME \
  --project CLUSTER_PROJECT_ID

Replace the following:

• CLUSTER_NAME: the name of your cluster.
• ZONE: the compute zone of the cluster.
• KEY_PROJECT_ID: your key project ID.
• LOCATION: the Cloud KMS location where you created your key ring.
• RING_NAME: the name of your key ring.
• KEY_NAME: the name of your key.
• CLUSTER_PROJECT_ID: your cluster's project ID.

Updating a Cloud KMS key

Console

To update a cluster to use a new Cloud KMS key:

1. Visit the Google Kubernetes Engine menu in Cloud Console.

   Visit the Google Kubernetes Engine menu

2. Click the name of the cluster you want to modify.

3. Under Security, in the Application-layer secrets encryption field, click Edit Application-layer Secrets Encryption.

4. Select the new encryption key you want to use.

5. Click Save Changes.

gcloud

Update your existing cluster to use a new Cloud KMS key:

gcloud container clusters update CLUSTER_NAME \
  --zone ZONE \
  --database-encryption-key projects/KEY_PROJECT_ID/locations/LOCATION/keyRings/RING_NAME/cryptoKeys/KEY_NAME \
  --project CLUSTER_PROJECT_ID

Replace the following:

• CLUSTER_NAME: the name of your cluster.
• ZONE: the compute zone of the cluster.
• KEY_PROJECT_ID: your key project ID.
• LOCATION: the Cloud KMS location where you created your key ring.
• RING_NAME: the name of your key ring.
• KEY_NAME: the name of your key.
• CLUSTER_PROJECT_ID: your cluster's project ID.

Disabling Applications-layer Secrets Encryption

You can use the gcloud tool or the Google Cloud Console.

Console

1. Visit the Google Kubernetes Engine menu in Cloud Console.

   Visit the Google Kubernetes Engine menu

2. Click the name of the cluster you want to modify.

3. Under Security, in the Application-layer secrets encryption field, click Edit Application-layer Secrets Encryption.

4. Clear the Enable Application-layer Secrets Encryption checkbox.

5. Click Save Changes.

gcloud

To disable Application-layer Secrets Encryption, run the following command:

gcloud container clusters update CLUSTER_NAME \
  --zone ZONE \
  --disable-database-encryption \
  --project CLUSTER_PROJECT_ID

Replace the following:

• CLUSTER_NAME: the name of your cluster.
• ZONE: the compute zone of the cluster.
• CLUSTER_PROJECT_ID: your cluster's project ID.

Verifying that Application-layer Secrets Encryption is enabled

You can check to see whether a cluster is using Application-layer Secrets Encryption using the Google Cloud Console or the gcloud command.

Console

1. Visit the Google Kubernetes Engine menu in Cloud Console.

   Visit the Google Kubernetes Engine menu

2. Click the name of the cluster you want to modify.

3. Under Security, verify that the Application-layer secrets encryption field displays Enabled and lists the correct key.

gcloud

Check to see whether a cluster is using Application-layer Secrets Encryption:

gcloud container clusters describe CLUSTER_NAME \
  --zone ZONE \
  --format 'value(databaseEncryption)' \
  --project CLUSTER_PROJECT_ID

Replace the following:

• CLUSTER_NAME: the name of your cluster.
• ZONE: the compute zone of the cluster.
• CLUSTER_PROJECT_ID: your cluster's project ID.

If the cluster is using Application-layer Secrets Encryption, the response contains EncryptionConfig:

keyName=projects/project/locations/LOCATION/keyRings/RING_NAME/cryptoKeys/KEY_NAME;state=ENCRYPTED

Limitations

Number of secrets

GKE supports up to 10,000 secrets per cluster for application-layer secrets encryption. If you store more secrets than 10,000, your cluster might become unstable at upgrade time, causing a potential outage for your workloads.

Key location

You must select a key in the same region as the cluster that it is being used in. For example, a zonal cluster in us-central1-a can only use a key in the region us-central1. For regional clusters, keys must be in the same location to decrease latency and to prevent cases where resources depend on services spread across multiple failure domains.

Note: The key does not need to be in the same project as the cluster. For more information about the supported locations for Cloud KMS, see Google Cloud locations.

Key rotation

When you perform a key rotation, your existing secrets remain encrypted with the previous key encryption key (KEK). To ensure a newer KEK wraps a Secret, create a new Secret after key rotation.

For example, you create and store a Secret, Secret1. It is encrypted with DEK1, which itself is wrapped with KEKv1. Before KEKv1 rotates, you create another secret, Secret2. Secret2 gets its own key, DEK2. KEKv1 is used again to encrypt the DEK.

After the KEK rotates, we create another secret, Secret3, which gets encrypted with DEK3. DEK3 is in turn encrypted with KEKv2, the rotated KEK.

For instructions on how to manually rotate the KEKs that encrypt your Secrets, see Re-encrypting your secrets.

Note: The key version rotation is an eventually consistent operation and there could be a delay before the new key version takes effect. See Consistency of key versions.

Re-encrypting your secrets

There is no way to force automatic re-encryption of your Secrets. If desired, you can manually rotate your KEK by creating a new key version:

gcloud kms keys versions create --location LOCATION \
   --keyring RING_NAME \
   --key KEY_NAME \
   --primary \
   --project KEY_PROJECT_ID

Replace the following:

• LOCATION: the Cloud KMS location where you created your key ring.
• RING_NAME: the name of your key ring.
• KEY_NAME: the name of your key.
• KEY_PROJECT_ID: your key project ID.

Then force GKE to re-encrypt by touching every Secret:

kubectl get secrets --all-namespaces -o json | kubectl annotate --overwrite -f - encryption-key-rotation-time="TIME"

Replace TIME with a string indicating when the rotation happens (for example, 20200909-090909).

Note: The old key version will still exist, and incur costs. You may want to destroy the old key version if it is no longer in use.

EncryptionConfig

At this time, only keys from Cloud KMS are supported in GKE. You cannot use another Kubernetes KMS provider or another encryption provider.

What's next

• Learn more about Secrets in Kubernetes.
• Learn more about Secret management using Cloud KMS.
• Learn how to harden your cluster.