Application-layer Secrets Encryption

This page explains how to encrypt Kubernetes Secrets at the application layer using a key you manage in Key Management Service (KMS). Since this feature relies on functionality from KMS, you should familiarize yourself with key rotation and envelope encryption.

Updating existing clusters to use Application-layer Secrets Encryption is supported in GKE versions:

• GKE 1.11: v1.11.10 and later
• GKE 1.12: v1.12.7 and later
• GKE 1.13 and later

Overview

By default, GKE encrypts customer content stored at rest, including Secrets. GKE handles and manages this default encryption for you without any additional action on your part.

Application-layer Secrets Encryption provides an additional layer of security for sensitive data, such as Secrets, stored in etcd. Using this functionality, you can use a key managed with KMS to encrypt data at the application layer. This protects against attackers who gain access to an offline copy of etcd.

To use Application-layer Secrets Encryption, you must first create a KMS key and give the GKE service account access to the key. The key must be in the same location as the cluster to decrease latency and to prevent cases where resources depend on services spread across multiple failure domains. Then, you can enable the feature on a new or existing cluster by specifying the key you would like to use.

Envelope encryption

Kubernetes offers envelope encryption of Secrets with a KMS provider, meaning that a local key, commonly called a data encryption key (DEK), is used to encrypt the Secrets. The DEK itself is encrypted with another key called the key encryption key. The key encryption key is not stored by Kubernetes.

Envelope encryption has two main benefits:

• The key encryption key can be rotated without requiring re-encryption of all the Secrets. This means that you can more easily follow the best practice of regular key rotation, without a significant impact on performance.

• Secrets that are stored in Kubernetes can rely on an external root of trust. This means that you can use a central root of trust, for example a Hardware Security Module, for all your Secrets, and that an adversary accessing your containers off line isn't able to obtain your Secrets.

With Application-layer Secrets Encryption in GKE, your Secrets are encrypted locally, using the AES-CBC provider, with local DEKs, and the DEKs are encrypted with a key encryption key that you manage in KMS.

To learn more about envelope encryption, see Envelope encryption.

What happens when you create a Secret

When you create a new Secret, here's what happens:

• The Kubernetes API server generates a unique DEK for the Secret by using a random number generator.

• The Kubernetes API server uses the DEK locally to encrypt the Secret.

• The KMS plugin sends the DEK to KMS for encryption. The KMS plugin uses your project's GKE service account to authenticate to KMS.

• KMS encrypts the DEK, and sends it back to the KMS plugin.

• The Kubernetes API server saves the encrypted Secret and the encrypted DEK. The plaintext DEK is not saved to disk.

When a client requests a Secret from the Kubernetes API server, the process described above is reversed.

What happens when you destroy a Key

Warning: Destroying a key is NOT reversible - any data encrypted with the destroyed version will be unrecoverable.

When you destroy a key in KMS used to encrypt a Secret in GKE, that Secret is no longer available. Unless using a Service Account Token Volume Projection, Service Accounts used by GKE also use Secrets, and if a key is destroyed these become unavailable. The inability to access these means that the cluster will fail to start.

Prior to destroying a key, it is recommended that you verify if it is being used by your cluster. You can also create an alerting policy for key destruction in KMS.

Before you begin

• To do the exercises in this topic, you need two Google Cloud projects.

  • Key project: This is where you create a key encryption key.

  • Cluster project: This is where you create a cluster that enables Application-layer Secrets Encryption.

  Note: You can use the same project for your key project and cluster project. But the recommended practice is to use separate projects.

• In your key project, ensure that you have enabled the KMS API.

  Enable KMS API

• In your key project, the user who creates the key ring and key needs the following Cloud IAM permissions:

  • cloudkms.keyRings.getIamPolicy
  • cloudkms.keyRings.setIamPolicy

  These permissions (and many more) are granted to the pre-defined roles/cloudkms.admin Cloud Identity and Access Management role. You can learn more about granting permissions to manage keys in the KMS documentation.

• In your cluster project, ensure that you have enabled the Google Kubernetes Engine API.

  Enable Google Kubernetes Engine API

• Ensure that you have installed the Cloud SDK.

• Update gcloud to the latest version:

  gcloud components update

Creating a KMS key

When you create key ring, specify a location that matches the location of your GKE cluster:

• A zonal cluster should use a key ring from a superset location. For example, a cluster in the zone us-central1-a can only use a key in the region us-central1.

• A regional cluster should use a key ring from the same location. For example, a cluster in the region asia-northeast1 should be protected with a key ring from region asia-northeast1.

• The KMS global region is not supported for use with GKE.

You can use the Google Cloud Console or the gcloud command.

console

In your key project, create a key ring:

1. Go to the Cryptographic Keys page in the Cloud Console.

   Go to the Cryptographic Keys page

2. Click Create key ring.

3. In the Key ring name field, enter the name for your key ring.

4. From the Location dropdown, select the location of your Kubernetes cluster. Your Create key ring page should look similar to:

   

5. Click Create.

Next, create a key:

1. Go to the Cryptographic Keys page in the Cloud Console.

   Go to the Cryptographic Keys page

2. Click the name of the key ring for which you will create a key.

3. Click Create key.

4. In the Key name field, enter the name for your key.

5. Accept the default values for Rotation period and Starting on, or set a key rotation period and starting time if you want to use different values.

6. [Optional] In the Labels field, click Add label if you want to add labels to your key.

   Your Create key page should look similar to:

   

7. Click Create.

gcloud

In your key project, create a key ring:

gcloud kms keyrings create [RING_NAME] \
    --location [LOCATION] \
    --project [KEY_PROJECT_ID]

where:

• [RING_NAME] is a name that you choose for your key ring.
• [LOCATION] is the region where you want to create the key ring.
• [KEY_PROJECT_ID] is your key project ID.

Create a key:

gcloud kms keys create [KEY_NAME] \
    --location [LOCATION] \
    --keyring [RING_NAME] \
    --purpose encryption \
    --project [KEY_PROJECT_ID]

where:

• [KEY_NAME] is a name that you choose for your key.
• [LOCATION] is the region where you created your key ring.
• [RING_NAME] is the name of your key ring.
• [KEY_PROJECT_ID] is your key project ID.

Grant permission to use the key

The GKE service account in your cluster project has this name:

service-[CLUSTER_PROJECT_NUMBER]@container-engine-robot.iam.gserviceaccount.com

where [CLUSTER_PROJECT_NUMBER] is your cluster project number.

To grant access to the service account, you can use the Google Cloud Console or the gcloud command.

console

Grant your GKE service account the KMS CryptoKey Encrypter/Decrypter role:

1. Open the Key Management Service Keys browser in the Google Cloud Console.
   Open the KMS Keys browser

2. Click on the name of the key ring that contains the desired key.

3. Select the checkbox for the desired key.

   The Permissions tab in the right window pane becomes available.

4. In the Add members dialog, specify the email address of the GKE service account you are granting access.

5. In the Select a role drop down, select Cloud KMS CryptoKey Encrypter/Decrypter.

6. Click Save.

gcloud

Grant your GKE service account the KMS CryptoKey Encrypter/Decrypter role:

gcloud kms keys add-iam-policy-binding [KEY_NAME] \
  --location [LOCATION] \
  --keyring [RING_NAME] \
  --member serviceAccount:[SERVICE_ACCOUNT_NAME] \
  --role roles/cloudkms.cryptoKeyEncrypterDecrypter \
  --project [KEY_PROJECT_ID]

where:

• [KEY_NAME] is the name of your key.
• [LOCATION] is the region where you created your key ring.
• [RING_NAME] is the name of your key ring.
• [SERVICE_ACCOUNT_NAME] is the name of your GKE service account.
• [KEY_PROJECT_ID] is your key project ID.

Enabling Application-layer Secrets Encryption

On a new cluster

You can create a new cluster using the Google Cloud Console or the gcloud command.

console

1. Visit the Google Kubernetes Engine menu in Cloud Console.

   Visit the Google Kubernetes Engine menu

2. Click Create cluster.

3. Configure your cluster and node pools as desired. Before saving, select Availability, networking, security, and additional features.

4. Select Enable Application-layer Secrets Encryption and choose the database encryption key you created in Creating a KMS key.

5. Continue configuring your cluster and save your changes.

gcloud

To create a cluster that supports Application-layer Secrets Encryption, specify a value for the --database-encryption-key parameter in your creation command.

gcloud container clusters create [CLUSTER_NAME] \
  --cluster-version=latest \
  --zone [ZONE] \
  --database-encryption-key projects/[KEY_PROJECT_ID]/locations/[LOCATION]/keyRings/[RING_NAME]/cryptoKeys/[KEY_NAME] \
  --project [CLUSTER_PROJECT_ID]

where:

• [CLUSTER_NAME] is a name that you choose for your cluster.
• [ZONE] is the zone where you want to create the cluster.
• [KEY_PROJECT_ID] is your key project ID.
• [LOCATION] is the location of your key ring.
• [RING_NAME] is the name of your key ring.
• [KEY_NAME] is the name of your key.
• [CLUSTER_PROJECT_ID] is your cluster project ID.

On an existing cluster

You can update an existing cluster to use Application-layer Secrets Encryption, as long as one of the following statements is true:

• Cluster version is greater than or equal to v1.11.9 and less than v1.12.0.
• Cluster version is greater than or equal to v1.12.7.

You can use the Google Cloud Console or the gcloud command.

console

To update a cluster to support Application-layer Secrets Encryption:

1. Visit the Google Kubernetes Engine menu in Cloud Console.

   Visit the Google Kubernetes Engine menu

2. Click the Edit icon for the cluster you want to modify. It looks like a pencil.

3. Enable Application-layer Secrets Encryption and choose the database encryption key you created in Creating a KMS key.

4. Click Save.

gcloud

To enable Application-layer Secrets Encryptions on an existing cluster, run the following command:

gcloud container clusters update [CLUSTER_NAME] \
  --zone [ZONE] \
  --database-encryption-key projects/[KEY_PROJECT_ID]/locations/[LOCATION]/keyRings/[RING_NAME]/cryptoKeys/[KEY_NAME] \
  --project [CLUSTER_PROJECT_ID]

where:

• [CLUSTER_NAME] is a name that you choose for your cluster.
• [ZONE] is the zone where you want to create the cluster.
• [KEY_PROJECT_ID] is your key project ID.
• [LOCATION] is the location of your key ring.
• [RING_NAME] is the name of your key ring.
• [KEY_NAME] is the name of your key.
• [CLUSTER_PROJECT_ID] is your cluster project ID.

Disabling Applications-layer Secrets Encryption

You can update an existing cluster to stop using Application-layer Secrets Encryption as long as one of the following statements is true:

• Cluster version is greater than or equal to v1.11.9 and less than v1.12.0.
• Cluster version is greater than or equal to v1.12.7.

You can use the Google Cloud Console or the gcloud command.

console

1. Visit the Google Kubernetes Engine menu in Cloud Console.

   Visit the Google Kubernetes Engine menu

2. Click the Edit icon for the cluster you want to modify. It looks like a pencil.

3. Disable Application-layer Secrets Encryption.

4. Click Save.

gcloud

To disable Application-layer Secrets Encryption, run the following command:

gcloud container clusters update [CLUSTER_NAME] \
  --zone [ZONE] \
  --disable-database-encryption \
  --project [CLUSTER_PROJECT_ID]

where:

• [CLUSTER_NAME] is a name that you choose for your cluster.
• [ZONE] is the zone where you want to create the cluster.
• [CLUSTER_PROJECT_ID] is your cluster project ID.

Verifying that Application-layer Secrets Encryption is enabled

You can check to see whether a cluster is using Application-layer Secrets Encryption using the Google Cloud Console or the gcloud command.

console

1. Visit the Google Kubernetes Engine menu in Cloud Console.

   Visit the Google Kubernetes Engine menu

2. Click the name of the cluster you want to modify. The cluster's Details page opens.

3. Verify that Application-layer Secrets Encryption is enabled, and that the correct key is listed.

gcloud

Check to see whether a cluster is using Application-layer Secrets Encryption:

gcloud container clusters describe [CLUSTER_NAME] \
  --zone [COMPUTE_ZONE] \
  --format 'value(databaseEncryption)' \
  --project [CLUSTER_PROJECT_ID]

where:

• [CLUSTER_NAME] is the name of an existing cluster.
• [COMPUTE_ZONE] is the name of the cluster's zone.
• [CLUSTER_PROJECT_ID] is your cluster project ID.

If the cluster is using Application-layer Secrets Encryption, the response contains EncryptionConfig:

keyName=projects/[PROJECT]/locations/[LOCATION]/keyRings/[RING_NAME]/cryptoKeys/[KEY_NAME];state=ENCRYPTED

Limitations

Key location

You must select a key in the same region as the cluster that it is being used in. For example, a zonal cluster in us-central1-a can only use a key in the region us-central1. For regional clusters, keys must be in the same location to decrease latency and to prevent cases where resources depend on services spread across multiple failure domains.

Note: The key does not need to be in the same project as the cluster. For more information about the supported locations for KMS, see Cloud Locations.

Key rotation

KEKs that encrypt Secrets are not rotated and remain with the Secret for its life. To ensure a different KEK wraps a Secret, rotate the KEK and then create a new Secret.

For example, you create and store a Secret, Secret1. It is encrypted with DEK1, which itself is wrapped with KEKv1. Before KEKv1 rotates, you create another secret, Secret2. Secret2 gets its own key, DEK2. KEKv1 is used again to encrypt the DEK.

After the KEK rotates, we create another secret, Secret3, which gets encrypted with DEK3. DEK3 is in turn encrypted with KEKv2, the rotated KEK.

For instructions on how to manually rotate the KEKs that encrypt your Secrets, see Re-encrypting your secrets.

Re-encrypting your secrets

At this time, there is no way to force automatic re-encryption of your Secrets. If desired, you can manually rotate your KEK by creating a new key version:

gcloud kms keys versions create --location [LOCATION] \
   --keyring [RING_NAME] \
   --key [KEY_NAME] \
   --primary \
   --project [KEY_PROJECT_ID]

Then force GKE to re-encrypt by touching every Secret:

kubectl get secrets --all-namespaces -o json | kubectl replace -f -

Note: The old key version will still exist, and incur costs. You may want to destroy the old key version if it is no longer in use.

EncryptionConfig

At this time, only keys from KMS are supported in GKE. You cannot use another Kubernetes KMS provider or another encryption provider.

What's next

• Learn more about Secrets in Kubernetes.
• Learn more about Secret management using Cloud KMS.
• Learn how to harden your cluster.