This page explains how to encrypt Kubernetes Secrets at the application layer using a key you manage in Cloud Key Management Service (Cloud KMS).
Application-layer Secrets Encryption is available starting with GKE version 1.11.2.
Overview
By default, GKE encrypts customer content stored at rest, including Secrets. GKE handles and manages this default encryption for you without any additional action on your part.
Application-layer Secrets Encryption provides an additional layer of security for sensitive data, such as Secrets, stored in etcd. Using this functionality, you can use a key, that you manage in Cloud KMS, to encrypt data at the application layer. This protects against attackers who gain access to an offline copy of etcd.
To use Application-layer Secrets Encryption, you must first create a Cloud KMS key and give the GKE service account access to the key. Then, you can enable the feature when you create a new cluster, specifying the key you would like to use.
Envelope encryption
Kubernetes offers envelope encryption of Secrets with a KMS provider, meaning that a local key, commonly called a data encryption key (DEK), is used to encrypt the Secrets. The DEK itself is encrypted with another key called the key encryption key. The key encryption key is not stored by Kubernetes.
Envelope encryption has two main benefits:
The key encryption key can be rotated without requiring re-encryption of all the Secrets. This means that you can more easily follow the best practice of regular key rotation, without a significant impact on performance.
Secrets that are stored in Kubernetes can rely on an external root of trust. This means that you can use a central root of trust, for example a Hardware Security Module, for all your Secrets, and that an adversary accessing your containers off line isn't able to obtain your Secrets.
With Application-layer Secrets Encryption in GKE, your Secrets are encrypted locally, using the AES-CBC provider, with local DEKs, and the DEKs are encrypted with a key encryption key that you manage in Cloud KMS.
To learn more about envelope encryption, see Envelope encryption.
What happens when you create a Secret
When you create a new Secret, here's what happens:
The Kubernetes API server generates a unique DEK for the Secret by using a random number generator.
The Kubernetes API server uses the DEK locally to encrypt the Secret.
The KMS plugin sends the DEK to Cloud KMS for encryption. The KMS plugin uses your project's GKE service account to authenticate to Cloud KMS.
Cloud KMS encrypts the DEK, and sends it back to the KMS plugin.
The Kubernetes API server saves the encrypted Secret and the encrypted DEK. The plaintext DEK is not saved to disk.
When a client requests a Secret from the Kubernetes API server, the process described above is reversed.
What happens when you destroy a Key
When you destroy a key in Cloud KMS used to encrypt a Secret in GKE, that Secret is no longer available. Unless using a Service Account Token Volume Projection, Service Accounts used by GKE also use Secrets, and if a key is destroyed these become unavailable. The inability to access these means that the cluster will fail to start. Destroying a key is NOT reversible - any data encrypted with this version will not be recoverable.
Prior to destroying a key, it is recommended that you verify if it is being used by your cluster. You can also create an alerting policy for key destruction in Cloud KMS.
Before you begin
To do the exercises in this topic, you need two Google Cloud Platform projects.
Key project: This is where you create a key encryption key.
Cluster project: This is where you create a cluster that enables Application-layer Secrets Encryption.
In your key project, ensure that you have enabled the Cloud KMS API.
In your cluster project, ensure that you have enabled the Google Kubernetes Engine API.
Ensure that you have installed the Cloud SDK.
Update
gcloudto the latest version:gcloud components update
Creating a Cloud KMS key
When you create key ring, specify a location that matches the location of your GKE cluster:
A zonal cluster should use a key ring from a superset location. For example, a cluster in the zone
us-central1-acan only use a key in the regionus-central1.A regional cluster should use a key ring from the same location. For example, a cluster in the region
asia-northeast1should be protected with a key ring from regionasia-northeast1.The Cloud KMS
globalregion is not supported for use with GKE.
In your key project, create a key ring:
gcloud kms keyrings create [RING_NAME] \
--location [LOCATION] \
--project [KEY_PROJECT_ID]
where:
[RING_NAME]is a name that you choose for your key ring.[LOCATION]is the region where you want to create the key ring.[KEY_PROJECT_ID]is your key project ID.
Create a key:
gcloud kms keys create [KEY_NAME] \
--location [LOCATION] \
--keyring [RING_NAME] \
--purpose encryption \
--project [KEY_PROJECT_ID]
where:
[KEY_NAME]is a name that you choose for your key.[LOCATION]is the region where you created your key ring.[RING_NAME]is the name of your key ring.[KEY_PROJECT_ID]is your key project ID.
Grant permission to use the key
The GKE service account in your cluster project has this name:
service-[CLUSTER_PROJECT_NUMBER]@container-engine-robot.iam.gserviceaccount.com
where [CLUSTER_PROJECT_NUMBER] is your cluster project number.
Grant your GKE service account the Cloud KMS CryptoKey Encrypter/Decrypter role:
gcloud kms keys add-iam-policy-binding [KEY_NAME] \ --location [LOCATION] \ --keyring [RING_NAME] \ --member serviceAccount:[SERVICE_ACCOUNT_NAME] \ --role roles/cloudkms.cryptoKeyEncrypterDecrypter \ --project [KEY_PROJECT_ID]
where:
[KEY_NAME]is the name of your key.[LOCATION]is the region where you created your key ring.[RING_NAME]is the name of your key ring.[SERVICE_ACCOUNT_NAME]is the name of your GKE service account.[KEY_PROJECT_ID]is your key project ID.
Creating a cluster with Application-layer Secrets Encryption
To create a cluster that supports Application-layer Secrets Encryption,
specify a value for the --database-encryption-key parameter in your creation
command. Use cluster version 1.11.2 or later:
gcloud beta container clusters create [CLUSTER_NAME] \ --cluster-version=latest \ --zone [ZONE] \ --database-encryption-key projects/[KEY_PROJECT_ID]/locations/[LOCATION]/keyRings/[RING_NAME]/cryptoKeys/[KEY_NAME] \ --project [CLUSTER_PROJECT_ID]
where:
[CLUSTER_NAME]is a name that you choose for your cluster.[ZONE]is the zone where you want to create the cluster.[KEY_PROJECT_ID]is your key project ID.[LOCATION]is the location of your key ring.[RING_NAME]is the name of your key ring.[KEY_NAME]is the name of your key.[CLUSTER_PROJECT_ID]is your cluster project ID.
Determining whether your cluster is using Application-layer Secrets Encryption
Check to see whether a cluster is using Application-layer Secrets Encryption:
gcloud beta container clusters describe [CLUSTER_NAME] \ --zone [COMPUTE_ZONE] \ --format 'value(databaseEncryption)' \ --project [CLUSTER_PROJECT_ID]
where:
[CLUSTER_NAME]is the name of an existing cluster.[COMPUTE_ZONE]is the name of the cluster's zone.[CLUSTER_PROJECT_ID]is your cluster project ID.
If the cluster is using Application-layer Secrets Encryption, the response
contains EncryptionConfig:
keyName=projects/[PROJECT]/locations/[LOCATION]/keyRings/[RING_NAME]/cryptoKeys/[KEY_NAME];state=ENCRYPTED
Limitations
Existing clusters
At this time, you cannot enable Application-layer Secrets Encryption for an existing cluster.
Key location
You must select a key in the same region as the cluster that it is being used
in. For example, a zonal cluster in us-central1-a can only use a key in the
region us-central1. For regional clusters, keys must be in the same region.
Key rotation
When you rotate a key in Cloud KMS, the data in GKE is not re-encrypted. Existing data that was previously encrypted is not re-encrypted, but any new data will be encrypted with the new key.
At this time, there is no way to force automatic re-encryption of your Secrets. If desired, you can manually rotate your key encryption key, by creating a new key version:
gcloud kms keys versions create --location [LOCATION] \ --keyring [RING_NAME] \ --key [KEY_NAME] \ --primary \ --project [KEY_PROJECT_ID]
Then in GKE, force re-encryption by touching every Secret:
kubectl get secrets --all-namespaces -o json | kubectl replace -f -
EncryptionConfig
At this time, only the KMS provider, with a key from Cloud KMS, is available in GKE. You cannot use another Kubernetes KMS provider or another encryption provider.
