Exploring container security: Encrypting Kubernetes secrets with Cloud KMS

Editor’s note: This post picks up again on our blog post series on container security at Google.

At Google Cloud, we care deeply about protecting your data. That’s why we encrypt data at rest by default, including data in Google Kubernetes Engine (GKE). For Kubernetes secrets—small bits of data your application needs at build or runtime—your threat model might be different, so storage-layer encryption is insufficient. Today, we’re excited to announce in beta GKE application-layer secrets encryption, using the same keys you manage in our hosted Cloud Key Management Service (KMS).

Secrets in Kubernetes

In a default Kubernetes installation, Kubernetes secrets are stored in etcd in plaintext. In GKE, this is managed for you: GKE encrypts these secrets on disk, and monitors this data for insider access. But this might not be enough to protect those secrets from a potentially malicious insider, or a malicious application in your environment.

Kubernetes 1.7 first introduced application-layer encryption of secrets for differential protection of secrets. Using this feature, you can encrypt your secrets locally, with a key also stored in etcd. If you’re running Kubernetes in an environment that doesn’t provide encryption by default, this helps meet security best practices; however, if a malicious intruder gained access to etcd, they would still practically have access to your secrets.

A few releases later in Kubernetes 1.10, envelope encryption of secrets with a KMS provider was introduced, meaning that a local key is used to encrypt the secrets (known as a “data encryption key”), and that the key is itself encrypted with another key not stored in Kubernetes (a “key encryption key”). This model means that you can regularly rotate the key encryption key without having to re-encrypt all the secrets. Furthermore, it means that you can rely on an external root of trust for your secrets in Kubernetes—systems ike Cloud KMS or HashiCorp Vault.

Using Cloud KMS to protect secrets in Kubernetes

Application-layer secrets encryption is now in beta in GKE, so you can protect secrets with envelope encryption: your secrets are encrypted locally in AES-CBC mode with a local data encryption key, and the data encryption key is encrypted with a key encryption key you manage in Cloud KMS as the root of trust. It’s pretty simple as all the hard work is done for you—all you have to do is choose the key you want to use for a particular cluster.

This approach provides flexibility in your security model, to meet specific requirements you may have:

• Root of trust: You can choose whether to protect your secrets using only Kubernetes, with application-layer software-based encryption with a key from Cloud KMS, or hardware-based encryption from Cloud HSM.
  
• Key rotation: You can implement best practices for regular key rotation for your root of trust.
  
• Separation of duties: You can separate who manages your cluster and who manages and protects your secrets.
  
• Centralized auditing: You can manage and audit key accesses centrally, and use the same key for your secrets in Kubernetes as you use for other secrets in GCP.
  

Getting started

Note that you must use a Cloud KMS key in the same location as your cluster, and that you need to give a GKE service account account to use your key for these operations.

That’s how easy it is to encrypt your Kubernetes secrets with GKE. For more detail, check out Application-layer Secrets Encryption, or this Cloud KMS hands-on lab¹. We also gave a talk on Kubernetes secrets at KubeCon China that provides more color. Now encrypt those secrets!

_{1. Take this lab for free before February 28, 2019 with the code 1j-shh-441)}