Using always-on secrets encryption

Google Distributed Cloud version 1.10 supports encrypting secrets without the need for an external KMS (Key Management Service), or any other dependencies.

Enable always-on secrets encryption

Always-on secrets encryption works by automatically generating an encryption key that is used to encrypt secrets before they are stored on the etcd database for that cluster. The secrets are encrypted using a Data Encryption Key (DEK) generated for every Secret write. This DEK is used in AES-GCM mode to encrypt the Secret object. The DEK is in turn encrypted with a Key Encrypting Key (KEK). The encryption algorithm is A256GCM (AES GCM using 256-bit key).

The key version is a version number to indicate the key currently in use.

You can enable secrets encryption after a cluster has already been created.

  • For the admin cluster:

    1. Edit the admin cluster configuration file to add the secretsEncryption section.

    2. Run the gkectl update command.

      gkectl update admin --config ADMIN_CLUSTER_CONFIG_FILE --kubeconfig ADMIN_CLUSTER_KUBECONFIG

  • For a user cluster:

    1. Edit the user cluster configuration file to add the secretsEncryption section.

    2. Run the gkectl update command.

      gkectl update cluster --config USER_CONFIG_FILE --kubeconfig ADMIN_CLUSTER_KUBECONFIG

Replace the following:

  • ADMIN_KUBECONFIG with the path of your admin cluster kubeconfig file.
  • ADMIN_CLUSTER_CONFIG with the path of your admin cluster configuration file.
  • USER_CLUSTER_CONFIG with the path of your user cluster configuration file.

The gkectl update commands provided in this section can also be used for any other updates to the corresponding cluster.

Key storage

The encryption keys for the admin cluster are stored on the admin cluster data disk. This disk is mounted on the admin master machine at /opt/data, and the encryption keys can be found at /opt/data/gke-k8s-kms-plugin/generatedkeys/. These keys must be backed up to retain access to the encrypted secrets used by that key. You should enable VM/storage encryption in the hypervisor or similar measures to ensure that the control plane VM disks are protected.

Key rotation

To rotate an existing encryption key for a cluster, increment the keyVersion in the corresponding admin cluster configuration file or user cluster configuration file, and run the appropriate gkectl update command. This creates a new key matching the new version number, re-encrypts each secret, and securely erases the old one. All subsequent new secrets are encrypted using the new encryption key.

Disable always-on secrets encryption

To disable secrets encryption on an existing cluster, add a disabled: true field. Next, run the corresponding gkectl update command. This update decrypts each existing secret and stores each secret in plain text. All subsequent new secrets are stored in plain text.

secretsEncryption:
  mode: GeneratedKey
  generatedKey:
    keyVersion: KEY_VERSION
    disabled: true