Synchronize secrets to Kubernetes Secrets

Preview

This product or feature is subject to the "Pre-GA Offerings Terms" in the General Service Terms section of the Service Specific Terms. Pre-GA products and features are available "as is" and might have limited support. For more information, see the launch stage descriptions.

This document describes how to synchronize secrets stored in Secret Manager to Kubernetes Secrets within your Google Kubernetes Engine (GKE) clusters.

The synchronization process lets applications running on GKE access secrets from Secret Manager using standard Kubernetes methods, such as environment variables or volume mounts. This is useful for applications that are already designed to read secrets from the Kubernetes Secret object, rather than having to be updated to directly access Secret Manager.

Recommendation: The secret synchronization feature is an alternative to the Secret Manager add-on, which mounts secrets as in-memory files directly into your Pods. Use the Secret Manager add-on whenever your application supports it, because it is a more secure method to access secrets from Secret Manager in GKE. Secret Manager add-on component,

Before you begin

Complete the following prerequisites before you synchronize secrets:

• Enable the Secret Manager and Google Kubernetes Engine APIs.

  Roles required to enable APIs

  To enable APIs, you need the Service Usage Admin IAM role (roles/serviceusage.serviceUsageAdmin), which contains the serviceusage.services.enable permission. Learn how to grant roles.

  Enable the APIs

• Install and then initialize the gcloud CLI. If you previously installed the gcloud CLI, get the latest version by running the gcloud components update command.

• Ensure you have a GKE cluster running. This feature requires GKE version 1.34 and later.

• Ensure Workload Identity Federation for GKE is enabled on your GKE cluster. This is required for authentication. Workload Identity Federation for GKE is enabled by default on an Autopilot cluster.

• Ensure you have the necessary Identity and Access Management permissions to manage GKE clusters and Secret Manager.

Enable secret synchronization on a GKE cluster

Enable the secret synchronization feature when you create a new cluster or update an existing cluster. This feature is available on both Standard and Autopilot clusters.

Enable secret synchronization on a new cluster

To create a new cluster with secret synchronization, use one of these gcloud CLI commands:

Standard cluster

To use Secret Manager on the command line, first Install or upgrade to version 378.0.0 or higher of the Google Cloud CLI. On Compute Engine or GKE, you must authenticate with the cloud-platform scope.

Enable secret synchronization without automatic rotation.

gcloud beta container clusters create CLUSTER_NAME \
    --location=CONTROL_PLANE_LOCATION \
    --workload-pool=PROJECT_ID.svc.id.goog \
    --enable-secret-sync

Enable secret synchronization with automatic rotation. The default interval is 2 minutes.

gcloud beta container clusters create CLUSTER_NAME \
    --location=CONTROL_PLANE_LOCATION \
    --workload-pool=PROJECT_ID.svc.id.goog \
    --enable-secret-sync \
    --enable-secret-sync-rotation

Enable secret synchronization with a custom rotation interval. The minimum interval is 1 minute.

gcloud beta container clusters create CLUSTER_NAME \
    --location=CONTROL_PLANE_LOCATION \
    --workload-pool=PROJECT_ID.svc.id.goog \
    --enable-secret-sync \
    --enable-secret-sync-rotation \
    --secret-sync-rotation-interval=1m

Replace the following:

• CLUSTER_NAME: the name of your GKE cluster
• CONTROL_PLANE_LOCATION: the region or zone of your cluster control plane, such as us-central1 or us-east1-a
• PROJECT_ID: your Google Cloud project ID

Autopilot cluster

To use Secret Manager on the command line, first Install or upgrade to version 378.0.0 or higher of the Google Cloud CLI. On Compute Engine or GKE, you must authenticate with the cloud-platform scope.

Enable secret synchronization without automatic rotation.

gcloud beta container clusters create-auto CLUSTER_NAME \
    --location=CONTROL_PLANE_LOCATION \
    --enable-secret-sync

Enable secret synchronization with automatic rotation. The default interval is 2 minutes.

gcloud beta container clusters create-auto CLUSTER_NAME \
    --location=CONTROL_PLANE_LOCATION \
    --enable-secret-sync \
    --enable-secret-sync-rotation

Enable secret synchronization with a custom rotation interval. The minimum interval is 1 minute.

gcloud beta container clusters create-auto CLUSTER_NAME \
    --location=CONTROL_PLANE_LOCATION \
    --enable-secret-sync \
    --enable-secret-sync-rotation \
    --secret-sync-rotation-interval=1m

Replace the following:

• CLUSTER_NAME: the name of your GKE cluster
• CONTROL_PLANE_LOCATION: the region or zone of your cluster control plane, such as us-central1 or us-east1-a

Enable secret synchronization on an existing cluster

To update existing clusters with secret synchronization, use one of the following gcloud CLI commands:

gcloud

To use Secret Manager on the command line, first Install or upgrade to version 378.0.0 or higher of the Google Cloud CLI. On Compute Engine or GKE, you must authenticate with the cloud-platform scope.

Enable secret synchronization on an existing cluster

gcloud beta container clusters update CLUSTER_NAME \
    --location=CONTROL_PLANE_LOCATION \
    --enable-secret-sync

Enable rotation with a custom interval

gcloud beta container clusters update CLUSTER_NAME \
    --location=CONTROL_PLANE_LOCATION \
    --enable-secret-sync-rotation \
    --secret-sync-rotation-interval=5m

Disable rotation

gcloud beta container clusters update CLUSTER_NAME \
    --location=CONTROL_PLANE_LOCATION \
    --no-enable-secret-sync-rotation

Replace the following:

• CLUSTER_NAME: the name of your GKE cluster
• CONTROL_PLANE_LOCATION: the region or zone of your cluster control plane, such as us-central1 or us-east1-a

Configure secret synchronization

To synchronize secrets, complete the following steps:

1. Create a Kubernetes ServiceAccount using the following command:

   kubectl create serviceaccount KSA_NAME --namespace NAMESPACE
   

   Replace the following:

   • KSA_NAME: the name of your Kubernetes ServiceAccount
   • NAMESPACE: the Kubernetes namespace where you want to create the ServiceAccount

2. Create an IAM allow policy that references the new Kubernetes ServiceAccount and grant the ServiceAccount permission to access the secret:

   gcloud

   To use Secret Manager on the command line, first Install or upgrade to version 378.0.0 or higher of the Google Cloud CLI. On Compute Engine or GKE, you must authenticate with the cloud-platform scope.

   gcloud secrets add-iam-policy-binding SECRET_PROJECT_ID \
      --role=roles/secretmanager.secretAccessor \
      --member=principal://iam.googleapis.com/projects/PROJECT_NUMBER/locations/global/workloadIdentityPools/PROJECT_IDsvc.id.goog/subject/ns/NAMESPACE/sa/KSA_NAME
   

   Replace the following:

   • SECRET_PROJECT_ID: the ID of the project where the secret is stored. This can be the same as the PROJECT_ID if the secret is stored in the same project as the GKE cluster.
   • PROJECT_NUMBER: your Google Cloud project number
   • PROJECT_ID: your Google Cloud project ID
   • NAMESPACE: the Kubernetes namespace
   • KSA_NAME: the name of your Kubernetes ServiceAccount

3. Create a SecretProviderClass custom resource using a YAML manifest. The SecretProviderClass resource defines the specific secrets to retrieve Secret Manager, including their resource names and paths. The Secret Manager add-on also uses the SecretProviderClass resource to list the secrets to mount and the filename to mount them as.

   Create a spc.yaml file with the following content:

   apiVersion: secrets-store.csi.x-k8s.io/v1
   kind: SecretProviderClass
   metadata:
     name: SECRET_PROVIDER_CLASS_NAME
     namespace: NAMESPACE
   spec:
     provider: gke
     parameters:
       secrets: |
         -   resourceName: "projects/SECRET_PROJECT_ID/secrets/SECRET_NAME/versions/SECRET_VERSION"
             path: "FILENAME.txt"
         -   resourceName: "projects/SECRET_PROJECT_ID/secrets/SECRET_NAME/versions/SECRET_VERSION"
             path: "FILENAME1.txt"
   

   Replace the following:

   • SECRET_PROVIDER_CLASS_NAME: the name for your SecretProviderClass object.
   • NAMESPACE: the Kubernetes namespace where you are creating this resource.
   • resourceName: the complete resource identifier for the secret in Secret Manager. This must include the project ID, secret name, and version following the format: projects/SECRET_PROJECT_ID/secrets/SECRET_NAME/versions/SECRET_VERSION.
     • SECRET_PROJECT_ID: the ID of the Google Cloud project where the secret is stored. This can be the same as the PROJECT_ID if the secret is stored in the same project as the GKE cluster.
     • SECRET_NAME: the secret name.
     • SECRET_VERSION: the secret version. The secret version must be in the same region as the cluster.
   • path: the local filename or alias that the secret value will be exposed as within the Kubernetes environment. It's the unique identifier that links a specific Secret Manager version to the local representation used by the cluster. When the secret is synced to a Kubernetes Secret using the SecretSync resource, this path is referenced by the sourcePath field to locate the secret value for syncing. You can map multiple secrets (defined by resourceName) to different path names within the same SecretProviderClass.

4. To apply the manifest, run the following command:

   kubectl apply -f spc.yaml
   

5. Create a SecretSync custom resource using a YAML manifest. This resource instructs the synchronization controller to create or update a Kubernetes Secret based on the contents defined in the SecretProviderClass.

   Create a secret-sync.yaml file with the following content:

   apiVersion: secret-sync.gke.io/v1
   kind: SecretSync
   metadata:
     name: KUBERNETES_SECRET_NAME
     namespace: NAMESPACE
   spec:
     serviceAccountName: KSA_NAME
     secretProviderClassName: SECRET_PROVIDER_CLASS_NAME
     secretObject:
       type: KUBERNETES_SECRET_TYPE
       data:
         -   sourcePath: "FILENAME.txt"
             targetKey: "TARGET_KEY1"
         -   sourcePath: "FILENAME1.txt"
             targetKey: "TARGET_KEY2"
   

   Replace the following:

   • KUBERNETES_SECRET_NAME: the name you want to give to the new Kubernetes Secret that will be created by the SecretSync resource.
   • NAMESPACE: the Kubernetes namespace where the new resource is created. It must be the same namespace as the SecretProviderClass.
   • KSA_NAME: the name of the Kubernetes ServiceAccount that the SecretSync controller uses to create and update the target Kubernetes Secret. This ServiceAccount must have the necessary permissions to access external secrets from Secret Manager.
   • SECRET_PROVIDER_CLASS_NAME: the name of the SecretProviderClass object that you created in the previous step. The SecretSync resource uses this to find the correct configuration for the secrets.
   • KUBERNETES_SECRET_TYPE: the type of Kubernetes Secret to be created, such as Opaque, tls, or docker-registry. This determines how Kubernetes handles the secret's data.
   • sourcePath: the local filename or alias (the value of the path field in the SecretProviderClass) that identifies the secret data to be retrieved. The SecretSync controller uses this sourcePath to request the specific secret content and convert it into a new Kubernetes Secret.
   • targetKey: the key that will be used in the data section of the new Kubernetes Secret being created. This defines how the secret content retrieved using the sourcePath is named and stored in the final Kubernetes Secret object. Using multiple entries in the data array lets you define multiple key-value pairs within the same target Secret.

6. To apply the manifest, run the following command:

   kubectl apply -f secret-sync.yaml
   

   The synchronization controller creates a Kubernetes Secret in the specified namespace. This secret contains the data mapped from Secret Manager.

7. Verify that the Kubernetes Secret is created:

   kubectl get secret KUBERNETES_SECRET_NAME -n NAMESPACE -o yaml
   

   Replace the following:

   • KUBERNETES_SECRET_NAME: the name of the new Kubernetes Secret
   • NAMESPACE: the Kubernetes namespace where the new Secret is created

8. To troubleshoot a synchronization issue, use the following command:

   kubectl describe secretSync KUBERNETES_SECRET_NAME -n NAMESPACE
   

   Replace the following:

   • KUBERNETES_SECRET_NAME: the name of the new Kubernetes Secret
   • NAMESPACE: the Kubernetes namespace where the new Secret should exist

Manage secret rotation

If you enable the --enable-secret-sync-rotation flag, on your cluster, the synchronization controller periodically checks Secret Manager for new versions of the secrets specified in the SecretProviderClass resource. The --secret-sync-rotation-interval flag determines how often the controller checks for updates.

If the controller detects a new secret version in Secret Manager, the controller updates the corresponding Kubernetes Secret. The controller compares hashes of the secret content to update secrets only when changes occur.

Applications using these secrets must detect and reload the updated secret values from the Kubernetes Secret. The application's design determines how this is handled.

Disable secret synchronization

To disable secret synchronization, run the following gcloud CLI command:

gcloud

To use Secret Manager on the command line, first Install or upgrade to version 378.0.0 or higher of the Google Cloud CLI. On Compute Engine or GKE, you must authenticate with the cloud-platform scope.

gcloud beta container clusters update CLUSTER_NAME \
    --location=CONTROL_PLANE_LOCATION \
    --no-enable-secret-sync

Replace the following:

• CLUSTER_NAME: the name of your GKE cluster
• CONTROL_PLANE_LOCATION: the region or zone of your cluster control plane, such as us-central1 or us-east1-a

This command stops the synchronization controller. It doesn't delete any Kubernetes Secrets that have already been created. You must manually delete any synced Kubernetes Secrets if you no longer need them.

Delete synced secrets

To delete a synced Kubernetes Secret, delete the SecretSync resource using the following command:

    kubectl delete secretsync KUBERNETES_SECRET_NAME --namespace NAMESPACE

Replace the following:

• KUBERNETES_SECRET_NAME: the name of the Kubernetes Secret
• NAMESPACE: the Kubernetes namespace where the Secret exists

Migrate from the existing Secrets Store CSI Driver

If you're migrating from your existing installation of the Secrets Store CSI Driver, follow these steps:

1. Update the provider field in your SecretProviderClass from gcp to gke. The following example shows how to update the provider field:

   apiVersion: secrets-store.csi.x-k8s.io/v1
   kind: SecretProviderClass
   metadata:
     name: app-secrets-gke
   spec:
     provider: gke
     parameters:
       secrets: |
         -   resourceName: "projects/87654321/secrets/api-key-secret/versions/2"
             path: "good1.txt"
   

2. Create a SecretSync resource. Use the following sample configuration:

   apiVersion: secret-sync.gke.io/v1
   kind: SecretSync
   metadata:
     name: my-kube-secret
     namespace: NAMESPACE
   spec:
     serviceAccountName: KSA_NAME
     secretProviderClassName: my-app-secrets
     secretObject:
       type: Opaque # Or other Kubernetes Secret types
       data:
         -   sourcePath: "my-secret.txt"
             targetKey: "USERNAME"
         -   sourcePath: "another-secret.txt"
             targetKey: "PASSWORD"
   

Security considerations

Secret Manager offers security features like access control with IAM, customer-managed encryption keys (CMEK), and audit logging. Secrets are encrypted at rest and in transit within Secret Manager. When you synchronize secrets to Kubernetes secrets, their security within the cluster depends on the configuration of your GKE cluster. Consider the following:

• Storage: Kubernetes secrets are stored in etcd, which is GKE's primary datastore. By default, GKE encrypts data at rest. For enhanced security, encrypt Kubernetes secrets at the application layer by using a key that you manage in Cloud Key Management Service(Cloud KMS). Encrypting secrets provides an additional layer of security for sensitive workloads.

• Access control: GKE supports multiple options for managing access to resources within projects and its clusters using role-based access control (RBAC). Overly broad RBAC permissions can expose secrets to unintended workloads or users. Grant access following the principle of least privilege and regularly audit access to both Secret Manager and the Kubernetes secrets.

• Environment variables: To improve security, mount Kubernetes secrets as volumes instead of using them as environment variables. This reduces the risk of accidental logging or exposure to other processes.

What's next

• Explore Kubernetes Secrets.
• Understand Workload Identity.