Use Secret Manager add-on with Google Kubernetes Engine

The integration between Secret Manager and Google Kubernetes Engine (GKE) lets you store sensitive data such as passwords and certificates used by GKE clusters as secrets in Secret Manager.

This page explains how you can use the Secret Manager add-on to access the secrets stored in Secret Manager as volumes mounted in Kubernetes Pods.

This process involves the following steps:

1. Enable the Secret Manager add-on on a new or existing GKE cluster.
2. Configure applications to authenticate to the Secret Manager API.
3. Define which secrets to mount onto Kubernetes Pods using a SecretProviderClass YAML file. The Secret Manager add-on supports both global and regional secrets.
4. Create a volume where the secrets will be mounted. After the volume is attached, applications in the container can access the data in the container file system.

The Secret Manager add-on is derived from the open source Kubernetes Secrets Store CSI Driver and the Google Secret Manager provider. If you're using the open source Secrets Store CSI Driver to access secrets, you can migrate to the Secret Manager add-on. For information, see Migrate from the existing Secrets Store CSI Driver.

Important: This document uses the term secret to refer to sensitive data stored in Secret Manager, and not to the Secret object in the GKE API.

Benefits

The Secret Manager add-on provides the following benefits:

• You can use a fully managed and supported solution to access Secret Manager secrets from within GKE without any operational overhead.
• You don't have to write custom code to access secrets stored in Secret Manager.
• You can store and manage all your secrets centrally in Secret Manager and selectively access secrets from GKE pods using the Secret Manager add-on. By doing this, you can use features offered by Secret Manager such as CMEK encryption, fine-grained access control, managed rotation, lifecycle management, and audit logs, along with using Kubernetes features such as passing secrets to containers in the form of mounted volumes.
• The Secret Manager add-on is supported on both Standard clusters and Autopilot clusters.
• The Secret Manager add-on supports nodes that use Container-Optimized OS or Ubuntu node images.

Limitations

The Secret Manager add-on has the following limitations:

• The Secret Manager add-on doesn't support the following features that are available in the open source Secrets Store CSI Driver:

  • Secret auto rotation
  • Sync as Kubernetes Secret

• The Secret Manager add-on doesn't support Windows Server nodes.

Before you begin

• Enable the Secret Manager and Google Kubernetes Engine APIs.

  Enable the APIs

• If you want to use the Google Cloud CLI for this task, install and then initialize the gcloud CLI. If you previously installed the gcloud CLI, get the latest version by running the gcloud components update command.

  You can't manually set up the Secret Manager add-on using the Google Cloud SDK or the Google Cloud console.

  Note: For existing gcloud CLI installations, make sure to set the compute/region and compute/zone properties. By setting default locations, you can avoid errors in gcloud CLI like the following: One of [--zone, --region, --location] must be supplied: Please specify location.

• Ensure that your cluster runs GKE version 1.27.14-gke.1042001 or later with a Linux node image.

• If you use a GKE Standard cluster, ensure that your cluster has Workload Identity Federation for GKE enabled. Workload Identity Federation for GKE is enabled by default on an Autopilot cluster. Kubernetes Pods use Workload Identity Federation for GKE to authenticate to the Secret Manager API.

Enable the Secret Manager add-on

You can enable the Secret Manager add-on on both Standard clusters as well as Autopilot clusters.

Enable the Secret Manager add-on on a new GKE cluster

To enable the Secret Manager add-on on cluster creation, do the following:

Console

1. Go to the Google Kubernetes Engine page in the Google Cloud console.

   Go to Google Kubernetes Engine

2. Click Create.

3. In the Create cluster dialog, click Configure.

4. In the navigation menu, in the Cluster section, click Security.

5. Select the Enable Secret Manager checkbox.

6. Select the Enable Workload Identity checkbox.

7. Continue configuring the cluster, and then click Create.

gcloud

{ Standard cluster}

To enable the Secret Manager add-on on a new Standard cluster, run the following command:

Before using any of the command data below, make the following replacements:

• CLUSTER_NAME: the name of your cluster.
• LOCATION: the Compute Engine region for the cluster, such as us-central1.
• VERSION: the specific GKE version that you want to use. Ensure that your cluster runs GKE version 1.27.14-gke.1042001 or later. If the default release channel doesn't include this version, use the --release-channel flag to choose a release channel that does.
• PROJECT_ID: the ID of your Google Cloud project.

Execute the following command:

Linux, macOS, or Cloud Shell

Note: Ensure you have initialized the Google Cloud CLI with authentication and a project by running either gcloud init; or gcloud auth login and gcloud config set project.

gcloud container clusters create CLUSTER_NAME \
    --enable-secret-manager \
    --location=LOCATION \
    --cluster-version=VERSION \
    --workload-pool=PROJECT_ID.svc.id.goog

Windows (PowerShell)

Note: Ensure you have initialized the Google Cloud CLI with authentication and a project by running either gcloud init; or gcloud auth login and gcloud config set project.

gcloud container clusters create CLUSTER_NAME `
    --enable-secret-manager `
    --location=LOCATION `
    --cluster-version=VERSION `
    --workload-pool=PROJECT_ID.svc.id.goog

Windows (cmd.exe)

Note: Ensure you have initialized the Google Cloud CLI with authentication and a project by running either gcloud init; or gcloud auth login and gcloud config set project.

gcloud container clusters create CLUSTER_NAME ^
    --enable-secret-manager ^
    --location=LOCATION ^
    --cluster-version=VERSION ^
    --workload-pool=PROJECT_ID.svc.id.goog

{ Autopilot cluster}

To enable the Secret Manager add-on on a new Autopilot cluster, run the following command:

Before using any of the command data below, make the following replacements:

• CLUSTER_NAME: the name of your cluster.
• VERSION: the specific GKE version that you want to use. Ensure that your cluster runs GKE version 1.27.14-gke.1042001 or later. To set a specific version, see Set the version and release channel of a new Autopilot cluster.
• LOCATION: the Compute Engine region for the cluster, such as us-central1.

Execute the following command:

Linux, macOS, or Cloud Shell

Note: Ensure you have initialized the Google Cloud CLI with authentication and a project by running either gcloud init; or gcloud auth login and gcloud config set project.

gcloud container clusters create-auto CLUSTER_NAME \
    --enable-secret-manager \
    --cluster-version=VERSION \
    --location=LOCATION

Windows (PowerShell)

Note: Ensure you have initialized the Google Cloud CLI with authentication and a project by running either gcloud init; or gcloud auth login and gcloud config set project.

gcloud container clusters create-auto CLUSTER_NAME `
    --enable-secret-manager `
    --cluster-version=VERSION `
    --location=LOCATION

Windows (cmd.exe)

Note: Ensure you have initialized the Google Cloud CLI with authentication and a project by running either gcloud init; or gcloud auth login and gcloud config set project.

gcloud container clusters create-auto CLUSTER_NAME ^
    --enable-secret-manager ^
    --cluster-version=VERSION ^
    --location=LOCATION

After you have enabled the Secret Manager add-on, you can use the Secrets Store CSI Driver in Kubernetes volumes using the driver and provisioner name: secrets-store-gke.csi.k8s.io.

Enable the Secret Manager add-on on an existing GKE cluster

To enable the Secret Manager add-on on an existing cluster, do the following:

Console

1. Go to the Google Kubernetes Engine page in the Google Cloud console.

   Go to Google Kubernetes Engine

2. In the cluster list, click the name of the cluster you want to modify.

3. On the cluster details page, in the Security section, click Secret Manager.

4. In the Edit Secret Manager dialog, select the Enable Secret Manager checkbox.

5. Click Save changes.

gcloud

Before using any of the command data below, make the following replacements:

• CLUSTER_NAME: the name of your cluster
• LOCATION: the Compute Engine region for the cluster, such as us-central1

Execute the following command:

Linux, macOS, or Cloud Shell

Note: Ensure you have initialized the Google Cloud CLI with authentication and a project by running either gcloud init; or gcloud auth login and gcloud config set project.

gcloud container clusters update CLUSTER_NAME \
    --enable-secret-manager \
    --location=LOCATION \

Windows (PowerShell)

Note: Ensure you have initialized the Google Cloud CLI with authentication and a project by running either gcloud init; or gcloud auth login and gcloud config set project.

gcloud container clusters update CLUSTER_NAME `
    --enable-secret-manager `
    --location=LOCATION `

Windows (cmd.exe)

Note: Ensure you have initialized the Google Cloud CLI with authentication and a project by running either gcloud init; or gcloud auth login and gcloud config set project.

gcloud container clusters update CLUSTER_NAME ^
    --enable-secret-manager ^
    --location=LOCATION ^

Verify the Secret Manager add-on installation

To verify that the Secret Manager add-on is installed on the Kubernetes cluster, run the following command:

  gcloud container clusters describe CLUSTER_NAME --location LOCATION | grep secretManagerConfig -A 1

Replace the following:

• CLUSTER_NAME: the name of the cluster
• LOCATION: the location of your cluster, such as us-central1

Configure applications to authenticate to the Secret Manager API

The Google Secret Manager provider uses the workload identity of the Pod that a secret is mounted onto when authenticating to the Secret Manager API. To allow your applications to authenticate to the Secret Manager API using Workload Identity Federation for GKE, follow these steps:

• Create a new Kubernetes ServiceAccount or use an existing Kubernetes ServiceAccount in the same namespace as the Pod onto which you want to mount the secret.

• Create an Identity and Access Management (IAM) allow policy for the secret in Secret Manager.

Pods that use the configured Kubernetes ServiceAccount automatically authenticate as the IAM principal identifier that corresponds to the Kubernetes ServiceAccount when accessing the Secret Manager API.

Create a new Kubernetes ServiceAccount

1. Save the following manifest as service-account.yaml:

   apiVersion: v1
   kind: ServiceAccount
   metadata:
     name: KSA_NAME
     namespace: NAMESPACE
   

   Replace the following:

   • KSA_NAME: the name of your new Kubernetes ServiceAccount
   • NAMESPACE: the name of the Kubernetes namespace for the ServiceAccount

2. Apply the manifest:

   kubectl apply -f service-account.yaml
   

3. Create an IAM allow policy that references the new Kubernetes ServiceAccount and grant it permission to access the secret:

   gcloud secrets add-iam-policy-binding SECRET_NAME \
       --role=roles/secretmanager.secretAccessor \
       --member=principal://iam.googleapis.com/projects/PROJECT_NUMBER/locations/global/workloadIdentityPools/PROJECT_ID.svc.id.goog/subject/ns/NAMESPACE/sa/KSA_NAME
   

   Replace the following:

   • SECRET_NAME: the name of the secret in Secret Manager
   • PROJECT_NUMBER: your numerical Google Cloud project number
   • PROJECT_ID: the project ID of the Google Cloud project that contains your GKE cluster
   • NAMESPACE: the name of the Kubernetes namespace for the ServiceAccount
   • KSA_NAME: the name of your existing Kubernetes ServiceAccount

Use an existing Kubernetes ServiceAccount

Create an IAM allow policy that references the existing Kubernetes ServiceAccount and grant it permission to access the secret:

gcloud secrets add-iam-policy-binding SECRET_NAME \
    --role=roles/secretmanager.secretAccessor \
    --member=principal://iam.googleapis.com/projects/PROJECT_NUMBER/locations/global/workloadIdentityPools/PROJECT_ID.svc.id.goog/subject/ns/NAMESPACE/sa/KSA_NAME

Replace the following:

• SECRET_NAME: the name of the secret in Secret Manager
• PROJECT_NUMBER: your numerical Google Cloud project number
• PROJECT_ID: the project ID of the Google Cloud project that contains your GKE cluster
• NAMESPACE: the name of the Kubernetes namespace for the ServiceAccount
• KSA_NAME: the name of your existing Kubernetes ServiceAccount

Define which secrets to mount

To specify which secrets to mount as files in the Kubernetes Pod, create a SecretProviderClass YAML manifest and list the secrets to mount and the filename to mount them as. Follow these steps:

1. Save the following manifest as app-secrets.yaml:

   apiVersion: secrets-store.csi.x-k8s.io/v1
   kind: SecretProviderClass
   metadata:
     name: SECRET_PROVIDER_CLASS_NAME
   spec:
     provider: gke
     parameters:
       secrets: |
         - resourceName: "projects/PROJECT_ID/secrets/SECRET_NAME/versions/SECRET_VERSION"
           path: "FILENAME.txt"
   

   Replace the following:

   • SECRET_PROVIDER_CLASS_NAME: the name for your SecretProviderClass object.
   • PROJECT_ID: your project ID.
   • SECRET_NAME: the secret name.
   • SECRET_VERSION: the secret version.
   • FILENAME.txt: the filename where the secret value will be mounted. You can create multiple files using the resourceName and path variables.

   For a regional secret, resourceName is the complete path to the secret resource that includes the location of the regional secret. For example, "projects/PROJECT_ID/locations/LOCATION/secrets/SECRET_NAME/versions/SECRET_VERSION"

2. Apply the manifest:

   kubectl apply -f app-secrets.yaml
   

3. Verify that the SecretProviderClass object is created:

   kubectl get SecretProviderClasses
   

Configure a volume where the secrets will be mounted

1. Save the following configuration as my-pod.yaml:

   apiVersion: v1
   kind: Pod
   metadata:
     name: POD_NAME
     namespace: NAMESPACE
   spec:
     serviceAccountName: KSA_NAME
     containers:
     - image: IMAGE_NAME
       imagePullPolicy: IfNotPresent
       name: POD_NAME
       resources:
         requests:
           cpu: 100m
       stdin: true
       stdinOnce: true
       terminationMessagePath: /dev/termination-log
       terminationMessagePolicy: File
       tty: true
       volumeMounts:
         - mountPath: "/var/secrets"
           name: mysecret
     volumes:
     - name: mysecret
       csi:
         driver: secrets-store-gke.csi.k8s.io
         readOnly: true
         volumeAttributes:
           secretProviderClass: SECRET_PROVIDER_CLASS_NAME
   

   Replace the following:

   • POD_NAME: the name of the Kubernetes Pod where the secret is mounted
   • NAMESPACE: the name of the Kubernetes namespace for the ServiceAccount
   • KSA_NAME: the Kubernetes ServiceAccount that you set up in the step Configure applications to authenticate to the Secret Manager API
   • IMAGE_NAME: name of the container image
   • SECRET_PROVIDER_CLASS_NAME: the name for your SecretProviderClass object

2. In Standard clusters only, add the following to the template.spec field to place the Pods on node pools that use Workload Identity Federation for GKE.

   Skip this step in Autopilot clusters, which reject this nodeSelector because every node uses Workload Identity Federation for GKE.

   spec:
     nodeSelector:
       iam.gke.io/gke-metadata-server-enabled: "true"
   

3. Apply the configuration to your cluster.

   kubectl apply -f my-pod.yaml
   

This step mounts a volume mysecret at /var/secrets using the CSI driver (secrets-store-gke.csi.k8s.io). This volume references the SecretProviderClass object which acts as the provider.

Migrate from the existing Secrets Store CSI Driver

If you're migrating to the Secret Manager add-on from your existing installation of the Secrets Store CSI Driver, update your Pod manifest as follows:

1. Update the name of your SecretProviderClass and the provider as described in the following manifest:

   apiVersion: secrets-store.csi.x-k8s.io/v1
   kind: SecretProviderClass
   metadata:
     name: app-secrets-gke
   spec:
     provider: gke
     parameters:
       secrets: |
         - resourceName: "projects/<project_id>/secrets/<secret_name>/versions/<secret_version>"
           path: "good1.txt"
   

2. Update the driver and the secretProviderClass for your Kubernetes volume as described in the following manifest:

   volumes:
     - name: mysecret
       csi:
         driver: secrets-store-gke.csi.k8s.io
         readOnly: true
         volumeAttributes:
           secretProviderClass: "app-secrets-gke"
   

Disable the Secret Manager add-on

To disable the Secret Manager add-on on an existing Standard cluster or on an Autopilot cluster, run the following command:

Console

1. Go to the Google Kubernetes Engine page in the Google Cloud console.

   Go to Google Kubernetes Engine

2. In the cluster list, click the name of the cluster you want to modify.

3. On the cluster details page, in the Security section, click Secret Manager.

4. In the Edit Secret Manager dialog, clear the Enable Secret Manager checkbox.

5. Click Save changes.

gcloud

Before using any of the command data below, make the following replacements:

• CLUSTER_NAME: the name of your cluster
• REGION: the Compute Engine region for the cluster, such as us-central1

Execute the following command:

Linux, macOS, or Cloud Shell

Note: Ensure you have initialized the Google Cloud CLI with authentication and a project by running either gcloud init; or gcloud auth login and gcloud config set project.

gcloud container clusters update CLUSTER_NAME \
    --no-enable-secret-manager \
    --region=REGION \

Windows (PowerShell)

Note: Ensure you have initialized the Google Cloud CLI with authentication and a project by running either gcloud init; or gcloud auth login and gcloud config set project.

gcloud container clusters update CLUSTER_NAME `
    --no-enable-secret-manager `
    --region=REGION `

Windows (cmd.exe)

Note: Ensure you have initialized the Google Cloud CLI with authentication and a project by running either gcloud init; or gcloud auth login and gcloud config set project.

gcloud container clusters update CLUSTER_NAME ^
    --no-enable-secret-manager ^
    --region=REGION ^