Access secrets stored outside GKE clusters using Workload Identity

Autopilot Standard

---

This tutorial shows you how to store the sensitive data used by your Google Kubernetes Engine (GKE) clusters in Secret Manager, and more securely access the data from your Pods using Workload Identity. This tutorial is intended for security administrators who want to move sensitive data out of in-cluster storage.

Storing your sensitive data outside your cluster storage reduces the risk of unauthorized access to the data if an attack occurs. Using Workload Identity to access the data lets you avoid the risks associated with managing long-lived service account keys, and lets you control access to your secrets using Identity and Access Management (IAM) instead of in-cluster RBAC rules. You can use any external secret store provider, such as Secret Manager or HashiCorp Vault.

This tutorial uses a GKE Autopilot cluster. To perform the steps using GKE Standard, you must enable Workload Identity manually.

Objectives

• Create a secret in Google Cloud Secret Manager.
• Create and configure IAM service accounts to access the secret.
• Create a GKE Autopilot cluster, Kubernetes namespaces, and Kubernetes service accounts.
• Use test applications to verify service account access.
• Run a sample app that accesses the secret using the Secret Manager API.

Costs

This tutorial uses the following billable components of Google Cloud:

• GKE Autopilot
• Secret Manager

To generate a cost estimate based on your projected usage, use the pricing calculator. New Google Cloud users might be eligible for a free trial.

When you finish this tutorial, you can avoid continued billing by deleting the resources you created. For more information, see Clean up.

Before you begin

1. Sign in to your Google Cloud account. If you're new to Google Cloud, create an account to evaluate how our products perform in real-world scenarios. New customers also get $300 in free credits to run, test, and deploy workloads.
2. Install the Google Cloud CLI.

3. To initialize the gcloud CLI, run the following command:

   gcloud init

4. Create or select a Google Cloud project.

   Note: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project.

   • Create a Cloud project:

     gcloud projects create PROJECT_ID

   • Select the Cloud project that you created:

     gcloud config set project PROJECT_ID

5. Make sure that billing is enabled for your Cloud project. Learn how to check if billing is enabled on a project.

6. Enable the Kubernetes Engine and Secret Manager APIs:

   gcloud services enable container.googleapis.com secretmanager.googleapis.com

7. Install the Google Cloud CLI.

8. To initialize the gcloud CLI, run the following command:

   gcloud init

9. Create or select a Google Cloud project.

   Note: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project.

   • Create a Cloud project:

     gcloud projects create PROJECT_ID

   • Select the Cloud project that you created:

     gcloud config set project PROJECT_ID

10. Make sure that billing is enabled for your Cloud project. Learn how to check if billing is enabled on a project.

11. Enable the Kubernetes Engine and Secret Manager APIs:

    gcloud services enable container.googleapis.com secretmanager.googleapis.com

12. Grant roles to your Google Account. Run the following command once for each of the following IAM roles: roles/secretmanager.admin, roles/container.clusterAdmin, roles/iam.serviceAccountAdmin

    gcloud projects add-iam-policy-binding PROJECT_ID --member="user:EMAIL_ADDRESS" --role=ROLE

    • Replace PROJECT_ID with your project ID.
    • Replace EMAIL_ADDRESS with your email address.
    • Replace ROLE with each individual role.

Prepare the environment

Clone the GitHub repository that contains the sample files for this tutorial:

git clone https://github.com/GoogleCloudPlatform/kubernetes-engine-samples
cd ~/kubernetes-engine-samples/security/wi-secrets

Create a secret in Secret Manager

1. The following example shows the data you'll use to create a secret:

   key=my-api-key

2. Create a secret to store the sample data:

   gcloud secrets create bq-readonly-key \
       --data-file=manifests/bq-readonly-key \
       --ttl=3600s
   

   This command does the following:

   • Creates a new Secret Manager secret with the sample key in the us-central1 Google Cloud region.
   • Sets the secret to expire one hour after you run the command.

Configure IAM service accounts

1. Create two new IAM service accounts for read-only access and read-write access:

   gcloud iam service-accounts create readonly-secrets --display-name="Read secrets"
   gcloud iam service-accounts create readwrite-secrets --display-name="Read write secrets"
   

2. Grant the readonly-secrets IAM service account read-only access to the secret:

   gcloud secrets add-iam-policy-binding bq-readonly-key \
       --member=serviceAccount:readonly-secrets@PROJECT_ID.iam.gserviceaccount.com \
       --role='roles/secretmanager.secretAccessor'
   

3. Grant the readwrite-secrets IAM service accounts read-write access to the secret:

   gcloud secrets add-iam-policy-binding bq-readonly-key \
       --member=serviceAccount:readwrite-secrets@PROJECT_ID.iam.gserviceaccount.com \
       --role='roles/secretmanager.secretAccessor'
   gcloud secrets add-iam-policy-binding bq-readonly-key \
       --member=serviceAccount:readwrite-secrets@PROJECT_ID.iam.gserviceaccount.com \
       --role='roles/secretmanager.secretVersionAdder'
   

Create the cluster and Kubernetes resources

Create a GKE cluster, Kubernetes namespaces, and Kubernetes service accounts. You create two namespaces, one for read-only access and one for read-write access to the secret. You also create a Kubernetes service account in each namespace to use with Workload Identity.

1. Create a GKE Autopilot cluster:

   gcloud container clusters create-auto secret-cluster \
       --region=us-central1
   

   The cluster might take about five minutes to deploy. Autopilot clusters always have Workload Identity enabled. If you want to use a GKE Standard cluster instead, you must manually enable Workload Identity before you continue.

2. Create a readonly-ns namespace and an admin-ns namespace:

   kubectl create namespace readonly-ns
   kubectl create namespace admin-ns
   

3. Create a readonly-sa Kubernetes service account and an admin-sa Kubernetes service account:

   kubectl create serviceaccount readonly-sa --namespace=readonly-ns
   kubectl create serviceaccount admin-sa --namespace=admin-ns
   

4. Bind the IAM service accounts to the Kubernetes service accounts:

   gcloud iam service-accounts add-iam-policy-binding readonly-secrets@PROJECT_ID.iam.gserviceaccount.com \
       --member=serviceAccount:PROJECT_ID.svc.id.goog[readonly-ns/readonly-sa] \
       --role='roles/iam.workloadIdentityUser'
   gcloud iam service-accounts add-iam-policy-binding readwrite-secrets@PROJECT_ID.iam.gserviceaccount.com \
       --member=serviceAccount:PROJECT_ID.svc.id.goog[admin-ns/admin-sa] \
       --role='roles/iam.workloadIdentityUser'
   

5. Annotate the Kubernetes service accounts with the names of the bound IAM service accounts:

   kubectl annotate serviceaccount readonly-sa \
       --namespace=readonly-ns \
       iam.gke.io/gcp-service-account=readonly-secrets@PROJECT_ID.iam.gserviceaccount.com
   kubectl annotate serviceaccount admin-sa \
       --namespace=admin-ns \
       iam.gke.io/gcp-service-account=readwrite-secrets@PROJECT_ID.iam.gserviceaccount.com
   

You now have a cluster that's configured to access the secret from Pods using Workload Identity.

Verify secret access

Deploy test Pods in each namespace to verify the read-only and read-write access.

1. Review the read-only Pod manifest:

   # Copyright 2022 Google LLC
   #
   # Licensed under the Apache License, Version 2.0 (the "License");
   # you may not use this file except in compliance with the License.
   # You may obtain a copy of the License at
   #
   #     http://www.apache.org/licenses/LICENSE-2.0
   #
   # Unless required by applicable law or agreed to in writing, software
   # distributed under the License is distributed on an "AS IS" BASIS,
   # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
   # See the License for the specific language governing permissions and
   # limitations under the License.
   
   apiVersion: v1
   kind: Pod
   metadata:
     name: readonly-test
     namespace: readonly-ns
   spec:
     containers:
     - image: google/cloud-sdk:slim
       name: workload-identity-test
       command: ["sleep","infinity"]
       resources:
         requests:
           cpu: "150m"
           memory: "150Mi"
     serviceAccountName: readonly-sa

   This Pod uses the readonly-sa service account in the readonly-ns namespace.

2. Review the read-write Pod manifest:

   # Copyright 2022 Google LLC
   #
   # Licensed under the Apache License, Version 2.0 (the "License");
   # you may not use this file except in compliance with the License.
   # You may obtain a copy of the License at
   #
   #     http://www.apache.org/licenses/LICENSE-2.0
   #
   # Unless required by applicable law or agreed to in writing, software
   # distributed under the License is distributed on an "AS IS" BASIS,
   # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
   # See the License for the specific language governing permissions and
   # limitations under the License.
   
   apiVersion: v1
   kind: Pod
   metadata:
     name: admin-test
     namespace: admin-ns
   spec:
     containers:
     - image: google/cloud-sdk:slim
       name: workload-identity-test
       command: ["sleep","infinity"]
       resources:
         requests:
           cpu: "150m"
           memory: "150Mi"
     serviceAccountName: admin-sa

   This Pod uses the admin-sa service account in the admin-ns namespace.

3. Deploy the test Pods:

   kubectl apply -f manifests/admin-pod.yaml
   kubectl apply -f manifests/readonly-pod.yaml
   

   The Pods might take a few minutes to start running. To monitor progress, run the following command:

   watch kubectl get pods -n readonly-ns
   

   When the Pod status changes to RUNNING, press Ctrl+C to return to the command-line.

Test read-only access

1. Open a shell in the readonly-test Pod:

   kubectl exec -it readonly-test --namespace=readonly-ns -- /bin/bash
   

2. Try to read the secret:

   gcloud secrets versions access 1 --secret=bq-readonly-key
   

   The output is key=my-api-key.

3. Try to write new data to the secret:

   printf "my-second-api-key" | gcloud secrets versions add bq-readonly-key --data-file=-
   

   The output is similar to the following:

   ERROR: (gcloud.secrets.versions.add) PERMISSION_DENIED: Permission 'secretmanager.versions.add' denied for resource 'projects/PROJECT_ID/secrets/bq-readonly-key' (or it may not exist).
   

   The Pod using the read-only service account can only read the secret, and can't write new data.

4. Exit the Pod:

   exit
   

Test read-write access

1. Open a shell in the admin-test Pod:

   kubectl exec -it admin-test --namespace=admin-ns -- /bin/bash
   

2. Try to read the secret:

   gcloud secrets versions access 1 --secret=bq-readonly-key
   

   The output is key=my-api-key.

3. Try to write new data to the secret:

   printf "my-second-api-key" | gcloud secrets versions add bq-readonly-key --data-file=-
   

   The output is similar to the following:

   Created version [2] of the secret [bq-readonly-key].
   

4. Read the new secret version:

   gcloud secrets versions access 2 --secret=bq-readonly-key
   

   The output is my-second-api-key.

5. Exit the Pod:

   exit
   

The Pods only get the level of access you granted to the IAM service account that is bound to the Kubernetes service account used in the Pod manifest. Any Pods that use the admin-sa Kubernetes account in the admin-ns namespace can write new versions of the secret, but any Pods in the readonly-ns namespace that use the readonly-sa Kubernetes service account can only read the secret.

Access secrets from your code

In this section, you do the following:

1. Deploy a sample application that reads your secret in Secret Manager using client libraries.

2. Check that the application can access your secret.

You should access Secret Manager secrets from your application code whenever possible, using the Secret Manager API.

1. Review the sample application source code:

   security/wi-secrets/main.go

   View on GitHub

   // Copyright 2022 Google LLC
   //
   // Licensed under the Apache License, Version 2.0 (the "License");
   // you may not use this file except in compliance with the License.
   // You may obtain a copy of the License at
   //
   //     http://www.apache.org/licenses/LICENSE-2.0
   //
   // Unless required by applicable law or agreed to in writing, software
   // distributed under the License is distributed on an "AS IS" BASIS,
   // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
   // See the License for the specific language governing permissions and
   // limitations under the License.
   
   package main
   
   import (
   	"context"
   	"fmt"
   	"log"
   	"os"
   
   	secretmanager "cloud.google.com/go/secretmanager/apiv1"
   	secretmanagerpb "google.golang.org/genproto/googleapis/cloud/secretmanager/v1"
   )
   
   func main() {
   
           // Get environment variables from Pod spec.
           projectID := os.Getenv("PROJECT_ID")
           secretId := os.Getenv("SECRET_ID")
           secretVersion := os.Getenv("SECRET_VERSION")
   
           // Create the Secret Manager client.
           ctx := context.Background()
           client, err := secretmanager.NewClient(ctx)
           if err != nil {
                   log.Fatalf("failed to setup client: %v", err)
           }
           defer client.Close()
   
           // Create the request to access the secret.
           accessSecretReq := &secretmanagerpb.AccessSecretVersionRequest{
                   Name: fmt.Sprintf("projects/%s/secrets/%s/versions/%s", projectID, secretId, secretVersion),
           }
   
           secret, err := client.AccessSecretVersion(ctx, accessSecretReq)
           if err != nil {
                   log.Fatalf("failed to access secret: %v", err)
           }
   
           // Print the secret payload.
           //
           // WARNING: Do not print the secret in a production environment - this
           // snippet is showing how to access the secret material.
           log.Printf("Welcome to the key store, here's your key:\nKey: %s", secret.Payload.Data)
   }
   

   This application calls the Secret Manager API to try and read the secret.

2. Review the sample application Pod manifest:

   security/wi-secrets/manifests/secret-app.yaml

   View on GitHub

   # Copyright 2022 Google LLC
   #
   # Licensed under the Apache License, Version 2.0 (the "License");
   # you may not use this file except in compliance with the License.
   # You may obtain a copy of the License at
   #
   #     http://www.apache.org/licenses/LICENSE-2.0
   #
   # Unless required by applicable law or agreed to in writing, software
   # distributed under the License is distributed on an "AS IS" BASIS,
   # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
   # See the License for the specific language governing permissions and
   # limitations under the License.
   
   apiVersion: v1
   kind: Pod
   metadata:
     name: readonly-secret-test
     namespace: readonly-ns
   spec:
     containers:
     - image: us-docker.pkg.dev/google-samples/containers/gke/wi-secret-store:latest
       name: secret-app
       env:
         - name: PROJECT_ID
           value: "YOUR_PROJECT_ID"
         - name: SECRET_ID
           value: "bq-readonly-key"
         - name: SECRET_VERSION
           value: "latest"
       resources:
         requests:
           cpu: "125m"
           memory: "64Mi"
     serviceAccountName: readonly-sa

   This manifest does the following:

   • Creates a Pod in the readonly-ns namespace that uses the readonly-sa service account.
   • Pulls a sample application from a Google image registry. This application calls the Secret Manager API using the Google Cloud client libraries. You can view the application code in /main.go in the repository.
   • Sets environment variables for the sample application to use.

3. Replace environment variables in the sample application:

   sed -i "s/YOUR_PROJECT_ID/PROJECT_ID/g" "manifests/secret-app.yaml"
   

4. Deploy the sample app:

   kubectl apply -f manifests/secret-app.yaml
   

   The Pod might take a few minutes to start working. If the Pod needs a new node in your cluster, you might notice CrashLoopBackOff type events while GKE provisions the node. The crashes stop when the node provisions successfully.

5. Verify the secret access:

   kubectl logs readonly-secret-test -n readonly-ns
   

   The output is my-second-api-key. If the output is blank, the Pod might not be running yet. Wait a few minutes and try again.

Alternative approaches

If you need to mount secrets to support third-party workloads running in your cluster, consider using the Google Secret Manager provider for the Kubernetes Secret Store CSI driver. This is a Google-provided DaemonSet that runs in your cluster and mounts your Secret Manager secrets to your Pods.

If you want to run this DaemonSet, you must use a GKE Standard cluster. GKE Autopilot clusters don't let you deploy workloads in the kube-system namespace.

Providing secrets as mounted volumes has the following risks:

1. Mounted volumes are susceptible to directory traversal attacks.
2. Environment variables can be compromised due to misconfigurations such as opening a debug endpoint.

Whenever possible, we recommend that you programmatically access secrets through the Secret Manager API. For instructions, use the sample application in this tutorial or refer to Secret Manager client libraries.

Clean up

To avoid incurring charges to your Google Cloud account for the resources used in this tutorial, either delete the project that contains the resources, or keep the project and delete the individual resources.

Delete individual resources

1. Delete the cluster:

   gcloud container clusters delete secret-cluster \
       --region=us-central1
   

2. Delete the IAM service accounts:

   gcloud iam service-accounts delete readonly-secrets@PROJECT_ID.iam.gserviceaccount.com
   gcloud iam service-accounts delete readwrite-secrets@PROJECT_ID.iam.gserviceaccount.com
   

3. Optional: Delete the secret in Secret Manager:

   gcloud secrets delete bq-readonly-key
   

   If you don't do this step, the secret automatically expires because you set the --ttl flag during creation.

Delete the project

Caution: Deleting a project has the following effects:

• Everything in the project is deleted. If you used an existing project for this tutorial, when you delete it, you also delete any other work you've done in the project.
• Custom project IDs are lost. When you created this project, you might have created a custom project ID that you want to use in the future. To preserve the URLs that use the project ID, such as an appspot.com URL, delete selected resources inside the project instead of deleting the whole project.

If you plan to explore multiple tutorials and quickstarts, reusing projects can help you avoid exceeding project quota limits.

Delete a Cloud project:

gcloud projects delete PROJECT_ID

What's next

• Learn more about how Workload Identity works.
• Read a community tutorial on using Workload Identity with HashiCorp Vault.
• Explore reference architectures, diagrams, and best practices about Google Cloud. Take a look at our Cloud Architecture Center.