Using Workload Identity

This page explains the recommended way for your Google Kubernetes Engine (GKE) applications to consume services provided by Google APIs.

Overview

Workload Identity is the recommended way to access Google Cloud services from applications running within GKE due to its improved security properties and manageability. For information about alternative ways to access Google Cloud APIs from GKE, refer to the alternatives section below.

Terminology

This document distinguishes between Kubernetes service accounts and Google service accounts. Kubernetes service accounts are Kubernetes resources, while Google service accounts are specific to Google Cloud. Other Google Cloud documentation refers to Google service accounts as "service accounts".

Concepts

Applications running on GKE must authenticate to use Google Cloud APIs such as the Compute APIs, Storage and Database APIs, or Machine Learning APIs.

With Workload Identity, you can configure a Kubernetes service account to act as a Google service account. Pods running as the Kubernetes service account will automatically authenticate as the Google service account when accessing Google Cloud APIs. This enables you to assign distinct, fine-grained identities and authorization for each application in your cluster.

To accomplish a secure mapping between Kubernetes service accounts and Google service accounts, Workload Identity introduces the concept of a cluster's workload identity pool, which allows Identity and Access Management (IAM) to trust and understand Kubernetes service account credentials.

When you enable Workload Identity on your GKE cluster, the cluster's workload identity pool will be set to PROJECT_ID.svc.id.goog. This lets IAM authenticate Kubernetes service accounts as the following member name:

serviceAccount:PROJECT_ID.svc.id.goog[K8S_NAMESPACE/KSA_NAME]

In this member name:

PROJECT_ID.svc.id.goog is the workload identity pool set on the cluster.
KSA_NAME is the name of the Kubernetes service account making the request.
K8S_NAMESPACE is the Kubernetes namespace where the Kubernetes service account is defined.

There is only one fixed workload identity pool per Google Cloud project, PROJECT_ID.svc.id.goog, and it is automatically created for you.

Identity sameness across clusters

All Kubernetes service accounts that share a name, namespace name, and workload identity pool resolve to the same member name, and therefore share access to Google Cloud resources. This common identity makes it possible to give the applications within a workload identity pool access to an external resource once rather than cluster-by-cluster.

To illustrate this point further, consider the following example. Clusters A, B, and C are enrolled in the same workload identity pool. When applications in the backend namespace access Google Cloud resources, their identities are mapped to a common Google service account called back, regardless of which cluster is hosting the application. The Google service account back can be authorized on any number of Google Cloud APIs, from Cloud Storage to Cloud SQL.

Because of identity sameness, it is important that all clusters in a workload identity pool are trusted and well-governed. Revisiting the previous example, if Cluster C is owned by a separate, untrusted team, they too can create a backend namespace and access Google Cloud APIs as if they were the backend in Cluster A or B.

Diagram illustrating identity sameness within a workload identity pool — Identity sameness accessing Google Cloud APIs with Workload Identity

To prevent clusters from sharing permissions, the clusters must be in separate projects or they must use distinct Kubernetes namespace names. As a specific example: users with permissive "dev" and locked down "prod" clusters should consider separating those clusters into separate projects to get separate workload identity pools.

Limitations

Currently, there is only one fixed workload identity pool per Google Cloud project, PROJECT_ID.svc.id.goog, and it is automatically created for you.
Currently, Workload Identity is not supported when a workload is running in Anthos clusters on VMware.
Workload Identity replaces the need to use Metadata Concealment and as such, the two approaches are incompatible. The sensitive metadata protected by Metadata Concealment is also protected by Workload Identity.
When the GKE metadata server is enabled on a node pool, Pods will no longer be able to access the Compute Engine metadata server. Instead, request made from these Pods to the Metadata APIs will be routed to the GKE metadata server. The one exception to this is Pods running on the host network (see next item).
Workload Identity can't be used by Pods running on the host network. Request made from these Pods to the Metadata APIs will be routed to the Compute Engine metadata server.
The GKE metadata server needs a few seconds before it can start accepting requests on a newly created Pod. Therefore, attempts to authenticate using Workload Identity within the first few seconds of a Pod's life might fail. Retrying the call will resolve the problem. See the Troubleshooting section for more details.
GKE built-in logging and monitoring agents will continue to use the node's service account.
Workload Identity requires manual setup for Cloud Run for Anthos on Google Cloud to continue releasing request metrics.
Workload Identity installs ip-masq-agent if the cluster is created without the --disable-default-snat flag.

Before you begin

Before you start, make sure you have performed the following tasks:

Ensure that you have enabled the Google Kubernetes Engine API.

Enable Google Kubernetes Engine API

Ensure that you have installed the Cloud SDK.

Set up default gcloud settings using one of the following methods:

Using gcloud init, if you want to be walked through setting defaults.
Using gcloud config, to individually set your project ID, zone, and region.

Using gcloud init

If you receive the error One of [--zone, --region] must be supplied: Please specify location, complete this section.

Run gcloud init and follow the directions:
```
gcloud init
```
If you are using SSH on a remote server, use the --console-only flag to prevent the command from launching a browser:
```
gcloud init --console-only
```
Follow the instructions to authorize gcloud to use your Google Cloud account.
Create a new configuration or select an existing one.
Choose a Google Cloud project.
Choose a default Compute Engine zone for zonal clusters or a region for regional or Autopilot clusters.

Note: You can override these default settings in gcloud commands using the --project, --zone, and --region flags.

Using gcloud config

Set your default project ID:
```
gcloud config set project PROJECT_ID
```
If you are working with zonal clusters, set your default compute zone:
```
gcloud config set compute/zone COMPUTE_ZONE
```
If you are working with Autopilot or regional clusters, set your default compute region:
```
gcloud config set compute/region COMPUTE_REGION
```
Update gcloud to the latest version:
```
gcloud components update
```

Note: You can override these default settings in gcloud commands using the --project, --zone, and --region flags.

Enabling Workload Identity on a cluster

You can enable Workload Identity on a new or existing GKE Standard cluster by using the gcloud tool. By default, Workload Identity is enabled on GKE Autopilot clusters.

Ensure that you have enabled the IAM Service Account Credentials API.

Enable IAM Credentials API
To create a new cluster with Workload Identity enabled, use the following command:
```
gcloud container clusters create CLUSTER_NAME \
  --workload-pool=PROJECT_ID.svc.id.goog
```
Replace the following:
- CLUSTER_NAME: the name of your cluster.
- PROJECT_ID: the ID of your Google Cloud project.
This action requires container.clusters.create permission on the project.
To enable Workload Identity on an existing cluster, modify the cluster with the following command:
```
gcloud container clusters update CLUSTER_NAME \
  --workload-pool=PROJECT_ID.svc.id.goog
```
Existing node pools are unaffected; new node pools default to --workload-metadata=GKE_METADATA.

This action requires container.clusters.update permissions on the cluster.

Migrate applications to Workload Identity

Select the migration strategy that is ideal for your environment. Node pools can be migrated in place or you can create new node pools with Workload Identity enabled. We recommend creating new node pools if you also need to modify your application to be compatible with this feature.

Option 1: Node pool creation with Workload Identity (recommended)

Add a new node pool to the cluster with Workload Identity enabled and manually migrate workloads to that pool. This succeeds only if Workload Identity is enabled on the cluster.

gcloud container node-pools create NODEPOOL_NAME \
  --cluster=CLUSTER_NAME \
  --workload-metadata=GKE_METADATA

If a cluster has Workload Identity enabled, you can selectively disable it on a specific node pool by explicitly specifying --workload-metadata=GCE_METADATA. See Protecting cluster metadata for more information.

Option 2: Node pool modification

Modify an existing node pool to enable GKE_METADATA. This update succeeds only if Workload Identity is enabled on the cluster. It immediately enables Workload Identity for workloads deployed to the node pool. This change will prevent workloads from using the Compute Engine service account and must be carefully rolled out.

gcloud container node-pools update NODEPOOL_NAME \
  --cluster=CLUSTER_NAME \
  --workload-metadata=GKE_METADATA

This action requires container.nodes.update permissions on the project.

Authenticating to Google Cloud

This section explains how an application can authenticate to Google Cloud using Workload Identity. To do this, assign a Kubernetes service account to the application and configure it to act as a Google service account:

Configure kubectl to communicate with the cluster:
```
gcloud container clusters get-credentials CLUSTER_NAME
```
Replace CLUSTER_NAME with the name of the cluster you created in the previous step.

This action requires container.clusters.get permission on the project.
Like most other resources, Kubernetes service accounts live in a namespace. Create the namespace to use for the Kubernetes service account.
```
kubectl create namespace K8S_NAMESPACE
```
This action requires Create namespace RBAC permission within the cluster.
Create the Kubernetes service account to use for your application:
```
kubectl create serviceaccount --namespace K8S_NAMESPACE KSA_NAME
```
Replace the following:
- K8S_NAMESPACE: the name of the Kubernetes namespace you created in the previous step.
- KSA_NAME: the name you would like to use for the Kubernetes service account.
This action requires create serviceaccounts RBAC permission within the namespace.

Alternatively, you can use the default namespace or the default Kubernetes service account in any namespace.
Create a Google service account for your application or use an existing Google service account instead. You can use any Google service account in any project in your organization. For Config Connector, apply the IAMServiceAccount object for your chosen service account.
gcloud
To create a new Google service account using the gcloud tool, run the following command.

Note: If you're using an existing Google service account with the gcloud tool, skip this step.
```
gcloud iam service-accounts create GSA_NAME
```
Replace GSA_NAME with the name of the new Google service account.
Config Connector
To use a new or existing Google service account with Config Connector, apply the following configuration file.

Note: You must apply the IAMServiceAccount object to the cluster using kubectl even if you use an existing Google service account. Replace GSA_NAME in the configuration file with the name of the service account to use.

Note: This step requires Config Connector. Follow the installation instructions to install Config Connector on your cluster.

View on GitHub
```
apiVersion: iam.cnrm.cloud.google.com/v1beta1
kind: IAMServiceAccount
metadata:
  name: [GSA_NAME]
spec:
  displayName: [GSA_NAME]
```
To deploy this manifest, download it to your machine as service-account.yaml.

Use kubectl to apply the manifest:
```
kubectl apply -f service-account.yaml
```
This action requires iam.serviceAccounts.create permission on the project.

For information on authorizing Google service accounts to access Google Cloud APIs, see Understanding service accounts.

Allow the Kubernetes service account to impersonate the Google service account by creating an IAM policy binding between the two. This binding allows the Kubernetes Service account to act as the Google service account.

gcloud

gcloud iam service-accounts add-iam-policy-binding \
  --role roles/iam.workloadIdentityUser \
  --member "serviceAccount:PROJECT_ID.svc.id.goog[K8S_NAMESPACE/KSA_NAME]" \
  GSA_NAME@PROJECT_ID.iam.gserviceaccount.com

Config Connector

Note: This step requires Config Connector. Follow the installation instructions to install Config Connector on your cluster.

View on GitHub

apiVersion: iam.cnrm.cloud.google.com/v1beta1
kind: IAMPolicy
metadata:
  name: iampolicy-workload-identity-sample
spec:
  resourceRef:
    apiVersion: iam.cnrm.cloud.google.com/v1beta1
    kind: IAMServiceAccount
    name: [GSA_NAME]
  bindings:
    - role: roles/iam.workloadIdentityUser
      members:
        - serviceAccount:[PROJECT_ID].svc.id.goog[[K8S_NAMESPACE]/[KSA_NAME]]

To deploy this manifest, download it to your machine as policy-binding.yaml. Replace GSA_NAME, PROJECT_ID, K8S_NAMESPACE and KSA_NAME the values for your environment. Then, run:

kubectl apply -f policy-binding.yaml

This action requires iam.serviceAccounts.setIamPolicy permission on the project.

Add the iam.gke.io/gcp-service-account=GSA_NAME@PROJECT_ID annotation to the Kubernetes service account, using the email address of the Google service account.
kubectl
```
kubectl annotate serviceaccount \
  --namespace K8S_NAMESPACE \
  KSA_NAME \
  iam.gke.io/gcp-service-account=GSA_NAME@PROJECT_ID.iam.gserviceaccount.com
```
yaml
```
apiVersion: v1
kind: ServiceAccount
metadata:
  annotations:
    iam.gke.io/gcp-service-account: GSA_NAME@PROJECT_ID.iam.gserviceaccount.com
  name: KSA_NAME
  namespace: K8S_NAMESPACE
```
This action requires RBAC edit permissions on the Kubernetes service account.

Note: This annotation by itself does not grant access to impersonate the Google service account. If the IAM binding does not exist, the Pod will not be able to use the Google service account.
Verify the service accounts are configured correctly by creating a Pod with the Kubernetes service account that runs the OS-specific container image, then connect to it with an interactive session.

For Linux nodes
1. Create a Pod with the Kubernetes service account that runs the cloud-sdk container image:
  kubectl
```
kubectl run -it \
  --image google/cloud-sdk:slim \
  --serviceaccount KSA_NAME \
  --namespace K8S_NAMESPACE \
  workload-identity-test
```
  yaml
  Save the following configuration as wi-test.yaml:
```
  apiVersion: v1
  kind: Pod
  metadata:
    name: workload-identity-test
    namespace: K8S_NAMESPACE
  spec:
    containers:
    - image: google/cloud-sdk:slim
      name: workload-identity-test
      command: ["sleep","infinity"]
    serviceAccountName: KSA_NAME
```
  Create a Pod:
```
  kubectl create -f wi-test.yaml
```
  Open an interactive session in the Pod:
```
  kubectl exec -it workload-identity-test \
    --namespace K8S_NAMESPACE \
    -- /bin/bash
```
  The google/cloud-sdk image includes the gcloud command-line tool which is a convenient way to consume Google Cloud APIs. It might take some time to download the image.
  
  This action requires create Pods RBAC permission within the namespace.
2. You are now connected to an interactive shell within the created Pod. Run the following command inside the Pod:
```
gcloud auth list
```
  If the service accounts are correctly configured, the Google service account email address is listed as the active (and only) identity. This demonstrates that by default, the Pod uses the Google service account's authority when calling Google Cloud APIs.
For Windows Server nodes
1. Create a Pod with the Kubernetes service account that runs the servercore container image:
```
kubectl run -it \
  --image IMAGE_NAME \
  --serviceaccount KSA_NAME \
  --namespace K8S_NAMESPACE \
  --overrides='{"spec":{"nodeSelector":{"kubernetes.io/os":"windows"}}}' \
  workload-identity-test
```
  Replace IMAGE_NAME with one of the following values:
  - For WINDOWS_LTSC nodes, use mcr.microsoft.com/windows/servercore:ltsc2019.
  - For WINDOWS_SAC nodes, use mcr.microsoft.com/windows/servercore:1909.
  This action requires create Pods RBAC permission within the namespace.
2. You are now connected to an interactive shell within the created Pod. Run the following powershell command inside the Pod:
```
Invoke-WebRequest  -Headers @{"Metadata-Flavor"="Google"} -Uri  http://169.254.169.254/computeMetadata/v1/instance/service-accounts/default/email  -UseBasicParsing
```
  If the service accounts are correctly configured, the Google service account email address is listed as the active (and only) identity. This demonstrates that by default, the Pod uses the Google service account's authority when calling Google Cloud APIs.

Using Workload Identity from your code

Authenticating to Google Cloud services from your code is the same process as authenticating using the Compute Engine metadata server. When you use Workload Identity, your requests to the instance metadata server are routed to the GKE metadata server. Existing code that authenticates using the instance metadata server (like code using the Google Cloud client libraries) should work without modification.

Understanding the GKE metadata server

The GKE metadata server is a new metadata server designed for use with Kubernetes. It runs as a daemonset , with one Pod on each cluster node. The metadata server intercepts HTTP requests to http://metadata.google.internal (169.254.169.254:80), including requests like GET /computeMetadata/v1/instance/service-accounts/default/token to retrieve a token for the Google service account the Pod is configured to act as. Traffic to the metadata server never leaves the VM instance that hosts the Pod.

The GKE metadata server implements only a subset of Compute Engine metadata server endpoints which are relevant and safe for Kubernetes workloads:

/computeMetadata/v1/instance/attributes/cluster-location
/computeMetadata/v1/instance/attributes/cluster-name
/computeMetadata/v1/instance/attributes/cluster-uid
/computeMetadata/v1/instance/hostname
/computeMetadata/v1/instance/id
/computeMetadata/v1/project/numeric-project-id
/computeMetadata/v1/project/project-id
/computeMetadata/v1/instance/service-accounts
/computeMetadata/v1/instance/service-accounts/default
/computeMetadata/v1/instance/service-accounts/default/aliases
/computeMetadata/v1/instance/service-accounts/default/email
/computeMetadata/v1/instance/service-accounts/default/identity
/computeMetadata/v1/instance/service-accounts/default/identity?audience=audience
/computeMetadata/v1/instance/service-accounts/default/scopes
/computeMetadata/v1/instance/service-accounts/default/token
/computeMetadata/v1/instance/service-accounts/default/token?scopes=comma-separated-list-of-scopes

Revoking access

Revoke access to the Google service account using IAM:
gcloud
```
gcloud iam service-accounts remove-iam-policy-binding \
  --role roles/iam.workloadIdentityUser \
  --member "serviceAccount:PROJECT_ID.svc.id.goog[K8S_NAMESPACE/KSA_NAME]" \
  GSA_NAME@GSA_PROJECT_ID.iam.gserviceaccount.com
```
Replace the following:
- PROJECT_ID: the project ID container the GKE cluster.
- K8S_NAMESPACE: the name of the Kubernetes namespace where your Kubernetes service account is located.
- KSA_NAME: the name of the Kubernetes service account that will have its access revoked.
- GSA_NAME: the name of the Google service account.
- GSA_PROJECT_ID: the project ID containing the Google service account.
Config Connector
If you used Config Connector to create the service account, delete the service account with kubectl.
```
kubectl delete -f service-account.yaml
```
This action requires iam.serviceAccounts.setIamPolicy permissions on the service account.

It can take up to 30 minutes for cached tokens to expire. You can check whether the cached tokens have expired with this command:
```
gcloud auth list
```
The cached tokens have expired if the output of that command no longer includes GSA_NAME@PROJECT_ID.iam.gserviceaccount.com.
Remove the annotation from the Kubernetes service account. This step is optional because access has been revoked by IAM.
```
kubectl annotate serviceaccount KSA_NAME \
  --namespace K8S_NAMESPACE \
  iam.gke.io/gcp-service-account-
```

Troubleshooting

Pod can't authenticate to Google Cloud

If your application can't authenticate to Google Cloud, ensure these settings are configured properly:

Ensure that you have enabled the IAM Service Account Credentials API in the project containing the GKE cluster.

Enable IAM Credentials API
Ensure that Workload Identity is enabled on the cluster by verifying that it has a workload identity pool set:
```
gcloud container clusters describe CLUSTER_NAME \
  --format="value(workloadIdentityConfig.workloadPool)"
```

Ensure that GKE metadata server (GKE_METADATA) is configured on the node pool where your application is running:

gcloud container node-pools describe NODEPOOL_NAME \
  --cluster=CLUSTER_NAME \
  --format="value(config.workloadMetadataConfig.mode)"

Ensure that the Kubernetes service account is annotated correctly

kubectl describe serviceaccount \
  --namespace K8S_NAMESPACE \
  KSA_NAME

There should be an annotation in the following format:

iam.gke.io/gcp-service-account: GSA_NAME@PROJECT_ID.iam.gserviceaccount.com

Ensure the Google service account is configured correctly

gcloud iam service-accounts get-iam-policy \
  GSA_NAME@PROJECT_ID.iam.gserviceaccount.com

Verify that there is a binding with in the following format:

- members:
  - serviceAccount:PROJECT_ID.svc.id.goog[K8S_NAMESPACE/KSA_NAME]
  role: roles/iam.workloadIdentityUser

If you have a cluster network policy , ensure egress to 127.0.0.1/32 on port 988 is allowed:
```
kubectl describe networkpolicy NETWORK_POLICY_NAME
```

Timeout errors at Pod start up

The GKE metadata server needs a few seconds before it can start accepting requests on a newly created Pod. Therefore, attempts to authenticate using Workload Identity within the first few seconds of a Pod's life may fail for applications and Google Cloud client libraries configured with a short timeout.

If you encounter timeout errors, you can change the application code to wait a few seconds and retry. Alternatively, you can deploy an initContainer that waits until the GKE metadata server is ready before running the Pod's main container.

Here is a Pod with an example initContainer:

apiVersion: v1
kind: Pod
metadata:
  name: pod-with-initcontainer
spec:
  serviceAccountName: ksa-name
  initContainers:
  - image:  gcr.io/google.com/cloudsdktool/cloud-sdk:326.0.0-alpine
    name: workload-identity-initcontainer
    command:
    - '/bin/bash'
    - '-c'
    - |
      curl -s -H 'Metadata-Flavor: Google' 'http://169.254.169.254/computeMetadata/v1/instance/service-accounts/default/token' --retry 30 --retry-connrefused --retry-max-time 30 > /dev/null || exit 1
  containers:
  - image: gcr.io/your-project/your-image
    name: your-main-application-container

Workload Identity fails

If the GKE metadata server is blocked for any reason, Workload Identity will fail.

If you are using Istio, you should add the following application-level annotation to all workloads that use Workload Identity:

"traffic.sidecar.istio.io/excludeOutboundIPRanges=169.254.169.254/32"

Alternatively, you can change the global.proxy.excludeIPRanges Istio ConfigMap key to do the same thing.

Disabling Workload Identity on a cluster

You can only disable Workload Identity on GKE Standard clusters.

Disable Workload Identity on each node pool:

gcloud container node-pools update NODEPOOL_NAME \
  --cluster=CLUSTER_NAME \
  --workload-metadata=GCE_METADATA

Repeat this command for every node pool in the cluster.

Disable Workload Identity in the cluster:
```
gcloud container clusters update CLUSTER_NAME \
  --disable-workload-identity
```
This action requires container.clusters.update permissions on the cluster.

Disabling Workload Identity in your organization

From a security perspective, Workload Identity allows GKE to assert Kubernetes service account identities that can be authenticated and authorized to Google Cloud resources. Administrators who have taken actions to isolate workloads from Google Cloud resources, like disabling service account creation or disabling service account key creation, might also wish to disable Workload Identity for your organization.

See these instructions for disabling Workload Identity for your organization.

Alternatives to Workload Identity

There are two alternative methods to access Cloud APIs from GKE. With the release of Workload Identity we no longer recommend these approaches because of the compromises they require.

Export service account keys and store them as Kubernetes Secrets. Google service account keys expire after 10 years and are rotated manually. Exporting service account keys has the potential to expand the scope of a security breach if it goes undetected.
Use the Compute Engine service account of your nodes. You can run node pools as any IAM service account in your project. If you do not specify a service account during node pool creation, GKE will use the Compute Engine default service account of the project. The Compute Engine service account is shared by all workloads deployed on that node. This can result in over-provisioning of permissions.

What's next

Read the GKE security overview.
Learn about Pods.
Learn about Protecting Cluster Metadata.
Learn about Google service accounts.