Using Workload Identity

This page explains the recommended way for your Google Kubernetes Engine (GKE) applications to consume services provided by Google APIs.

Overview

Workload Identity is the recommended way to access Google Cloud services from applications running within GKE due to its improved security properties and manageability. For information about alternative ways to access Google Cloud APIs from GKE, refer to the alternatives section below.

Fleet workload identity provides workload identity federation support for clusters registered in fleets, including Anthos clusters.

Terminology

This document distinguishes between Kubernetes service accounts and Google service accounts. Kubernetes service accounts are Kubernetes resources, while Google service accounts are specific to Google Cloud. Other Google Cloud documentation refers to Google service accounts as "service accounts".

Concepts

Applications running on GKE must authenticate to use Google Cloud APIs such as the Compute APIs, Storage and Database APIs, or Machine Learning APIs.

With Workload Identity, you can configure a Kubernetes service account to act as a Google service account. Pods running as the Kubernetes service account will automatically authenticate as the Google service account when accessing Google Cloud APIs. This enables you to assign distinct, fine-grained identities and authorization for each application in your cluster.

To accomplish a secure mapping between Kubernetes service accounts and Google service accounts, Workload Identity introduces the concept of a cluster's workload identity pool, which allows Identity and Access Management (IAM) to trust and understand Kubernetes service account credentials.

When you enable Workload Identity on your GKE cluster, the cluster's workload identity pool will be set to PROJECT_ID.svc.id.goog. This lets IAM authenticate Kubernetes service accounts as the following member name:

serviceAccount:PROJECT_ID.svc.id.goog[K8S_NAMESPACE/KSA_NAME]

In this member name:

PROJECT_ID.svc.id.goog is the workload identity pool set on the cluster.
KSA_NAME is the name of the Kubernetes service account making the request.
K8S_NAMESPACE is the Kubernetes namespace where the Kubernetes service account is defined.

GKE automatically creates a fixed workload identity pool, PROJECT_ID.svc.id.goog, for each Google Cloud project. This workload identity pool is used for all GKE clusters in the project.

Identity sameness across clusters

All Kubernetes service accounts that share a name, namespace name, and workload identity pool resolve to the same member name, and therefore share access to Google Cloud resources. This common identity makes it possible to give the applications within a workload identity pool access to an external resource once rather than cluster-by-cluster.

To illustrate this point further, consider the following example. Clusters A, B, and C are enrolled in the same workload identity pool. When applications in the backend namespace access Google Cloud resources, their identities are mapped to a common Google service account called back, regardless of which cluster is hosting the application. The Google service account back can be authorized on any number of Google Cloud APIs, from Cloud Storage to Cloud SQL.

Because of identity sameness, it is important that all clusters in a workload identity pool are trusted and well-governed. Revisiting the previous example, if Cluster C is owned by a separate, untrusted team, they too can create a backend namespace and access Google Cloud APIs as if they were the backend in Cluster A or B.

Diagram illustrating identity sameness within a workload identity pool — Identity sameness accessing Google Cloud APIs with Workload Identity

To prevent clusters from sharing permissions, the clusters must be in separate projects or they must use distinct Kubernetes namespace names. As a specific example: users with permissive "dev" and locked down "prod" clusters should consider separating those clusters into separate projects to get separate workload identity pools.

Limitations

Currently, there is only one fixed workload identity pool per Google Cloud project, PROJECT_ID.svc.id.goog, and it is automatically created for you.
Workload Identity replaces the need to use Metadata Concealment and as such, the two approaches are incompatible. The sensitive metadata protected by Metadata Concealment is also protected by Workload Identity.
When the GKE metadata server is enabled on a node pool, Pods will no longer be able to access the Compute Engine metadata server. Instead, requests made from these Pods to the Metadata APIs will be routed to the GKE metadata server. The one exception to this is Pods running on the host network (see next item).
Workload Identity can't be used by Pods running on the host network. Requests made from these Pods to the Metadata APIs will be routed to the Compute Engine metadata server.
The GKE metadata server needs a few seconds before it can start accepting requests on a newly created Pod. Therefore, attempts to authenticate using Workload Identity within the first few seconds of a Pod's life might fail. Retrying the call will resolve the problem. See the Troubleshooting section for more details.
GKE built-in logging and monitoring agents will continue to use the node's service account.
Workload Identity requires manual setup for Cloud Run for Anthos to continue releasing request metrics.
Workload Identity installs ip-masq-agent if the cluster is created without the --disable-default-snat flag.
Workload Identity sets a limit of 200 connections to the GKE metadata server for each node to avoid memory issues. You may experience timeouts if your nodes exceed this limit.

Before you begin

Before you start, make sure you have performed the following tasks:

Ensure that you have enabled the Google Kubernetes Engine API.

Enable Google Kubernetes Engine API

Ensure that you have installed the Cloud SDK.

Set up default gcloud settings using one of the following methods:

Using gcloud init, if you want to be walked through setting defaults.
Using gcloud config, to individually set your project ID, zone, and region.

Using gcloud init

If you receive the error One of [--zone, --region] must be supplied: Please specify location, complete this section.

Run gcloud init and follow the directions:
```
gcloud init
```
If you are using SSH on a remote server, use the --console-only flag to prevent the command from launching a browser:
```
gcloud init --console-only
```
Follow the instructions to authorize gcloud to use your Google Cloud account.
Create a new configuration or select an existing one.
Choose a Google Cloud project.
Choose a default Compute Engine zone for zonal clusters or a region for regional or Autopilot clusters.

Note: You can override these default settings in gcloud commands using the --project, --zone, and --region flags.

Using gcloud config

Set your default project ID:
```
gcloud config set project PROJECT_ID
```
If you are working with zonal clusters, set your default compute zone:
```
gcloud config set compute/zone COMPUTE_ZONE
```
If you are working with Autopilot or regional clusters, set your default compute region:
```
gcloud config set compute/region COMPUTE_REGION
```
Update gcloud to the latest version:
```
gcloud components update
```

Note: You can override these default settings in gcloud commands using the --project, --zone, and --region flags.

Ensure that you have the following IAM roles:
- roles/container.admin
- roles/iam.serviceAccountAdmin

Enabling Workload Identity on a cluster

You can enable Workload Identity on a new or existing GKE Standard cluster by using the gcloud tool. By default, Workload Identity is enabled on GKE Autopilot clusters.

Ensure that you have enabled the IAM Service Account Credentials API.

Enable IAM Credentials API
To create a new cluster with Workload Identity enabled, use the following command:
```
gcloud container clusters create CLUSTER_NAME \
    --workload-pool=PROJECT_ID.svc.id.goog
```
Replace the following:
- CLUSTER_NAME: the name of your cluster.
- PROJECT_ID: the ID of your Google Cloud project.
To enable Workload Identity on an existing cluster, modify the cluster with the following command:
```
gcloud container clusters update CLUSTER_NAME \
    --workload-pool=PROJECT_ID.svc.id.goog
```
Existing node pools are unaffected; new node pools default to --workload-metadata=GKE_METADATA.

Migrate applications to Workload Identity

Select the migration strategy that is ideal for your environment. Node pools can be migrated in place or you can create new node pools with Workload Identity enabled. We recommend creating new node pools if you also need to modify your application to be compatible with this feature.

Option 1: Node pool creation with Workload Identity (recommended)

Add a new node pool to the cluster with Workload Identity enabled and manually migrate workloads to that pool. This succeeds only if Workload Identity is enabled on the cluster.

gcloud container node-pools create NODEPOOL_NAME \
    --cluster=CLUSTER_NAME \
    --workload-metadata=GKE_METADATA

If a cluster has Workload Identity enabled, you can selectively disable it on a specific node pool by explicitly specifying --workload-metadata=GCE_METADATA. See Protecting cluster metadata for more information.

Option 2: Node pool modification

Modify an existing node pool to enable GKE_METADATA. This update succeeds only if Workload Identity is enabled on the cluster. It immediately enables Workload Identity for workloads deployed to the node pool. This change will prevent workloads from using the Compute Engine service account and must be carefully rolled out.

gcloud container node-pools update NODEPOOL_NAME \
    --cluster=CLUSTER_NAME \
    --workload-metadata=GKE_METADATA

Authenticating to Google Cloud

This section explains how an application can authenticate to Google Cloud using Workload Identity. To do this, assign a Kubernetes service account to the application and configure it to act as a Google service account:

Configure kubectl to communicate with the cluster:
```
gcloud container clusters get-credentials CLUSTER_NAME
```
Replace CLUSTER_NAME with the name of the cluster you created in the previous step.

This action requires container.clusters.get permission on the project.
Like most other resources, Kubernetes service accounts live in a namespace. Create the namespace to use for the Kubernetes service account.
```
kubectl create namespace K8S_NAMESPACE
```
Create the Kubernetes service account to use for your application:
```
kubectl create serviceaccount --namespace K8S_NAMESPACE KSA_NAME
```
Replace the following:
- K8S_NAMESPACE: the name of the Kubernetes namespace you created in the previous step.
- KSA_NAME: the name you would like to use for the Kubernetes service account.
Alternatively, you can use the default Kubernetes service account in the default namespace or any other existing namespace.
Configure your application to use the Kubernetes service account:

Note: You can skip this step if you're using the default Kubernetes service account.
```
spec:
  serviceAccountName: KSA_NAME
```
Create a Google service account for your application or use an existing Google service account instead. You can use any Google service account in any project in your organization. For Config Connector, apply the IAMServiceAccount object for your chosen service account.
gcloud
To create a new Google service account using the gcloud tool, run the following command.

Note: If you're using an existing Google service account with the gcloud tool, skip this step.
```
gcloud iam service-accounts create GSA_NAME
```
Replace GSA_NAME with the name of the new Google service account.
Config Connector
To use a new or existing Google service account with Config Connector, apply the following configuration file.

Note: You must apply the IAMServiceAccount object to the cluster using kubectl even if you use an existing Google service account. Replace GSA_NAME in the configuration file with the name of the service account to use.

Note: This step requires Config Connector. Follow the installation instructions to install Config Connector on your cluster.

View on GitHub
```
apiVersion: iam.cnrm.cloud.google.com/v1beta1
kind: IAMServiceAccount
metadata:
  name: [GSA_NAME]
spec:
  displayName: [GSA_NAME]
```
To deploy this manifest, download it to your machine as service-account.yaml.

Use kubectl to apply the manifest:
```
kubectl apply -f service-account.yaml
```
For information on authorizing Google service accounts to access Google Cloud APIs, see Understanding service accounts.
Ensure that your Google service account has the IAM roles you need. You can grant additional roles using the following command:
```
gcloud projects add-iam-policy-binding PROJECT_ID \
    --member "serviceAccount:GSA_NAME@PROJECT_ID.iam.gserviceaccount.com" \
    --role "ROLE_NAME"
```
Replace the following:
- PROJECT_ID: your Google Cloud project ID.
- GSA_NAME: the name of your Google service account.
- ROLE_NAME: the IAM role to assign to your service account, like roles/spanner.viewer.

Allow the Kubernetes service account to impersonate the Google service account by creating an IAM policy binding between the two. This binding allows the Kubernetes Service account to act as the Google service account.

gcloud

gcloud iam service-accounts add-iam-policy-binding \
    --role roles/iam.workloadIdentityUser \
    --member "serviceAccount:PROJECT_ID.svc.id.goog[K8S_NAMESPACE/KSA_NAME]" \
    GSA_NAME@PROJECT_ID.iam.gserviceaccount.com

Config Connector

Note: This step requires Config Connector. Follow the installation instructions to install Config Connector on your cluster.

View on GitHub

apiVersion: iam.cnrm.cloud.google.com/v1beta1
kind: IAMPolicy
metadata:
  name: iampolicy-workload-identity-sample
spec:
  resourceRef:
    apiVersion: iam.cnrm.cloud.google.com/v1beta1
    kind: IAMServiceAccount
    name: [GSA_NAME]
  bindings:
    - role: roles/iam.workloadIdentityUser
      members:
        - serviceAccount:[PROJECT_ID].svc.id.goog[[K8S_NAMESPACE]/[KSA_NAME]]

To deploy this manifest, download it to your machine as policy-binding.yaml. Replace GSA_NAME, PROJECT_ID, K8S_NAMESPACE and KSA_NAME the values for your environment. Then, run:

kubectl apply -f policy-binding.yaml

Add the iam.gke.io/gcp-service-account=GSA_NAME@PROJECT_ID annotation to the Kubernetes service account, using the email address of the Google service account.
kubectl
```
kubectl annotate serviceaccount \
    --namespace K8S_NAMESPACE KSA_NAME \
    iam.gke.io/gcp-service-account=GSA_NAME@PROJECT_ID.iam.gserviceaccount.com
```
yaml
```
apiVersion: v1
kind: ServiceAccount
metadata:
  annotations:
    iam.gke.io/gcp-service-account: GSA_NAME@PROJECT_ID.iam.gserviceaccount.com
  name: KSA_NAME
  namespace: K8S_NAMESPACE
```
Note: This annotation by itself does not grant access to impersonate the Google service account. If the IAM binding does not exist, the Pod will not be able to use the Google service account.

Verify the service accounts are configured correctly by creating a Pod with the Kubernetes service account that runs the OS-specific container image, then connect to it with an interactive session.

For Linux nodes

Create a Pod with the Kubernetes service account that runs the cloud-sdk container image:

Save the following configuration as wi-test.yaml:

apiVersion: v1
kind: Pod
metadata:
  name: workload-identity-test
  namespace: K8S_NAMESPACE
spec:
  containers:
  - image: google/cloud-sdk:slim
    name: workload-identity-test
    command: ["sleep","infinity"]
  serviceAccountName: KSA_NAME

Create a Pod:

kubectl apply -f wi-test.yaml

Open an interactive session in the Pod:

kubectl exec -it workload-identity-test \
    --namespace K8S_NAMESPACE -- /bin/bash

The google/cloud-sdk image includes the gcloud command-line tool which is a convenient way to consume Google Cloud APIs. It might take some time to download the image.

You are now connected to an interactive shell within the created Pod. Run the following command inside the Pod:
```
curl -H "Metadata-Flavor: Google" http://169.254.169.254/computeMetadata/v1/instance/service-accounts/
```
If the service accounts are correctly configured, the IAM service account email address is listed as the active (and only) identity. This demonstrates that by default, the Pod acts as the IAM service account's authority when calling Google Cloud APIs.

For Windows Server nodes

Create a Pod with the Kubernetes service account that runs the servercore container image:

apiVersion: v1
kind: Pod
metadata:
  name: workload-identity-test
  namespace: K8S_NAMESPACE
spec:
  containers:
  - image: IMAGE_NAME
    name: workload-identity-test
    command: ["powershell.exe", "sleep", "3600"]
  serviceAccountName: KSA_NAME
  nodeSelector:
    kubernetes.io/os: windows
    cloud.google.com/gke-os-distribution: windows_ltsc

Replace IMAGE_NAME with one of the following container servercore image values:

Windows Server node image	Container `servercore` image
`WINDOWS_LTSC`, `WINDOWS_LTSC_CONTAINERD`	`mcr.microsoft.com/windows/servercore:ltsc2019`
`WINDOWS_SAC`, `WINDOWS_SAC_CONTAINERD`	Check the version mapping between the GKE node version and the Windows SAC version. For Windows Server version 1909, specify `mcr.microsoft.com/windows/servercore:1909`; otherwise, specify `mcr.microsoft.com/windows/servercore:20H2`.

Open an interactive session in the Pod:

kubectl exec -it workload-identity-test \
    --namespace K8S_NAMESPACE -- powershell

You are now connected to an interactive shell within the created Pod. Run the following powershell command inside the Pod:
```
Invoke-WebRequest  -Headers @{"Metadata-Flavor"="Google"} -Uri  http://169.254.169.254/computeMetadata/v1/instance/service-accounts/default/email  -UseBasicParsing
```
If the service accounts are correctly configured, the Google service account email address is listed as the active (and only) identity. This demonstrates that by default, the Pod uses the Google service account's authority when calling Google Cloud APIs.

Using Workload Identity from your code

Authenticating to Google Cloud services from your code is the same process as authenticating using the Compute Engine metadata server. When you use Workload Identity, your requests to the instance metadata server are routed to the GKE metadata server. Existing code that authenticates using the instance metadata server (like code using the Google Cloud client libraries) should work without modification.

Understanding the GKE metadata server

The GKE metadata server is a new metadata server designed for use with Kubernetes. The server runs as a daemonset with one Pod on each cluster Linux node or a native Windows service on each Windows node. The metadata server intercepts HTTP requests to http://metadata.google.internal (169.254.169.254:80), including requests like GET /computeMetadata/v1/instance/service-accounts/default/token to retrieve a token for the Google service account the Pod is configured to act as. Traffic to the metadata server never leaves the VM instance that hosts the Pod.

The GKE metadata server implements only a subset of Compute Engine metadata server endpoints which are relevant and safe for Kubernetes workloads:

/computeMetadata/v1/instance/attributes/cluster-location
/computeMetadata/v1/instance/attributes/cluster-name
/computeMetadata/v1/instance/attributes/cluster-uid
/computeMetadata/v1/instance/hostname
/computeMetadata/v1/instance/id
/computeMetadata/v1/project/numeric-project-id
/computeMetadata/v1/project/project-id
/computeMetadata/v1/instance/service-accounts
/computeMetadata/v1/instance/service-accounts/default
/computeMetadata/v1/instance/service-accounts/default/aliases
/computeMetadata/v1/instance/service-accounts/default/email
/computeMetadata/v1/instance/service-accounts/default/identity
/computeMetadata/v1/instance/service-accounts/default/identity?audience=audience
/computeMetadata/v1/instance/service-accounts/default/scopes
/computeMetadata/v1/instance/service-accounts/default/token
/computeMetadata/v1/instance/service-accounts/default/token?scopes=comma-separated-list-of-scopes

Revoking access

Revoke access to the Google service account using IAM:
gcloud
```
gcloud iam service-accounts remove-iam-policy-binding \
    --role roles/iam.workloadIdentityUser \
    --member "serviceAccount:PROJECT_ID.svc.id.goog[K8S_NAMESPACE/KSA_NAME]" \
    GSA_NAME@GSA_PROJECT_ID.iam.gserviceaccount.com
```
Replace the following:
- PROJECT_ID: the project ID container the GKE cluster.
- K8S_NAMESPACE: the name of the Kubernetes namespace where your Kubernetes service account is located.
- KSA_NAME: the name of the Kubernetes service account that will have its access revoked.
- GSA_NAME: the name of the Google service account.
- GSA_PROJECT_ID: the project ID containing the Google service account.
Config Connector
If you used Config Connector to create the service account, delete the service account with kubectl.
```
kubectl delete -f service-account.yaml
```
This action requires iam.serviceAccounts.setIamPolicy permissions on the service account.

It can take up to 30 minutes for cached tokens to expire. You can check whether the cached tokens have expired with this command:
```
gcloud auth list
```
The cached tokens have expired if the output of that command no longer includes GSA_NAME@PROJECT_ID.iam.gserviceaccount.com.
Remove the annotation from the Kubernetes service account. This step is optional because access has been revoked by IAM.
```
kubectl annotate serviceaccount KSA_NAME \
    --namespace K8S_NAMESPACE iam.gke.io/gcp-service-account-
```
Note: If you do not remove the annotation, the IAM service account you use with Workload Identity might continue to display when you run gcloud auth list. However, GKE does not use the IAM service account to authenticate to Google Cloud services.

Troubleshooting

Pod can't authenticate to Google Cloud

If your application can't authenticate to Google Cloud, ensure these settings are configured properly:

Ensure that you have enabled the IAM Service Account Credentials API in the project containing the GKE cluster.

Enable IAM Credentials API
Ensure that Workload Identity is enabled on the cluster by verifying that it has a workload identity pool set:
```
gcloud container clusters describe CLUSTER_NAME \
    --format="value(workloadIdentityConfig.workloadPool)"
```
If you haven't already specified a default zone or region for gcloud, you may also need to specify a --region or --zone flag when running this command.

Ensure that GKE metadata server (GKE_METADATA) is configured on the node pool where your application is running:

gcloud container node-pools describe NODEPOOL_NAME \
    --cluster=CLUSTER_NAME \
    --format="value(config.workloadMetadataConfig.mode)"

Ensure that the Kubernetes service account is annotated correctly

kubectl describe serviceaccount \
    --namespace K8S_NAMESPACE KSA_NAME

There should be an annotation in the following format:

iam.gke.io/gcp-service-account: GSA_NAME@PROJECT_ID.iam.gserviceaccount.com

Ensure the Google service account is configured correctly

gcloud iam service-accounts get-iam-policy \
    GSA_NAME@PROJECT_ID.iam.gserviceaccount.com

Verify that there is a binding with in the following format:

- members:
  - serviceAccount:PROJECT_ID.svc.id.goog[K8S_NAMESPACE/KSA_NAME]
  role: roles/iam.workloadIdentityUser

If you have a cluster network policy, ensure egress to 127.0.0.1/32 on port 988 is allowed:
```
kubectl describe networkpolicy NETWORK_POLICY_NAME
```

Timeout errors at Pod start up

The GKE metadata server needs a few seconds before it can start accepting requests on a newly created Pod. Therefore, attempts to authenticate using Workload Identity within the first few seconds of a Pod's life may fail for applications and Google Cloud client libraries configured with a short timeout.

If you encounter timeout errors, you can change the application code to wait a few seconds and retry. Alternatively, you can deploy an initContainer that waits until the GKE metadata server is ready before running the Pod's main container.

Here is a Pod with an example initContainer:

apiVersion: v1
kind: Pod
metadata:
  name: pod-with-initcontainer
spec:
  serviceAccountName: KSA_NAME
  initContainers:
  - image:  gcr.io/google.com/cloudsdktool/cloud-sdk:326.0.0-alpine
    name: workload-identity-initcontainer
    command:
    - '/bin/bash'
    - '-c'
    - |
      curl -s -H 'Metadata-Flavor: Google' 'http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/token' --retry 30 --retry-connrefused --retry-max-time 30 > /dev/null || exit 1
  containers:
  - image: gcr.io/your-project/your-image
    name: your-main-application-container

Workload Identity fails due to control plane unavailability

The metadata server is not able to return the Workload Identity when the cluster control plane is unavailable. Calls to the metadata server return status code 500.

The log entry might appear similar to the following in the Logs Explorer:

dial tcp 35.232.136.58:443: connect: connection refused

This will lead to expected unavailability of Workload Identity.

The control plane might be unavailable on zonal clusters on cluster maintenance like rotating IPs, upgrading control plane VMs, or resizing clusters or node pools. See more on Choosing a regional or zonal control plane to learn about control plane availability. Switching to regional cluster will eliminate this issue.

Workload Identity fails

If the GKE metadata server is blocked for any reason, Workload Identity will fail.

If you are using Istio, you should add the following application-level annotation to all workloads that use Workload Identity:

"traffic.sidecar.istio.io/excludeOutboundIPRanges=169.254.169.254/32"

Alternatively, you can change the global.proxy.excludeIPRanges Istio ConfigMap key to do the same thing.

Disabling Workload Identity on a cluster

You can only disable Workload Identity on GKE Standard clusters.

Disable Workload Identity on each node pool:

gcloud container node-pools update NODEPOOL_NAME \
    --cluster=CLUSTER_NAME \
    --workload-metadata=GCE_METADATA

Repeat this command for every node pool in the cluster.

Disable Workload Identity in the cluster:
```
gcloud container clusters update CLUSTER_NAME --disable-workload-identity
```
This action requires container.clusters.update permissions on the cluster.

Disabling Workload Identity in your organization

From a security perspective, Workload Identity allows GKE to assert Kubernetes service account identities that can be authenticated and authorized to Google Cloud resources. Administrators who have taken actions to isolate workloads from Google Cloud resources, like disabling service account creation or disabling service account key creation, might also wish to disable Workload Identity for your organization.

See these instructions for disabling Workload Identity for your organization.

Alternatives to Workload Identity

There are two alternative methods to access Cloud APIs from GKE. With the release of Workload Identity we no longer recommend these approaches because of the compromises they require.

Export service account keys and store them as Kubernetes Secrets. Google service account keys do not expire and require manual rotation. Exporting service account keys has the potential to expand the scope of a security breach if it goes undetected.
Use the Compute Engine service account of your nodes. You can run node pools as any IAM service account in your project. If you do not specify a service account during node pool creation, GKE will use the Compute Engine default service account of the project. The Compute Engine service account is shared by all workloads deployed on that node. This can result in over-provisioning of permissions.

What's next

Read the GKE security overview.
Learn about Pods.
Learn about Protecting Cluster Metadata.
Learn about Google service accounts.