This document shows you how to enable and configure Workload Identity on your Google Kubernetes Engine (GKE) clusters. Workload Identity allows workloads in your GKE clusters to impersonate Identity and Access Management (IAM) service accounts to access Google Cloud services. Workload Identity is enabled by default on Autopilot clusters.
To learn more about how Workload Identity works, see Workload Identity.
Limitations
GKE creates a fixed workload identity pool for each Google Cloud project, with the format
PROJECT_ID.svc.id.goog
.Workload Identity replaces the need to use Metadata concealment. The sensitive metadata protected by metadata concealment is also protected by Workload Identity.
When GKE enables the GKE metadata server on a node pool, Pods can no longer access the Compute Engine metadata server. Instead, the GKE metadata server intercepts requests made from these pods to metadata endpoints, with the exception of Pods running on the host network.
Workload Identity can't be used by Pods running on the host network. Requests made from these pods to metadata endpoints are routed to the Compute Engine metadata server.
The GKE metadata server takes a few seconds to start accepting requests on a newly created Pod. Therefore, attempts to authenticate using Workload Identity within the first few seconds of a Pod's life might fail. Retrying the call will resolve the problem. See the Troubleshooting section for more details.
GKE built-in logging and monitoring agents continue to use the node's service account.
Workload Identity requires manual setup for Cloud Run for Anthos to continue releasing request metrics.
Workload Identity installs
ip-masq-agent
if the cluster is created without the--disable-default-snat
flag.Workload Identity sets a limit of 200 connections to the GKE metadata server for each node to avoid memory issues. You may experience timeouts if your nodes exceed this limit.
Workload Identity for Windows Server nodes is available in GKE versions 1.18.16-gke.1200, 1.19.8-gke.1300, 1.20.4-gke.1500 and later.
Before you begin
Before you start, make sure you have performed the following tasks:
- Ensure that you have enabled the Google Kubernetes Engine API. Enable Google Kubernetes Engine API
- Ensure that you have installed the Google Cloud CLI.
- Set up default Google Cloud CLI settings for your project by using one of the following methods:
- Use
gcloud init
, if you want to be walked through setting project defaults. - Use
gcloud config
, to individually set your project ID, zone, and region. -
Run
gcloud init
and follow the directions:gcloud init
If you are using SSH on a remote server, use the
--console-only
flag to prevent the command from launching a browser:gcloud init --console-only
- Follow the instructions to authorize the gcloud CLI to use your Google Cloud account.
- Create a new configuration or select an existing one.
- Choose a Google Cloud project.
- Choose a default Compute Engine zone.
- Choose a default Compute Engine region.
- Set your default project ID:
gcloud config set project PROJECT_ID
- Set your default Compute Engine region (for example,
us-central1
):gcloud config set compute/region COMPUTE_REGION
- Set your default Compute Engine zone (for example,
us-central1-c
):gcloud config set compute/zone COMPUTE_ZONE
- Update
gcloud
to the latest version:gcloud components update
gcloud init
gcloud config
By setting default locations, you can avoid errors in gcloud CLI like the
following: One of [--zone, --region] must be supplied: Please specify location
.
Ensure that you have enabled the IAM Service Account Credentials API.
Ensure that you have the following IAM roles:
roles/container.admin
roles/iam.serviceAccountAdmin
Enable Workload Identity
You can enable Workload Identity on clusters and node pools using the Google Cloud CLI or the Google Cloud console.
Workload Identity must be enabled at the cluster level before you can enable Workload Identity on node pools.
Create a new cluster
You can enable Workload Identity on a new Standard cluster by using the gcloud CLI or the Cloud console.
gcloud
To enable Workload Identity on a new cluster, run the following command:
gcloud container clusters create CLUSTER_NAME \
--region=COMPUTE_REGION \
--workload-pool=PROJECT_ID.svc.id.goog
Replace the following:
CLUSTER_NAME
: the name of your new cluster.COMPUTE_REGION
: the Compute Engine region of your cluster. For zonal clusters, use--zone=COMPUTE_ZONE
.PROJECT_ID
: your Google Cloud project ID.
Console
To enable Workload Identity on a new cluster, do the following:
Go to the Google Kubernetes Engine page in the Cloud console.
Click add_box Create.
In the Create cluster dialog, for GKE Standard, click Configure.
In the navigation menu, in the Cluster section, click Security.
Select the Enable Workload Identity checkbox.
Continue configuring the cluster, and then click Create.
Update an existing cluster
You can enable Workload Identity on an existing Standard cluster by using the gcloud CLI or the Cloud console. Existing node pools are unaffected, but any new node pools in the cluster use Workload Identity.
gcloud
To enable Workload Identity on an existing cluster, run the following command:
gcloud container clusters update CLUSTER_NAME \
--region=COMPUTE_REGION \
--workload-pool=PROJECT_ID.svc.id.goog
Replace the following:
CLUSTER_NAME
: the name of your existing cluster.COMPUTE_REGION
: the Compute Engine region of your cluster. For zonal clusters, use--zone=COMPUTE_ZONE
.PROJECT_ID
: your Google Cloud project ID.
Console
To enable Workload Identity on an existing cluster, do the following:
Go to the Google Kubernetes Engine page in Cloud console.
In the cluster list, click the name of the cluster you want to modify.
On the cluster details page, in the Security section, click
Edit Workload Identity.In the Edit Workload Identity dialog, select the Enable Workload Identity checkbox.
Click Save changes.
Migrate existing workloads to Workload Identity
After you enable Workload Identity on an existing cluster, you might want to migrate your running workloads to use Workload Identity. Select the migration strategy that is ideal for your environment. You can create new node pools with Workload Identity enabled, or update existing node pools to enable Workload Identity.
We recommend creating new node pools if you also need to modify your applications to be compatible with Workload Identity.
Create a new node pool
All new node pools that you create default to using Workload Identity if the cluster has Workload Identity enabled. To create a new node pool with Workload Identity enabled, run the following command:
gcloud container node-pools create NODEPOOL_NAME \
--cluster=CLUSTER_NAME \
--workload-metadata=GKE_METADATA
Replace the following:
NODEPOOL_NAME
: the name of the new node pool.CLUSTER_NAME
: the name of the existing cluster that has Workload Identity enabled.
The --workload-metadata=GKE_METADATA
flag configures the node pool to use
the GKE metadata server. We recommend that you include the flag
so that node pool creation fails if Workload Identity is not enabled on the
cluster.
Update an existing node pool
You can manually enable Workload Identity on existing node pools after you enable Workload Identity on the cluster.
gcloud
To modify an existing node pool to use Workload Identity, run the following command:
gcloud container node-pools update NODEPOOL_NAME \
--cluster=CLUSTER_NAME \
--workload-metadata=GKE_METADATA
If a cluster has Workload Identity enabled, you can selectively disable it on a
specific node pool by explicitly specifying
--workload-metadata=GCE_METADATA
. See Protecting cluster
metadata for more
information.
Console
To modify an existing node pool to use Workload Identity, perform the following steps:
Go to the Google Kubernetes Engine page in Cloud console.
In the cluster list, click the name of the cluster that you want to modify.
Click the Nodes tab.
In the Node Pools section, click the name of the node pool that you want to modify.
On the Node pool details page, click
Edit.On the Edit node pool page, in the Security section, select the Enable GKE Metadata Server checkbox.
Click Save.
Configure applications to use Workload Identity
After enabling Workload Identity, you should configure your applications to authenticate to Google Cloud using Workload Identity before you migrate the applications to the new node pools.
You must assign a Kubernetes service account to the application and configure that Kubernetes service account to act as an IAM service account.
The following steps show you how to configure your applications to use Workload Identity if it is enabled on the cluster.
Get credentials for your cluster:
gcloud container clusters get-credentials CLUSTER_NAME
Replace
CLUSTER_NAME
with the name of your cluster that has Workload Identity enabled.Create a namespace to use for the Kubernetes service account. You can also use the default namespace or any existing namespace.
kubectl create namespace NAMESPACE
Create a Kubernetes service account for your application to use. You can also use the default Kubernetes service account in the default or any existing namespace.
kubectl create serviceaccount KSA_NAME \ --namespace NAMESPACE
Replace the following:
KSA_NAME
: the name of your new Kubernetes service account.NAMESPACE
: the name of the Kubernetes namespace for the service account.
Create an IAM service account for your application or use an existing IAM service account instead. You can use any IAM service account in any project in your organization. For Config Connector, apply the
IAMServiceAccount
object for your selected service account.gcloud
To create a new IAM service account using the gcloud CLI, run the following command.
gcloud iam service-accounts create GSA_NAME \ --project=GSA_PROJECT
Replace the following:
GSA_NAME
: the name of the new IAM service account.GSA_PROJECT
: the project ID of the Google Cloud project for your IAM service account.
Config Connector
To use a new or existing IAM service account with Config Connector, apply the following configuration file.
Note: This step requires Config Connector. Follow the installation instructions to install Config Connector on your cluster.
To deploy this manifest, download it to your machine asservice-account.yaml
.Use
kubectl
to apply the manifest:kubectl apply -f service-account.yaml
For information on authorizing IAM service accounts to access Google Cloud APIs, see Understanding service accounts.
Ensure that your IAM service account has the roles you need. You can grant additional roles using the following command:
gcloud projects add-iam-policy-binding PROJECT_ID \ --member "serviceAccount:GSA_NAME@GSA_PROJECT.iam.gserviceaccount.com" \ --role "ROLE_NAME"
Replace the following:
PROJECT_ID
: your Google Cloud project ID.GSA_NAME
: the name of your IAM service account.GSA_PROJECT
: the project ID of the Google Cloud project of your IAM service account.ROLE_NAME
: the IAM role to assign to your service account, likeroles/spanner.viewer
.
Allow the Kubernetes service account to impersonate the IAM service account by adding an IAM policy binding between the two service accounts. This binding allows the Kubernetes service account to act as the IAM service account.
gcloud
gcloud iam service-accounts add-iam-policy-binding GSA_NAME@GSA_PROJECT.iam.gserviceaccount.com \ --role roles/iam.workloadIdentityUser \ --member "serviceAccount:PROJECT_ID.svc.id.goog[NAMESPACE/KSA_NAME]"
Config Connector
Note: This step requires Config Connector. Follow the installation instructions to install Config Connector on your cluster.
To deploy this manifest, download it to your machine aspolicy-binding.yaml
. ReplaceGSA_NAME
,PROJECT_ID
,NAMESPACE
andKSA_NAME
the values for your environment. Then, run:kubectl apply -f policy-binding.yaml
Annotate the Kubernetes service account with the email address of the IAM service account.
kubectl
kubectl annotate serviceaccount KSA_NAME \ --namespace NAMESPACE \ iam.gke.io/gcp-service-account=GSA_NAME@GSA_PROJECT.iam.gserviceaccount.com
yaml
apiVersion: v1 kind: ServiceAccount metadata: annotations: iam.gke.io/gcp-service-account: GSA_NAME@PROJECT_ID.iam.gserviceaccount.com name: KSA_NAME namespace: NAMESPACE
Update your Pod spec to schedule the workloads on nodes that use Workload Identity and to use the annotated Kubernetes service account.
spec: serviceAccountName: KSA_NAME nodeSelector: iam.gke.io/gke-metadata-server-enabled: "true"
Apply the updated configuration to your cluster:
kubectl apply -f DEPLOYMENT_FILE
Replace
DEPLOYMENT_FILE
with the path to the updated Pod spec.
Verify the Workload Identity setup
Verify the service accounts are configured correctly by creating a Pod with the Kubernetes service account that runs the OS-specific container image, then connect to it with an interactive session.
Linux
Create a Pod that uses the annotated Kubernetes service account and curl
the service-accounts
endpoint.
Save the following configuration as
wi-test.yaml
:apiVersion: v1 kind: Pod metadata: name: workload-identity-test namespace: NAMESPACE spec: containers: - image: google/cloud-sdk:slim name: workload-identity-test command: ["sleep","infinity"] serviceAccountName: KSA_NAME nodeSelector: iam.gke.io/gke-metadata-server-enabled: "true"
The
google/cloud-sdk
image includes the Google Cloud CLI which is a convenient way to consume Google Cloud APIs. It might take some time to download the image.Create the Pod:
kubectl apply -f wi-test.yaml
Open an interactive session in the Pod:
kubectl exec -it workload-identity-test \ --namespace NAMESPACE \ -- /bin/bash
Run the following command inside the Pod:
curl -H "Metadata-Flavor: Google" http://169.254.169.254/computeMetadata/v1/instance/service-accounts/
If the service accounts are correctly configured, the IAM service account email address is listed as the active (and only) identity. This demonstrates that by default, the Pod acts as the IAM service account's authority when calling Google Cloud APIs.
Windows
Create a Pod with the Kubernetes service account that runs the
servercore
container image.
Save the following manifest:
apiVersion: v1 kind: Pod metadata: name: workload-identity-test namespace: NAMESPACE spec: containers: - image: IMAGE_NAME name: workload-identity-test command: ["powershell.exe", "sleep", "3600"] serviceAccountName: KSA_NAME nodeSelector: kubernetes.io/os: windows cloud.google.com/gke-os-distribution: windows_ltsc iam.gke.io/gke-metadata-server-enabled: "true"
Replace
IMAGE_NAME
with one of the following container servercore image values:Windows Server node image Container servercore
imageWINDOWS_LTSC
,WINDOWS_LTSC_CONTAINERD
mcr.microsoft.com/windows/servercore:ltsc2019
WINDOWS_SAC
,WINDOWS_SAC_CONTAINERD
Check the version mapping between the GKE node version and the Windows SAC version. For Windows Server version 1909, specify mcr.microsoft.com/windows/servercore:1909
; otherwise, specifymcr.microsoft.com/windows/servercore:20H2
.Open an interactive session in the Pod:
kubectl exec -it workload-identity-test \ --namespace NAMESPACE -- powershell
Run the following powershell command inside the Pod:
Invoke-WebRequest -Headers @{"Metadata-Flavor"="Google"} -Uri http://169.254.169.254/computeMetadata/v1/instance/service-accounts/default/email -UseBasicParsing
If the service accounts are correctly configured, the IAM service account email address is listed as the active (and only) identity. This demonstrates that by default, the Pod uses the IAM service account's authority when calling Google Cloud APIs.
Use Workload Identity from your code
Authenticating to Google Cloud services from your code is the same process as authenticating using the Compute Engine metadata server. When you use Workload Identity, your requests to the instance metadata server are routed to the GKE metadata server. Existing code that authenticates using the instance metadata server (like code using the Google Cloud client libraries) should work without modification.
Clean up
To stop using Workload Identity, revoke access to the IAM service account and disable Workload Identity on the cluster.
Revoke access
Revoke access to the IAM service account:
gcloud
gcloud iam service-accounts remove-iam-policy-binding GSA_NAME@GSA_PROJECT.iam.gserviceaccount.com \ --role roles/iam.workloadIdentityUser \ --member "serviceAccount:PROJECT_ID.svc.id.goog[NAMESPACE/KSA_NAME]"
Replace the following:
PROJECT_ID
: the project ID of the GKE cluster.NAMESPACE
: the name of the Kubernetes namespace where your Kubernetes service account is located.KSA_NAME
: the name of the Kubernetes service account that will have its access revoked.GSA_NAME
: the name of the IAM service account.GSA_PROJECT
: the project ID of the IAM service account.
Config Connector
If you used Config Connector to create the service account, delete the service account with
kubectl
.kubectl delete -f service-account.yaml
It can take up to 30 minutes for cached tokens to expire. You can check whether the cached tokens have expired with this command:
gcloud auth list
The cached tokens have expired if the output of that command no longer includes
GSA_NAME@GSA_PROJECT.iam.gserviceaccount.com
.Remove the annotation from the Kubernetes service account. This step is optional because access has been revoked by IAM.
kubectl annotate serviceaccount KSA_NAME \ --namespace NAMESPACE iam.gke.io/gcp-service-account-
Disable Workload Identity
You can only disable Workload Identity on GKE Standard clusters.
gcloud
Disable Workload Identity on each node pool:
gcloud container node-pools update NODEPOOL_NAME \ --cluster=CLUSTER_NAME \ --workload-metadata=GCE_METADATA
Repeat this command for every node pool in the cluster.
Disable Workload Identity in the cluster:
gcloud container clusters update CLUSTER_NAME \ --disable-workload-identity
Console
Go to the Google Kubernetes Engine page in Cloud console.
In the cluster list, click the name of the cluster that you want to modify.
Click the Nodes tab.
To disable Workload Identity on each node pool, do the following for each node pool in the Node Pools section:
- Click the name of the node pool that you want to modify.
- On the Node pool details page, click Edit.
- On the Edit node pool page, in the Security section, clear the Enable GKE Metadata Server checkbox.
- Click Save.
To disable Workload Identity for the cluster, do the following:
- Click the Details tab.
- In the Security section, next to Workload Identity, click Edit.
- In the Edit Workload Identity dialog, clear the Enable Workload Identity checkbox.
- Click Save changes.
Disable Workload Identity in your organization
From a security perspective, Workload Identity allows GKE to assert Kubernetes service account identities that can be authenticated and authorized to Google Cloud resources. If you are an administrator who has taken actions to isolate workloads from Google Cloud resources, like disabling service account creation or disabling service account key creation, might also want to disable Workload Identity for your organization.
See these instructions for disabling Workload Identity for your organization.
Troubleshooting
Pod can't authenticate to Google Cloud
If your application can't authenticate to Google Cloud, ensure the following settings are configured properly:
Ensure that you have enabled the IAM Service Account Credentials API in the project containing the GKE cluster.
Ensure that Workload Identity is enabled on the cluster by verifying that it has a workload identity pool set:
gcloud container clusters describe CLUSTER_NAME \ --format="value(workloadIdentityConfig.workloadPool)"
If you haven't already specified a default zone or region for
gcloud
, you may also need to specify a--region
or--zone
flag when running this command.Ensure that the GKE metadata server is configured on the node pool where your application is running:
gcloud container node-pools describe NODEPOOL_NAME \ --cluster=CLUSTER_NAME \ --format="value(config.workloadMetadataConfig.mode)"
Ensure that the Kubernetes service account is annotated correctly:
kubectl describe serviceaccount \ --namespace NAMESPACE KSA_NAME
The output contains an annotation similar to the following:
iam.gke.io/gcp-service-account: GSA_NAME@PROJECT_ID.iam.gserviceaccount.com
Ensure the IAM service account is configured correctly:
gcloud iam service-accounts get-iam-policy \ GSA_NAME@GSA_PROJECT.iam.gserviceaccount.com
The output contains a binding similar to the following:
- members: - serviceAccount:PROJECT_ID.svc.id.goog[NAMESPACE/KSA_NAME] role: roles/iam.workloadIdentityUser
If you have a cluster network policy, ensure that you allowed egress to
127.0.0.1/32
on port988
for clusters running GKE versions prior to 1.21.0-gke.1000, or to169.254.169.252/32
on port988
for clusters running GKE version 1.21.0-gke.1000 and later.kubectl describe networkpolicy NETWORK_POLICY_NAME
Timeout errors at Pod start up
The GKE metadata server needs a few seconds before it can start accepting requests on a new Pod. Therefore, attempts to authenticate using Workload Identity within the first few seconds of a Pod's life may fail for applications and Google Cloud client libraries configured with a short timeout.
If you encounter timeout errors, you can change the application code to wait a few seconds and retry. Alternatively, you can deploy an initContainer that waits until the GKE metadata server is ready before running the Pod's main container.
For example, the following manifest is for a Pod with an initContainer
:
apiVersion: v1
kind: Pod
metadata:
name: pod-with-initcontainer
spec:
serviceAccountName: KSA_NAME
initContainers:
- image: gcr.io/google.com/cloudsdktool/cloud-sdk:326.0.0-alpine
name: workload-identity-initcontainer
command:
- '/bin/bash'
- '-c'
- |
curl -s -H 'Metadata-Flavor: Google' 'http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/token' --retry 30 --retry-connrefused --retry-max-time 30 > /dev/null || exit 1
containers:
- image: gcr.io/your-project/your-image
name: your-main-application-container
Workload Identity fails due to control plane unavailability
The metadata server is not able to return the Workload Identity when the cluster control plane is unavailable. Calls to the metadata server return status code 500.
The log entry might appear similar to the following in the Logs Explorer:
dial tcp 35.232.136.58:443: connect: connection refused
This will lead to expected unavailability of Workload Identity.
The control plane might be unavailable on zonal clusters on cluster maintenance like rotating IPs, upgrading control plane VMs, or resizing clusters or node pools. See more on Choosing a regional or zonal control plane to learn about control plane availability. Switching to regional cluster will eliminate this issue.
Workload Identity fails
If the GKE metadata server is blocked for any reason, Workload Identity will fail.
If you are using Istio, you should add the following application-level annotation to all workloads that use Workload Identity:
"traffic.sidecar.istio.io/excludeOutboundIPRanges=169.254.169.254/32"
Alternatively, you can change the global.proxy.excludeIPRanges
Istio ConfigMap key to do the same thing.
What's next
- Learn more about Workload Identity.
- Read the GKE security overview.
- Learn about Protecting Cluster Metadata.
- Learn about IAM service accounts.