Using Workload Identity

This page explains the recommended way for your Google Kubernetes Engine (GKE) applications to consume services provided by Google APIs.

Overview

Workload Identity is the recommended way to access Google Cloud services from applications running within GKE due to its improved security properties and manageability. For information about alternative ways to access Google Cloud APIs from GKE, refer to the alternatives section below.

Terminology

This document distinguishes between Kubernetes service accounts and Google service accounts. Kubernetes service accounts are Kubernetes resources, while Google service accounts are specific to Google Cloud. Other Google Cloud documentation refers to Google service accounts as "service accounts".

Concepts

Applications running on GKE must authenticate to use Google Cloud APIs such as the Compute APIs, Storage and Database APIs, or Machine Learning APIs.

With Workload Identity, you can configure a Kubernetes service account to act as a Google service account. Any application running as the Kubernetes service account automatically authenticates as the Google service account when accessing Google Cloud APIs. This enables you to assign fine-grained identity and authorization for applications in your cluster.

To accomplish a secure mapping between Kubernetes service accounts and Google service accounts, Workload Identity introduces the concept of a cluster's workload identity pool, which allows Identity and Access Management (IAM) to trust and understand Kubernetes service account credentials.

When you enable Workload Identity on your GKE cluster, set the cluster's workload identity pool to my-pool.svc.id.goog. This lets IAM authenticate Kubernetes service accounts as the following member name:

serviceAccount:my-pool.svc.id.goog[k8s-namespace/ksa-name]

In this member name:

my-pool.svc.id.goog is the workload identity pool set on the cluster.
ksa-name is the name of the Kubernetes service account making the request.
k8s-namespace is the Kubernetes namespace where the Kubernetes service account is defined.

There is only one fixed workload identity pool per Google Cloud project, project-id.svc.id.goog, and it is automatically created for you.

Sharing identities across clusters

All Kubernetes service accounts that share a name, namespace name, and workload identity pool resolve to the same member name, and therefore share access to Google Cloud resources. This can be useful if multiple clusters contain the same identities, but potentially dangerous if Kubernetes service account names and namespaces are not carefully managed.

For example, if Workload Identity is enabled on the cluster, the following command grants the same access to all Kubernetes workloads in any cluster in the project that use the default service account and namespace:

gcloud iam service-accounts add-iam-policy-binding \
  --role roles/iam.workloadIdentityUser \
  --member "serviceAccount:project-id.svc.id.goog[default/default]" \
  gsa-name@project-id.iam.gserviceaccount.com

Limitations

Currently, there is only one fixed workload identity pool per Google Cloud project, project-id.svc.id.goog, and it is automatically created for you.
Currently, Workload Identity is not supported when a workload is running in an GKE on-prem cluster.
Workload Identity replaces the need to use Metadata Concealment and as such, the two approaches are incompatible. The sensitive metadata protected by Metadata Concealment is also protected by Workload Identity.
When the GKE metadata server is enabled on a node pool, Pods will no longer be able to access the Compute Engine metadata server. Instead, request made from these Pods to the Metadata APIs will be routed to the GKE metadata server. The one exception to this is Pods running on the host network (see next item).
Workload Identity can't be used by Pods running on the host network. Request made from these Pods to the Metadata APIs will be routed to the Compute Engine metadata server.
The GKE metadata server takes a few seconds to start to run on a newly created Pod. Therefore, attempts to authenticate or authorize using Workload Identity made within the first few seconds of a Pod's life may fail. Retrying the call will resolve the problem.
GKE logging and monitoring agents will continue to use the node's service account.
Workload Identity is not supported on Windows nodes.
Workload Identity requires manual setup for Cloud Run for Anthos on Google Cloud to continue releasing request metrics.
Workload Identity installs ip-masq-agent if the cluster is created without the --disable-default-snat flag.

Before you begin

Before you start, make sure you have performed the following tasks:

Ensure that you have enabled the Google Kubernetes Engine API.

Enable Google Kubernetes Engine API

Ensure that you have installed the Cloud SDK.

Set up default gcloud settings using one of the following methods:

Using gcloud init, if you want to be walked through setting defaults.
Using gcloud config, to individually set your project ID, zone, and region.

Using gcloud init

If you receive the error One of [--zone, --region] must be supplied: Please specify location, complete this section.

Run gcloud init and follow the directions:
```
gcloud init
```
If you are using SSH on a remote server, use the --console-only flag to prevent the command from launching a browser:
```
gcloud init --console-only
```
Follow the instructions to authorize gcloud to use your Google Cloud account.
Create a new configuration or select an existing one.
Choose a Google Cloud project.
Choose a default Compute Engine zone.

Note: You can override these default settings in gcloud commands using the --project, --zone, and --region flags.

Using gcloud config

Set your default project ID:
```
gcloud config set project project-id
```
If you are working with zonal clusters, set your default compute zone:
```
gcloud config set compute/zone compute-zone
```
If you are working with regional clusters, set your default compute region:
```
gcloud config set compute/region compute-region
```
Update gcloud to the latest version:
```
gcloud components update
```

Note: You can override these default settings in gcloud commands using the --project, --zone, and --region flags.

Enabling Workload Identity on a cluster

You can enable Workload Identity on a new or existing cluster by using the gcloud tool.

Ensure that you have enabled the IAM Service Account Credentials API.

Enable IAM Credentials API
To create a new cluster with Workload Identity enabled, use the following command:
```
gcloud container clusters create cluster-name \
  --workload-pool=project-id.svc.id.goog
```
Replace the following:
- cluster-name: the name of your cluster.
- project-id: the ID of your Google Cloud project.
This action requires container.clusters.create permission on the project.
To enable Workload Identity on an existing cluster, modify the cluster with the following command:
```
gcloud container clusters update cluster-name \
  --workload-pool=project-id.svc.id.goog
```
Existing node pools are unaffected; new node pools default to --workload-metadata=GKE_METADATA.

This action requires container.clusters.update permissions on the cluster.

Migrate applications to Workload Identity

Select the migration strategy that is ideal for your environment. Node pools can be migrated in place or you can create new node pools with Workload Identity enabled. We recommend creating new node pools if you also need to modify your application to be compatible with this feature.

Option 1: Node pool creation with Workload Identity (recommended)

Add a new node pool to the cluster with Workload Identity enabled and manually migrate workloads to that pool. This succeeds only if Workload Identity is enabled on the cluster.

gcloud container node-pools create nodepool-name \
  --cluster=cluster-name \
  --workload-metadata=GKE_METADATA

If a cluster has Workload Identity enabled, you can selectively disable it on a specific node pool by explicitly specifying --workload-metadata=GCE_METADATA. See Protecting cluster metadata for more information.

Option 2: Node pool modification

Modify an existing node pool to enable GKE_METADATA. This update succeeds only if Workload Identity is enabled on the cluster. It immediately enables Workload Identity for workloads deployed to the node pool. This change will prevent workloads from using the Compute Engine service account and must be carefully rolled out.

gcloud container node-pools update nodepool-name \
  --cluster=cluster-name \
  --workload-metadata=GKE_METADATA

This action requires container.nodes.update permissions on the project.

Authenticating to Google Cloud

This section explains how an application can authenticate to Google Cloud using Workload Identity. To do this, assign a Kubernetes service account to the application and configure it to act as a Google service account:

Configure kubectl to communicate with the cluster:
```
gcloud container clusters get-credentials cluster-name
```
Replace cluster-name with the name of the cluster you created in the previous step.

This action requires container.clusters.get permission on the project.
Like most other resources, Kubernetes service accounts live in a namespace. Create the namespace to use for the Kubernetes service account.
```
kubectl create namespace k8s-namespace
```
This action requires Create namespace RBAC permission within the cluster.
Create the Kubernetes service account to use for your application:
```
kubectl create serviceaccount --namespace k8s-namespace ksa-name
```
Replace the following:
- k8s-namespace: the name of the Kubernetes namespace you created in the previous step.
- ksa-name: the name you would like to use for the Kubernetes service account.
This action requires create serviceaccounts RBAC permission within the namespace.

Alternatively, you can use the default namespace or the default Kubernetes service account in any namespace.
Create a Google service account for your application. If you have an existing service account, you can use it instead of creating a new service account. The service account doesn't need to be in the same project as your cluster. You can use any Google service account in your organization.
gcloud
Replace gsa-name with the name you choose for the service account.
```
gcloud iam service-accounts create gsa-name
```
Config Connector
If you have Config Connector already installed on a cluster, you can create a new GKE cluster with Workload Identity enabled using a Config Connector configuration.

Note: This step requires Config Connector. Follow the installation instructions to install Config Connector on your cluster.

View on GitHub
```
apiVersion: iam.cnrm.cloud.google.com/v1beta1
kind: IAMServiceAccount
metadata:
  name: [GSA_NAME]
spec:
  displayName: [GSA_NAME]
```
To deploy this manifest, download it to your machine as service-account.yaml. Replace gsa-name with the name you choose for the service account. Then use kubectl to apply the manifest.
```
kubectl apply -f service-account.yaml
```
This action requires iam.serviceAccounts.create permission on the project.

For information on authorizing Google service accounts to access Google Cloud APIs, see Understanding service accounts.

Allow the Kubernetes service account to impersonate the Google service account by creating an IAM policy binding between the two. This binding allows the Kubernetes Service account to act as the Google service account.

gcloud

gcloud iam service-accounts add-iam-policy-binding \
  --role roles/iam.workloadIdentityUser \
  --member "serviceAccount:project-id.svc.id.goog[k8s-namespace/ksa-name]" \
  gsa-name@project-id.iam.gserviceaccount.com

Config Connector

Note: This step requires Config Connector. Follow the installation instructions to install Config Connector on your cluster.

View on GitHub

apiVersion: iam.cnrm.cloud.google.com/v1beta1
kind: IAMPolicy
metadata:
  name: iampolicy-workload-identity-sample
spec:
  resourceRef:
    apiVersion: iam.cnrm.cloud.google.com/v1beta1
    kind: IAMServiceAccount
    name: [GSA_NAME]
  bindings:
    - role: roles/iam.workloadIdentityUser
      members:
        - serviceAccount:[PROJECT_ID].svc.id.goog[[K8S_NAMESPACE]/[KSA_NAME]]

To deploy this manifest, download it to your machine as policy-binding.yaml. Replace gsa-name, project-id, k8s-namespace and ksa-name the values for your environment. Then, run:

kubectl apply -f policy-binding.yaml

This action requires iam.serviceAccounts.setIamPolicy permission on the project.

Add the iam.gke.io/gcp-service-account=gsa-name@project-id annotation to the Kubernetes service account, using the email address of the Google service account.
kubectl
```
kubectl annotate serviceaccount \
  --namespace k8s-namespace \
  ksa-name \
  iam.gke.io/gcp-service-account=gsa-name@project-id.iam.gserviceaccount.com
```
yaml
```
apiVersion: v1
kind: ServiceAccount
metadata:
  annotations:
    iam.gke.io/gcp-service-account: gsa-name@project-id.iam.gserviceaccount.com
  name: ksa-name
  namespace: k8s-namespace
```
This action requires RBAC edit permissions on the Kubernetes service account.

Note: This annotation by itself does not grant access to impersonate the Google service account. If the IAM binding does not exist, the Pod will not be able to use the Google service account.
Verify the service accounts are configured correctly by creating a Pod with the Kubernetes service account that runs the cloud-sdk container image, and connecting to it with an interactive session.
kubectl
```
kubectl run -it \
  --image google/cloud-sdk:slim \
  --serviceaccount ksa-name \
  --namespace k8s-namespace \
  workload-identity-test
```
yaml
```
apiVersion: v1
kind: Pod
metadata:
  name: workload-identity-test
  namespace: k8s-namespace
spec:
  containers:
  - image: google/cloud-sdk:slim
    name: workload-identity-test
    command: ["sleep","infinity"]
  serviceAccountName: ksa-name
```
The google/cloud-sdk image includes the gcloud command-line tool which is a convenient way to consume Google Cloud APIs. It may take some time to download the image.

This action requires create Pods RBAC permission within the namespace.

You are now connected to an interactive shell within the created Pod. Run the following command inside the Pod:
```
gcloud auth list
```
If the service accounts are correctly configured, the Google service account email address is listed as the active (and only) identity. This demonstrates that by default, the Pod uses the Google service account's authority when calling Google Cloud APIs.

Using Workload Identity from your code

Authenticating to Google Cloud services from your code is the same process as authenticating using the Compute Engine metadata server. When you use Workload Identity, your requests to the instance metadata server are routed to the GKE metadata server. Existing code that authenticates using the instance metadata server (like code using the Google Cloud client libraries) should work without modification.

Understanding the GKE metadata server

The GKE metadata server is a new metadata server designed for use with Kubernetes. It runs as a daemonset , with one Pod on each cluster node. The metadata server intercepts HTTP requests to http://metadata.google.internal (169.254.169.254:80), including requests like GET /computeMetadata/v1/instance/service-accounts/default/token to retrieve a token for the Google service account the Pod is configured to act as. Traffic to the metadata server never leaves the VM instance that hosts the Pod.

The GKE metadata server implements only a subset of Compute Engine metadata server endpoints which are relevant and safe for Kubernetes workloads:

/computeMetadata/v1/instance/attributes/cluster-location
/computeMetadata/v1/instance/attributes/cluster-name
/computeMetadata/v1/instance/attributes/cluster-uid
/computeMetadata/v1/instance/hostname
/computeMetadata/v1/instance/id
/computeMetadata/v1/project/numeric-project-id
/computeMetadata/v1/project/project-id
/computeMetadata/v1/instance/service-accounts
/computeMetadata/v1/instance/service-accounts/default
/computeMetadata/v1/instance/service-accounts/default/aliases
/computeMetadata/v1/instance/service-accounts/default/email
/computeMetadata/v1/instance/service-accounts/default/identity
/computeMetadata/v1/instance/service-accounts/default/identity?audience=audience
/computeMetadata/v1/instance/service-accounts/default/scopes
/computeMetadata/v1/instance/service-accounts/default/token
/computeMetadata/v1/instance/service-accounts/default/token?scopes=comma-separated-list-of-scopes

Revoking access

Revoke access to the Google service account using IAM:
gcloud
```
gcloud iam service-accounts remove-iam-policy-binding \
  --role roles/iam.workloadIdentityUser \
  --member "serviceAccount:project-id.svc.id.goog[k8s-namespace/ksa-name]" \
  gsa-name@gsa-project-id.iam.gserviceaccount.com
```
Replace the following:
- project-id: the project ID container the GKE cluster.
- k8s-namespace: the name of the Kubernetes namespace where your Kubernetes service account is located.
- ksa-name: the name of the Kubernetes service account that will have its access revoked.
- gsa-name: the name of the Google service account.
- gsa-project-id: the project ID containing the Google service account.
Config Connector
If you used Config Connector to create the service account, delete the service account with kubectl.
```
kubectl delete -f service-account.yaml
```
This action requires iam.serviceAccounts.setIamPolicy permissions on the service account.

It can take up to 30 minutes for cached tokens to expire. You can check whether the cached tokens have expired with this command:
```
gcloud auth list
```
The cached tokens have expired if the output of that command no longer includes gsa-name@project-id.iam.gserviceaccount.com.
Remove the annotation from the Kubernetes service account. This step is optional because access has been revoked by IAM.
```
kubectl annotate serviceaccount \
  --namespace k8s-namespace \
  ksa-name \
  iam.gke.io/gcp-service-account-
```

Troubleshooting

If your application can't authenticate to Google Cloud, ensure these settings are configured properly:

Ensure that you have enabled the IAM Service Account Credentials API in the project containing the GKE cluster.

Enable IAM Credentials API
Ensure that Workload Identity is enabled on the cluster by verifying that it has a workload identity pool set:
```
gcloud container clusters describe cluster-name \
  --format="value(workloadIdentityConfig.workloadPool)"
```

Ensure that GKE metadata server (GKE_METADATA) is configured on the node pool where your application is running:

gcloud container node-pools describe node-pool-name \
  --cluster=cluster-name \
  --format="value(config.workloadMetadataConfig.mode)"

Ensure that the Kubernetes service account is annotated correctly

kubectl describe serviceaccount \
  --namespace k8s-namespace \
  ksa-name

There should be an annotation in the following format:

iam.gke.io/gcp-service-account: gsa-name@project-id.iam.gserviceaccount.com

Ensure the Google service account is configured correctly

gcloud iam service-accounts get-iam-policy \
  gsa-name@project-id.iam.gserviceaccount.com

Verify that there is a binding with in the following format:

- members:
  - serviceAccount:project-id.svc.id.goog[k8s-namespace/ksa-name]
  role: roles/iam.workloadIdentityUser

If you have a cluster network policy , ensure egress to 127.0.0.1/32 on port 988 is allowed:
```
kubectl describe networkpolicy network-policy-name
```

Disabling Workload Identity on a cluster

Disable Workload Identity on each node pool:

gcloud container node-pools update nodepool-name \
  --cluster=cluster-name \
  --workload-metadata=GCE_METADATA

Repeat this command for every node pool in the cluster.

Disable Workload Identity in the cluster:
```
gcloud container clusters update cluster-name \
  --disable-workload-identity
```
This action requires container.clusters.update permissions on the cluster.

Disabling Workload Identity in your organization

From a security perspective, Workload Identity allows GKE to assert Kubernetes service account identities that can be authenticated and authorized to Google Cloud resources. Administrators who have taken actions to isolate workloads from Google Cloud resources, like disabling service account creation or disabling service account key creation, might also wish to disable Workload Identity for your organization.

See these instructions for disabling Workload Identity for your organization.

Alternatives to Workload Identity

There are two alternative methods to access Cloud APIs from GKE. With the release of Workload Identity we no longer recommend these approaches because of the compromises they require.

Export service account keys and store them as Kubernetes Secrets. Google service account keys expire after 10 years and are rotated manually. Exporting service account keys has the potential to expand the scope of a security breach if it goes undetected.
Use the Compute Engine service account of your nodes. You can run node pools as any IAM service account in your project. If you do not specify a service account during node pool creation, GKE will use the Compute Engine default service account of the project. The Compute Engine service account is shared by all workloads deployed on that node. This can result in over-provisioning of permissions.

What's next

Read the GKE security overview.
Learn about Pods.
Learn about Protecting Cluster Metadata.
Learn about Google service accounts.