Workload Identity

This page explains the recommended way for your Google Kubernetes Engine (GKE) applications to consume services provided by Google APIs.

Overview

Workload Identity is the recommended way to access Google Cloud services from within GKE due to its improved security properties and manageability. For information about alternative ways to access Cloud APIs from GKE, refer to the alternatives section below.

Workloads running on GKE must authenticate to use Google Cloud APIs such as the Compute APIs, Storage and Database APIs, or Machine Learning APIs.

With Workload Identity, you configure a Kubernetes service account (KSA) (KSA) to act as a Google service account (GSA). Any workload running as the KSA automatically authenticates as the GSA when accessing Google Cloud APIs.

For information on authorizing GSAs to access Google Cloud APIs, see Understanding service accounts.

Terminology

This document distinguishes between Kubernetes service accounts (KSAs) and Google service accounts (GSAs). KSAs are Kubernetes resources, while GSAs are specific to Google Cloud. Other Google Cloud documentation refers to GSAs as "service accounts".

Authenticating Kubernetes service accounts

By default, Google Cloud can't authenticate requests from a Kubernetes service account (KSA), because the workload doesn't understand which authentication credentials to present to Google Cloud, and because the only credentials available to the workload originates from the Kubernetes identity system, which is not trusted by Google Cloud.

Workload Identity introduces the concept of a cluster's workload identity pool, which allows Cloud IAM to trust and understand external identity systems, such as KSA credentials. Workload Identity intercepts calls to the Compute Engine metadata server to exchange workload identity pool credentials for credentials from the associated GSAs. This allows workloads to be authenticated to all other Google Cloud APIs.

When you enable Workload Identity for your GKE cluster, set the cluster's workload identity pool to some.workload-pool.id.goog. GKE authenticates requests from KSAs to Google Cloud using the following member name:

serviceAccount:some.workload-pool.id.goog[k8s_namespace/ksa_name]

In this member name:

  • some.workload-pool.id.goog is the workload identity pool set on the cluster.
  • ksa_name is the name of the KSA making the request.
  • k8s_namespace is the Kubernetes namespace where the KSA is defined.

There is only one fixed workload identity pool per Google Cloud project, project-id.svc.id.goog, and it is automatically created for you.

Creating a relationship between KSAs and GSAs

Because Workload Identity lets Google Cloud authenticate KSAs, you can use Cloud IAM to authorize a KSA to act as a GSA.

The GSA doesn't need to be in the same project as your cluster. You can use any GSA in your organization. For example, the following commands authorize a KSA hosted in cluster_project to act as a GSA in gsa_project:

gcloud config set project gsa_project
gcloud iam service-accounts add-iam-policy-binding \
  --role roles/iam.workloadIdentityUser \
  --member "serviceAccount:cluster_project.svc.id.goog[k8s_namespace/ksa_name]" \
  gsa_name@gsa_project.iam.gserviceaccount.com

In this example:

  • cluster_project is the project ID for a cluster project.
  • gsa_project is the project ID for a GSA project that's not in the cluster.

To annotate the KSA to complete the binding between the KSA and GSA:

kubectl annotate serviceaccount \
  --namespace k8s_namespace \
   ksa_name \
   iam.gke.io/gcp-service-account=gsa_name@gsa_project.iam.gserviceaccount.com

Using Workload Identity from your code

Authenticating to Google Cloud services from your code is the same process as authenticating using the Compute Engine metadata server. When you use Workload Identity, your requests to the instance metadata server are routed to the GKE metadata server. Existing code that authenticates using the instance metadata server (like code using the Google Cloud client libraries) should work without modification.

The GKE metadata server is a new metadata server designed for use with Kubernetes. It runs as a daemonset, with one Pod on each cluster node. The metadata server intercepts HTTP requests to http://metadata.google.internal (169.254.169.254:80), including requests like GET /computeMetadata/v1/instance/service-accounts/default/token to retrieve a token for the GSA the Pod is configured to act as. Traffic to the metadata server never leaves the VM instance that hosts the Pod.

Sharing identities across clusters

All KSAs that share a name, Namespace name, and workload identity pool resolve to the same member name, and therefore share access to GSAs. This can be useful if multiple clusters contain the same identities, but potentially dangerous if KSA names and Namespaces are not carefully managed.

For example, if Workload Identity is enabled on the cluster, the following command grants the same access to all Kubernetes workloads in any cluster in the project that use the default service account and Namespace:

gcloud iam service-accounts add-iam-policy-binding \
  --role roles/iam.workloadIdentityUser \
  --member "serviceAccount:project-id.svc.id.goog[default/default]" \
  gsa-name@project-id.iam.gserviceaccount.com

Replace the following:

  • project-id: the project ID that uses the default service account.
  • gsa-name: the name of the GSA.

Limitations

  • Currently, there is only one fixed workload identity pool per Google Cloud project, project-id.svc.id.goog, and it is automatically created for you.

  • Currently, Workload Identity is not supported when a workload is running in an Anthos GKE on-prem cluster.

  • Workload Identity replaces the need to use Metadata Concealment and as such, the two approaches are incompatible. The sensitive metadata protected by Metadata Concealment is also protected by Workload Identity.

  • When the GKE Metadata Server is enabled on a node pool, pods will no longer be able to access the Compute Engine Metadata Server. Instead, request made from these pods to the Metadata APIs will be routed to the GKE Metadata Server. The one exception to this is pods running on the host network (see next item).

  • Workload Identity can't be used with pods running on the host network. Request made from these pods to the Metadata APIs will be routed to the Compute Engine Metadata Server.

  • The GKE Metadata Server takes a few seconds to start to run on a newly created pod. Therefore, attempts to authenticate or authorize using Workload Identity made within the first few seconds of a pod's life may fail. Retrying the call will resolve the problem.

  • GKE logging and monitoring agents will continue to use the node's service account.

  • Workload Identity is not supported on Windows nodes.

  • Workload Identity requires manual setup for Cloud Run for Anthos on Google Cloud to continue releasing request metrics.

Before you begin

Before you start, make sure you have performed the following tasks:

Set up default gcloud settings using one of the following methods:

  • Using gcloud init, if you want to be walked through setting defaults.
  • Using gcloud config, to individually set your project ID, zone, and region.

Using gcloud init

If you receive the error One of [--zone, --region] must be supplied: Please specify location, complete this section.

  1. Run gcloud init and follow the directions:

    gcloud init

    If you are using SSH on a remote server, use the --console-only flag to prevent the command from launching a browser:

    gcloud init --console-only
  2. Follow the instructions to authorize gcloud to use your Google Cloud account.
  3. Create a new configuration or select an existing one.
  4. Choose a Google Cloud project.
  5. Choose a default Compute Engine zone.

Using gcloud config

  • Set your default project ID:
    gcloud config set project project-id
  • If you are working with zonal clusters, set your default compute zone:
    gcloud config set compute/zone compute-zone
  • If you are working with regional clusters, set your default compute region:
    gcloud config set compute/region compute-region
  • Update gcloud to the latest version:
    gcloud components update

Enable Workload Identity on a new cluster

This section explains how to create a cluster with Workload Identity enabled, and how to start a Pod with a Google service account (GSA) identity.

  1. Ensure that you have enabled the Cloud IAM Service Account Credentials API.

    Enable Cloud IAM Credentials API

  2. Create a cluster with Workload Identity enabled, using the following command:

    gcloud beta container clusters create cluster-name \
      --release-channel regular \
      --workload-pool=project-id.svc.id.goog
    

    Replace the following:

    • cluster-name: the name of your cluster.
    • project-id: the ID of your Google Cloud project.

    Creating the cluster takes several minutes.

    This action requires container.clusters.create permission on the project.

    Perform the remainder of the steps with a less privileged role, in accordance with the principle of least privilege. Each step notes the permissions that are required.

  3. Configure kubectl to communicate with the cluster:

    gcloud container clusters get-credentials cluster-name
    

    Replace cluster-name with the name of the cluster you created in the previous step.

    This action requires container.clusters.get permission on the project.

  4. Create the Google service account. This action requires iam.serviceAccounts.create permission on the project. If you have an existing service account, you can use it instead of creating a new service account.

    gcloud

    Replace gsa-name with the name you choose for the service account.

    gcloud iam service-accounts create gsa-name
    

    Config Connector

    If you have Config Connector already installed on a cluster, you can create a new GKE cluster with Workload Identity enabled using a Config Connector configuration.

    Note: This step requires Config Connector. Follow the installation instructions to install Config Connector on your cluster.

    apiVersion: iam.cnrm.cloud.google.com/v1beta1
    kind: IAMServiceAccount
    metadata:
      name: [GSA_NAME]
    spec:
      displayName: [GSA_NAME]
    To deploy this manifest, download it to your machine as service-account.yaml. Replace GSA_NAME with the name you choose for the service account. Then use kubectl to apply the manifest.

    kubectl apply -f service-account.yaml
  5. Like most other resources, Kubernetes service accounts live in a Namespace. Create the Namespace to use for the Kubernetes service account.

    kubectl create namespace k8s-namespace
    

    This action requires Create Namespace RBAC permission within the cluster.

  6. Create the Kubernetes service account:

    kubectl create serviceaccount --namespace k8s-namespace ksa-name
    

    Replace the following:

    • k8s-namespace: the name of the namespace you created in the previous step.
    • ksa-name: the name you would like to use for the Kubernetes service account.

    This action requires create serviceaccounts RBAC permission within the Namespace.

    Alternatively, you can use the default Namespace or the default Kubernetes service account in any Namespace.

  7. Allow the Kubernetes service account to use the Google service account by creating a Cloud IAM policy binding between the two. This binding allows the Kubernetes Service account to act as the Google service account. This action requires iam.serviceAccounts.setIamPolicy permission on the project.

    gcloud

    gcloud iam service-accounts add-iam-policy-binding \
      --role roles/iam.workloadIdentityUser \
      --member "serviceAccount:project-id.svc.id.goog[k8s-namespace/ksa-name]" \
      gsa-name@project-id.iam.gserviceaccount.com
    

    Config Connector

    Note: This step requires Config Connector. Follow the installation instructions to install Config Connector on your cluster.

    apiVersion: iam.cnrm.cloud.google.com/v1beta1
    kind: IAMPolicy
    metadata:
      name: iampolicy-workload-identity-sample
    spec:
      resourceRef:
        apiVersion: iam.cnrm.cloud.google.com/v1beta1
        kind: IAMServiceAccount
        name: [GSA_NAME]
      bindings:
        - role: roles/iam.workloadIdentityUser
          members:
            - serviceAccount:[PROJECT_ID].svc.id.goog[[K8S_NAMESPACE]/[KSA_NAME]]
    To deploy this manifest, download it to your machine as policy-binding.yaml. Replace GSA_NAME, PROJECT_ID, K8S_NAMESPACE and KSA_NAME the values for your environment. Then, run:

    kubectl apply -f policy-binding.yaml
  8. Add the iam.gke.io/gcp-service-account=gsa-name@project-id annotation to the Kubernetes service account, using the email address of the Google service account:

    kubectl annotate serviceaccount \
      --namespace k8s-namespace \
      ksa-name \
      iam.gke.io/gcp-service-account=gsa-name@project-id.iam.gserviceaccount.com
    

    This action requires RBAC edit permissions on the Kubernetes service account.

  9. Verify the service accounts are configured correctly by creating a Pod with the Kubernetes service account that runs the cloud-sdk container image, and connecting to it with an interactive session.

    kubectl run -it \
      --image google/cloud-sdk:slim \
      --serviceaccount ksa-name \
      --namespace k8s-namespace \
      workload-identity-test
    

    The google/cloud-sdk image includes the gcloud command-line tool which is a convenient way to consume Google Cloud APIs. It may take some time to download the image.

    This action requires create pods RBAC permission within the Namespace.

    You are now connected to an interactive shell within the created Pod. Run the following command:

    gcloud auth list
    

    If the service accounts are correctly configured, the Google service account email address is listed as the active (and only) identity. This demonstrates that by default, the Pod uses the Google service account's authority when calling Google Cloud APIs.

Cleaning up

  1. Revoke access to the Google service account:

    gcloud

    Replace gsa-name with the name you choose for the service account.

    gcloud iam service-accounts delete gsa-name
    

    Config Connector

    If you used Config Connector to create the service account, delete the service account with kubectl.

    kubectl delete -f service-account.yaml
    

    This action requires iam.serviceAccounts.setIamPolicy permissions on the service account.

    It can take up to 30 minutes for cached tokens to expire. You can check whether the cached tokens have expired with this command:

    gcloud auth list
    

    The cached tokens have expired if the output of that command no longer includes gsa-name@project-id.iam.gserviceaccount.com.

  2. Disable Workload Identity in the cluster:

    gcloud container clusters update cluster-name \
      --disable-workload-identity
    

    This action requires container.clusters.update permissions on the cluster.

Enable Workload Identity on an existing cluster

  1. Ensure that you have enabled the Cloud IAM Service Account Credentials API.

    Enable Cloud IAM Credentials API

  2. Modify the cluster to enable Workload Identity. Existing node pools are unaffected. New node pools default to --workload-metadata=GKE_METADATA.

    gcloud container clusters update cluster-name \
      --workload-pool=project-id.svc.id.goog
    

Migrate workloads to Workload Identity

Select the migration strategy that is ideal for your environment. Node pools can be migrated in place or you can create new node pools with Workload Identity enabled. We recommend creating new node pools if you also need to modify your application to be compatible with this feature.

Option 1: Node pool modification

Modify an existing node pool to enable GKE_METADATA. This update succeeds only if Workload Identity is enabled on the cluster. It immediately enables Workload Identity for workloads deployed to the node pool.

gcloud container node-pools update nodepool-name \
  --cluster=cluster-name \
  --workload-metadata=GKE_METADATA

This action requires container.nodes.update permissions on the project.

Option 2: Node pool creation with Workload Identity

Add a new node pool to the cluster with Workload Identity enabled and manually migrate workloads to that pool. This succeeds only if Workload Identity is enabled on the cluster.

gcloud container node-pools create nodepool-name \
  --cluster=cluster-name \
  --workload-metadata=GKE_METADATA

If a cluster has Workload Identity enabled, you can selectively disable it on a specific node pool by explicitly specifying --workload-metadata=GCE_METADATA. See Protecting cluster metadata for more information.

Cleaning Up

  1. Modify existing node pools to disable Workload Identity:

    gcloud container node-pools update nodepool-name \
      --cluster=cluster-name \
      --workload-metadata=GCE_METADATA
    

    This action requires container.nodes.update permissions on the project.

  2. Modify the cluster to disable Workload Identity. Existing node pools are unaffected (but the modification is blocked if any node pools are using GKE_METADATA). In node pools created after disabling Workload Identity, Pods running in the node pool have access to their node's underlying Compute Engine metadata server.

    gcloud container clusters update cluster-name \
      --disable-workload-identity
    

    This action requires container.clusters.update permissions on the cluster.

Alternatives

There are two alternative methods to access Cloud APIs from GKE. With the release of Workload Identity we no longer recommend these approaches because of the compromises they require.

  1. Export service account keys and store them as Kubernetes Secrets. Google service account keys expire after 10 years and are rotated manually. Exporting service account keys has the potential to expand the scope of a security breach if it goes undetected.

  2. Use the Compute Engine service account of your nodes. You can run node pools as any IAM service account in your project. If you do not specify a service account during node pool creation, GKE will use the Compute Engine default service account of the project. The Compute Engine service account is shared by all workloads deployed on that node. This can result in over-provisioning of permissions.

Disabling Workload Identity in your organization

From a security perspective, Workload Identity allows GKE to assert KSA identities that can be authenticated and authorized to Google Cloud resources. For administrators who have taken actions to isolate workloads from Google Cloud resources, like disabling service account creation or disabling service account key creation, administrators might also wish to disable Workload Identity for your organization.

See these instructions for disabling Workload Identity for your organization.

What's next