This page explains the recommended way for your Google Kubernetes Engine (GKE) applications to consume services provided by Google APIs.
Overview
Workload Identity is the recommended way to access Google Cloud services from applications running within GKE due to its improved security properties and manageability. For information about alternative ways to access Google Cloud APIs from GKE, refer to the alternatives section below.
Terminology
This document distinguishes between Kubernetes service accounts and Google service accounts. Kubernetes service accounts are Kubernetes resources, while Google service accounts are specific to Google Cloud. Other Google Cloud documentation refers to Google service accounts as "service accounts".
Concepts
Applications running on GKE must authenticate to use Google Cloud APIs such as the Compute APIs, Storage and Database APIs, or Machine Learning APIs.
With Workload Identity, you can configure a Kubernetes service account to act as a Google service account. Any application running as the Kubernetes service account automatically authenticates as the Google service account when accessing Google Cloud APIs. This enables you to assign fine-grained identity and authorization for applications in your cluster.
To accomplish a secure mapping between Kubernetes service accounts and Google service accounts, Workload Identity introduces the concept of a cluster's workload identity pool, which allows Identity and Access Management (IAM) to trust and understand Kubernetes service account credentials.
When you enable Workload Identity on your
GKE cluster, set the cluster's workload identity pool to my-pool.svc.id.goog
.
This lets IAM authenticate Kubernetes service accounts as the
following member name:
serviceAccount:my-pool.svc.id.goog[K8S_NAMESPACE/KSA_NAME]
In this member name:
my-pool.svc.id.goog
is the workload identity pool set on the cluster.KSA_NAME
is the name of the Kubernetes service account making the request.K8S_NAMESPACE
is the Kubernetes namespace where the Kubernetes service account is defined.
There is only one fixed workload identity pool per Google Cloud project,
PROJECT_ID.svc.id.goog
, and it is automatically created for you.
Sharing identities across clusters
All Kubernetes service accounts that share a name, namespace name, and workload identity pool resolve to the same member name, and therefore share access to Google Cloud resources. This can be useful if multiple clusters contain the same identities, but potentially dangerous if Kubernetes service account names and namespaces are not carefully managed.
For example, if Workload Identity is enabled on the cluster, the following command grants the same access to all Kubernetes workloads in any cluster in the project that use the default service account and namespace:
gcloud iam service-accounts add-iam-policy-binding \
--role roles/iam.workloadIdentityUser \
--member "serviceAccount:PROJECT_ID.svc.id.goog[default/default]" \
GSA_NAME@PROJECT_ID.iam.gserviceaccount.com
Limitations
Currently, there is only one fixed workload identity pool per Google Cloud project,
PROJECT_ID.svc.id.goog
, and it is automatically created for you.Currently, Workload Identity is not supported when a workload is running in Anthos clusters on VMware.
Workload Identity replaces the need to use Metadata Concealment and as such, the two approaches are incompatible. The sensitive metadata protected by Metadata Concealment is also protected by Workload Identity.
When the GKE metadata server is enabled on a node pool, Pods will no longer be able to access the Compute Engine metadata server. Instead, request made from these Pods to the Metadata APIs will be routed to the GKE metadata server. The one exception to this is Pods running on the host network (see next item).
Workload Identity can't be used by Pods running on the host network. Request made from these Pods to the Metadata APIs will be routed to the Compute Engine metadata server.
The GKE metadata server takes a few seconds to start to run on a newly created Pod. Therefore, attempts to authenticate or authorize using Workload Identity made within the first few seconds of a Pod's life may fail. Retrying the call will resolve the problem.
GKE logging and monitoring agents will continue to use the node's service account.
Workload Identity is not supported on Windows nodes.
Workload Identity requires manual setup for Cloud Run for Anthos on Google Cloud to continue releasing request metrics.
Workload Identity installs
ip-masq-agent
if the cluster is created without the--disable-default-snat
flag.
Before you begin
Before you start, make sure you have performed the following tasks:
- Ensure that you have enabled the Google Kubernetes Engine API. Enable Google Kubernetes Engine API
- Ensure that you have installed the Cloud SDK.
Set up default gcloud
settings using one of the following methods:
- Using
gcloud init
, if you want to be walked through setting defaults. - Using
gcloud config
, to individually set your project ID, zone, and region.
Using gcloud init
If you receive the error One of [--zone, --region] must be supplied: Please specify
location
, complete this section.
-
Run
gcloud init
and follow the directions:gcloud init
If you are using SSH on a remote server, use the
--console-only
flag to prevent the command from launching a browser:gcloud init --console-only
-
Follow the instructions to authorize
gcloud
to use your Google Cloud account. - Create a new configuration or select an existing one.
- Choose a Google Cloud project.
- Choose a default Compute Engine zone.
Using gcloud config
- Set your default project ID:
gcloud config set project PROJECT_ID
- If you are working with zonal clusters, set your default compute zone:
gcloud config set compute/zone COMPUTE_ZONE
- If you are working with regional clusters, set your default compute region:
gcloud config set compute/region COMPUTE_REGION
- Update
gcloud
to the latest version:gcloud components update
Enabling Workload Identity on a cluster
You can enable Workload Identity on a new or existing cluster by using the gcloud
tool.
Ensure that you have enabled the IAM Service Account Credentials API.
To create a new cluster with Workload Identity enabled, use the following command:
gcloud container clusters create CLUSTER_NAME \ --workload-pool=PROJECT_ID.svc.id.goog
Replace the following:
CLUSTER_NAME
: the name of your cluster.PROJECT_ID
: the ID of your Google Cloud project.
This action requires
container.clusters.create
permission on the project.To enable Workload Identity on an existing cluster, modify the cluster with the following command:
gcloud container clusters update CLUSTER_NAME \ --workload-pool=PROJECT_ID.svc.id.goog
Existing node pools are unaffected; new node pools default to
--workload-metadata=GKE_METADATA
.This action requires
container.clusters.update
permissions on the cluster.
Migrate applications to Workload Identity
Select the migration strategy that is ideal for your environment. Node pools can be migrated in place or you can create new node pools with Workload Identity enabled. We recommend creating new node pools if you also need to modify your application to be compatible with this feature.
Option 1: Node pool creation with Workload Identity (recommended)
Add a new node pool to the cluster with Workload Identity enabled and manually migrate workloads to that pool. This succeeds only if Workload Identity is enabled on the cluster.
gcloud container node-pools create NODEPOOL_NAME \
--cluster=CLUSTER_NAME \
--workload-metadata=GKE_METADATA
If a cluster has Workload Identity enabled, you can selectively disable it on a
specific node pool by explicitly specifying
--workload-metadata=GCE_METADATA
. See Protecting cluster
metadata for more
information.
Option 2: Node pool modification
Modify an existing node pool to enable GKE_METADATA
. This update succeeds
only if Workload Identity is enabled on the cluster. It immediately enables
Workload Identity for workloads deployed to the node pool. This change will prevent
workloads from using the Compute Engine service account
and must be carefully rolled out.
gcloud container node-pools update NODEPOOL_NAME \
--cluster=CLUSTER_NAME \
--workload-metadata=GKE_METADATA
This action requires container.nodes.update
permissions on the project.
Authenticating to Google Cloud
This section explains how an application can authenticate to Google Cloud using Workload Identity. To do this, assign a Kubernetes service account to the application and configure it to act as a Google service account:
Configure
kubectl
to communicate with the cluster:gcloud container clusters get-credentials CLUSTER_NAME
Replace
CLUSTER_NAME
with the name of the cluster you created in the previous step.This action requires
container.clusters.get
permission on the project.Like most other resources, Kubernetes service accounts live in a namespace. Create the namespace to use for the Kubernetes service account.
kubectl create namespace K8S_NAMESPACE
This action requires Create namespace RBAC permission within the cluster.
Create the Kubernetes service account to use for your application:
kubectl create serviceaccount --namespace K8S_NAMESPACE KSA_NAME
Replace the following:
K8S_NAMESPACE
: the name of the Kubernetes namespace you created in the previous step.KSA_NAME
: the name you would like to use for the Kubernetes service account.
This action requires create serviceaccounts RBAC permission within the namespace.
Alternatively, you can use the default namespace or the default Kubernetes service account in any namespace.
Create a Google service account for your application. If you have an existing service account, you can use it instead of creating a new service account. The service account doesn't need to be in the same project as your cluster. You can use any Google service account in your organization.
gcloud
Replace
GSA_NAME
with the name you choose for the service account.gcloud iam service-accounts create GSA_NAME
Config Connector
If you have Config Connector already installed on a cluster, you can create a new GKE cluster with Workload Identity enabled using a Config Connector configuration.
Note: This step requires Config Connector. Follow the installation instructions to install Config Connector on your cluster.
To deploy this manifest, download it to your machine asservice-account.yaml
. ReplaceGSA_NAME
with the name you choose for the service account. Then usekubectl
to apply the manifest.kubectl apply -f service-account.yaml
This action requires
iam.serviceAccounts.create
permission on the project.For information on authorizing Google service accounts to access Google Cloud APIs, see Understanding service accounts.
Allow the Kubernetes service account to impersonate the Google service account by creating an IAM policy binding between the two. This binding allows the Kubernetes Service account to act as the Google service account.
gcloud
gcloud iam service-accounts add-iam-policy-binding \ --role roles/iam.workloadIdentityUser \ --member "serviceAccount:PROJECT_ID.svc.id.goog[K8S_NAMESPACE/KSA_NAME]" \ GSA_NAME@PROJECT_ID.iam.gserviceaccount.com
Config Connector
Note: This step requires Config Connector. Follow the installation instructions to install Config Connector on your cluster.
To deploy this manifest, download it to your machine aspolicy-binding.yaml
. ReplaceGSA_NAME
,PROJECT_ID
,K8S_NAMESPACE
andKSA_NAME
the values for your environment. Then, run:kubectl apply -f policy-binding.yaml
This action requires
iam.serviceAccounts.setIamPolicy
permission on the project.Add the
iam.gke.io/gcp-service-account=GSA_NAME@PROJECT_ID
annotation to the Kubernetes service account, using the email address of the Google service account.kubectl
kubectl annotate serviceaccount \ --namespace K8S_NAMESPACE \ KSA_NAME \ iam.gke.io/gcp-service-account=GSA_NAME@PROJECT_ID.iam.gserviceaccount.com
yaml
apiVersion: v1 kind: ServiceAccount metadata: annotations: iam.gke.io/gcp-service-account: GSA_NAME@PROJECT_ID.iam.gserviceaccount.com name: KSA_NAME namespace: K8S_NAMESPACE
This action requires RBAC edit permissions on the Kubernetes service account.
Verify the service accounts are configured correctly by creating a Pod with the Kubernetes service account that runs the
cloud-sdk
container image, and connecting to it with an interactive session.kubectl
kubectl run -it \ --image google/cloud-sdk:slim \ --serviceaccount KSA_NAME \ --namespace K8S_NAMESPACE \ workload-identity-test
yaml
apiVersion: v1 kind: Pod metadata: name: workload-identity-test namespace: K8S_NAMESPACE spec: containers: - image: google/cloud-sdk:slim name: workload-identity-test command: ["sleep","infinity"] serviceAccountName: KSA_NAME
The
google/cloud-sdk
image includes thegcloud
command-line tool which is a convenient way to consume Google Cloud APIs. It may take some time to download the image.This action requires create Pods RBAC permission within the namespace.
You are now connected to an interactive shell within the created Pod. Run the following command inside the Pod:
gcloud auth list
If the service accounts are correctly configured, the Google service account email address is listed as the active (and only) identity. This demonstrates that by default, the Pod uses the Google service account's authority when calling Google Cloud APIs.
Using Workload Identity from your code
Authenticating to Google Cloud services from your code is the same process as authenticating using the Compute Engine metadata server. When you use Workload Identity, your requests to the instance metadata server are routed to the GKE metadata server. Existing code that authenticates using the instance metadata server (like code using the Google Cloud client libraries) should work without modification.
Understanding the GKE metadata server
The GKE metadata server is a new metadata server
designed for use with Kubernetes. It runs as a daemonset
, with one Pod on each cluster node. The metadata server intercepts HTTP requests to
http://metadata.google.internal (169.254.169.254:80
), including requests like
GET /computeMetadata/v1/instance/service-accounts/default/token
to retrieve a
token for the Google service account the Pod is configured to act as. Traffic to the metadata
server never leaves the VM instance that hosts the Pod.
The GKE metadata server implements only a subset of Compute Engine metadata server endpoints which are relevant and safe for Kubernetes workloads:
/computeMetadata/v1/instance/attributes/cluster-location
/computeMetadata/v1/instance/attributes/cluster-name
/computeMetadata/v1/instance/attributes/cluster-uid
/computeMetadata/v1/instance/hostname
/computeMetadata/v1/instance/id
/computeMetadata/v1/project/numeric-project-id
/computeMetadata/v1/project/project-id
/computeMetadata/v1/instance/service-accounts
/computeMetadata/v1/instance/service-accounts/default
/computeMetadata/v1/instance/service-accounts/default/aliases
/computeMetadata/v1/instance/service-accounts/default/email
/computeMetadata/v1/instance/service-accounts/default/identity
/computeMetadata/v1/instance/service-accounts/default/identity?audience=audience
/computeMetadata/v1/instance/service-accounts/default/scopes
/computeMetadata/v1/instance/service-accounts/default/token
/computeMetadata/v1/instance/service-accounts/default/token?scopes=comma-separated-list-of-scopes
Revoking access
Revoke access to the Google service account using IAM:
gcloud
gcloud iam service-accounts remove-iam-policy-binding \ --role roles/iam.workloadIdentityUser \ --member "serviceAccount:PROJECT_ID.svc.id.goog[K8S_NAMESPACE/KSA_NAME]" \ GSA_NAME@GSA_PROJECT_ID.iam.gserviceaccount.com
Replace the following:
PROJECT_ID
: the project ID container the GKE cluster.K8S_NAMESPACE
: the name of the Kubernetes namespace where your Kubernetes service account is located.KSA_NAME
: the name of the Kubernetes service account that will have its access revoked.GSA_NAME
: the name of the Google service account.GSA_PROJECT_ID
: the project ID containing the Google service account.
Config Connector
If you used Config Connector to create the service account, delete the service account with
kubectl
.kubectl delete -f service-account.yaml
This action requires
iam.serviceAccounts.setIamPolicy
permissions on the service account.It can take up to 30 minutes for cached tokens to expire. You can check whether the cached tokens have expired with this command:
gcloud auth list
The cached tokens have expired if the output of that command no longer includes
GSA_NAME@PROJECT_ID.iam.gserviceaccount.com
.Remove the annotation from the Kubernetes service account. This step is optional because access has been revoked by IAM.
kubectl annotate serviceaccount \ --namespace K8S_NAMESPACE \ KSA_NAME \ iam.gke.io/gcp-service-account=GSA_NAME@PROJECT_ID.iam.gserviceaccount.com
Troubleshooting
If your application can't authenticate to Google Cloud, ensure these settings are configured properly:
Ensure that you have enabled the IAM Service Account Credentials API in the project containing the GKE cluster.
Ensure that Workload Identity is enabled on the cluster by verifying that it has a workload identity pool set:
gcloud container clusters describe CLUSTER_NAME \ --format="value(workloadIdentityConfig.workloadPool)"
Ensure that GKE metadata server (GKE_METADATA) is configured on the node pool where your application is running:
gcloud container node-pools describe NODEPOOL_NAME \ --cluster=CLUSTER_NAME \ --format="value(config.workloadMetadataConfig.mode)"
Ensure that the Kubernetes service account is annotated correctly
kubectl describe serviceaccount \ --namespace K8S_NAMESPACE \ KSA_NAME
There should be an annotation in the following format:
iam.gke.io/gcp-service-account: GSA_NAME@PROJECT_ID.iam.gserviceaccount.com
Ensure the Google service account is configured correctly
gcloud iam service-accounts get-iam-policy \ GSA_NAME@PROJECT_ID.iam.gserviceaccount.com
Verify that there is a binding with in the following format:
- members: - serviceAccount:PROJECT_ID.svc.id.goog[K8S_NAMESPACE/KSA_NAME] role: roles/iam.workloadIdentityUser
If you have a cluster network policy , ensure egress to 127.0.0.1/32 on port 988 is allowed:
kubectl describe networkpolicy NETWORK_POLICY_NAME
Disabling Workload Identity on a cluster
Disable Workload Identity on each node pool:
gcloud container node-pools update NODEPOOL_NAME \ --cluster=CLUSTER_NAME \ --workload-metadata=GCE_METADATA
Repeat this command for every node pool in the cluster.
Disable Workload Identity in the cluster:
gcloud container clusters update CLUSTER_NAME \ --disable-workload-identity
This action requires
container.clusters.update
permissions on the cluster.
Disabling Workload Identity in your organization
From a security perspective, Workload Identity allows GKE to assert Kubernetes service account identities that can be authenticated and authorized to Google Cloud resources. Administrators who have taken actions to isolate workloads from Google Cloud resources, like disabling service account creation or disabling service account key creation, might also wish to disable Workload Identity for your organization.
See these instructions for disabling Workload Identity for your organization.
Alternatives to Workload Identity
There are two alternative methods to access Cloud APIs from GKE. With the release of Workload Identity we no longer recommend these approaches because of the compromises they require.
Export service account keys and store them as Kubernetes Secrets. Google service account keys expire after 10 years and are rotated manually. Exporting service account keys has the potential to expand the scope of a security breach if it goes undetected.
Use the Compute Engine service account of your nodes. You can run node pools as any IAM service account in your project. If you do not specify a service account during node pool creation, GKE will use the Compute Engine default service account of the project. The Compute Engine service account is shared by all workloads deployed on that node. This can result in over-provisioning of permissions.
What's next
- Read the GKE security overview.
- Learn about Pods.
- Learn about Protecting Cluster Metadata.
- Learn about Google service accounts.