Using Kubernetes Workload Identity for client-server authorization
Contributed by Google employees.
Kubernetes service accounts have been the primary mechanism of identity in Kubernetes clusters. They have been usable with Kubernetes (role-based access control) RBAC to control authorization to the Kubernetes API server. Historically, these have been globally scoped credentials, and subject to relatively high replay risk and not usable outside the context of the cluster.
With Kubernetes 1.12 and later, service account credentials can be exposed to workloads with a specific audience applied, using the Service Account Token Volume Projection feature.
A service account token is exposed as a signed JSON Web Token (JWT). The private keys that sign these tokens are specific to the cluster. In Google Kubernetes Engine (GKE), when Workload Identity is enabled, the corresponding public keys are published at a URL that can be derived from the issuer field in the token conforming to the OpenID Connect (OIDC) Discovery standard.
In GKE, the Workload Identity feature allows these identities to also be associated with IAM service accounts. This allows a pod running as a Kubernetes service account to act as the associated service account for authorized access to Google APIs and to services that verify identity based on Google Cloud-specific OIDC.
To make the use of this identity system easier for application developers, the mechanism can be made more transparent to both the client and the server. On the client side, a derived authorized HTTP client can be used that automatically injects the service account token into all outgoing HTTP requests. On the server side, authorization middleware can use OIDC discovery and the cluster's public key to verify the request, before passing on the the business logic of the function/handler. This tutorial demonstrates the use of these conveniences in the Go programming language.
Objectives
- Set up a GKE cluster with Workload Identity.
- Use Service Account Token Volume Projection to provide a cluster-issued OIDC ID token to a workload.
- Use the cluster-specific public keys to verify the workload's ID token in a remote service.
- Associate Workload Identity and namespace with a Google service account.
- Retrieve credentials from the Google service account through Workload Identity association.
- Use the Google-issued ID token to call a Cloud Run service behind IAM invoker authorization.
Before you begin
We recommend that you run all commands in this tutorial in Cloud Shell,
the command-line interface built into the Cloud Console, which includes current versions of packages such as kubectl
and
envsubst
.
If you choose instead to use the Google Cloud SDK, then you need to install the SDK and the
kubectl
and envsubst
packages. This tutorial requires
kubectl
version 1.14 and later with
kustomize integration.
You can check your kubectl
version with this command:
kubectl version
Setup
In this section, you create a project, download the tutorial files, and create a GKE cluster.
- Create a project in the Cloud Console.
- Enable billing for your project.
Clone the repository that contains the tutorial files, set up a branch, and navigate to the tutorial directory:
git clone https://github.com/GoogleCloudPlatform/community.git cd community git checkout workload-id cd tutorials/gke-workload-id-clientserver
(Only required if you are not using Cloud Shell) Set the project ID by replacing
[YOUR_PROJECT_ID]
with your project ID in this command:gcloud config set project [YOUR_PROJECT_ID]
You can skip setting the project variable if you're using Cloud Shell because Cloud Shell will already have this value set for the current project.
Set environment variables:
export PROJECT=$(gcloud config list project --format "value(core.project)" ) export CLUSTER=workload-id export ZONE=us-central1-a gcloud config set compute/zone $ZONE gcloud config set compute/region us-central1
Create the cluster:
gcloud beta container clusters create $CLUSTER \ --workload-pool=$PROJECT.svc.id.goog
The
identity-namespace
flag is part of the enablement of the Workload Identity feature. Currently only the single, project-level namespace is supported.Creation of the cluster can take several minutes.
Retrieve the cluster credentials:
gcloud container clusters get-credentials $CLUSTER
Build and deploy the test server
The test server in this sample is a simple HTTP service that exposes endpoints with and without middleware that enforces header-based request authentication.
Build the container:
gcloud builds submit --tag gcr.io/${PROJECT}/auth-server ./server
Configure the server authorization:
export ISSUER="https://container.googleapis.com/v1/projects/${PROJECT}/locations/${ZONE}/clusters/${CLUSTER}"
The issuer is an OIDC identity provider that the server trusts, in this case a specific cluster.
Deploy the server with Cloud Run:
gcloud run deploy --image gcr.io/${PROJECT}/auth-server --allow-unauthenticated --platform managed auth-server --set-env-vars="ID_ISSUER=${ISSUER},JWT_AUDIENCE=TBD"
Cloud Run offers a convenient way of deploying services with HTTPS, and it also offers its own authorization system that is covered in a later section of this tutorial. In this first case, you simply use Cloud Run as a quick way to deploy a service external to the cluster that can represent any HTTPS service running on the network.
By setting the flag
--allow-unauthenticated
, you are requesting that the service be exposed publicly by the Cloud Run service. However, there is middleware inside the container that will demonstrate authentication using the cluster-issued workload identity.Get the service URL:
export SERVICE_URL=$(gcloud run services describe auth-server --platform managed --format='value(status.address.url)')
Verify authorization of the base path:
curl $SERVICE_URL
The output should be the following:
{"msg": "hello-exposed"}
This base path returns a message that is not guarded by any authorization middleware in the service.
Verify authorization of the
private
endpoint:curl $SERVICE_URL/private
The output should be
401 - Unauthorized
, because theprivate
endpoint checks for an authorization header and validates the request with it.Redeploy the server, now that you know the service URL:
gcloud run deploy --image gcr.io/${PROJECT}/auth-server --allow-unauthenticated --platform managed auth-server --set-env-vars="ID_ISSUER=${ISSUER},JWT_AUDIENCE=${SERVICE_URL}"
Understanding the authorization middleware
The authorization middleware in this sample is in the server/auth.go
file. It uses the
go-oidc
package.
This package and middleware perform the following steps:
- Uses the
ID_ISSUER
environment variable to retrieve the issuer's public keys through OIDC Discovery. GKE publishes the public keys at a public URL (jwks_uri
link in the discovery document athttps://[issuer]/.well-known/openid-configuration
). - Verifies that the token has not expired.
- Verifies that the signature on the JSON Web Token (JWT) sent as a header by the workload was signed by the retrieved issuer's public key.
- Verifies that the
aud
(audience) claim inside the JWT payload matches the service URL set as theJWT_AUDIENCE
environment variable.
The authorization middleware does not perform any authorization actions based on which workload identity has been
verified—only 1) that it is a valid identity issued by a trusted issuer and 2) that the token was generated for use with
this target service. The aud
claim helps to prevent the risk of replay attacks where the same token can be used to
authenticate against a different service.
Deploy the client to the cluster
Because Workload Identity is available to a running workload, it is not easily demonstrated with a local curl
command.
Instead, in this tutorial, you deploy a simple workload that loops and makes HTTP requests to the server.
The workload makes a set of requests to the exposed and guarded paths on the server. It creates two different HTTP clients to make these requests. One is authenticated with the cluster-issued projected service account token, and the other is authenticated with the Google-issued IAM service account ID token.
Build the client:
gcloud builds submit --tag gcr.io/${PROJECT}/auth-client ./client
Configure the client:
cat client/k-project/project-details-template.yaml | envsubst > client/k-project/project-details.yaml
This command uses kustomize to create a project-specific overlay variant of the client Kubernetes resource YAML file. For information about why we recommend that you capture these variables in an overlay, instead of modifying the Kubernetes YAML file dynamically as part of the
apply
operation, see this explanation.The base resources in the configuration are the following:
- A namespace:
id-namespace
- A service account:
gke-sa
- A client configuration map
- A test client pod
- A namespace:
Create the client resources in the cluster:
kubectl create -k client/k-project/
Observe the client logs:
kubectl logs -f auth-client -n id-namespace
Under
##### Projected token client
, you should see both the exposed and guarded API responses in the logs.The guarded response is only returned if the middleware succeeds in validating the JWT as configured above.
Press Ctrl-C to stop logging.
Understanding the projected service account token
In the k-project/project-details.yaml
file is a pod overlay that includes the following:
serviceAccountName: gke-sa
volumes:
- name: client-token
projected:
sources:
- serviceAccountToken:
path: gke-sa
expirationSeconds: 600
audience: https://auth-server-[hash]-uc.a.run.app
The serviceAccountName
value specifies which Kubernetes service account the pod runs as.
The projected:serviceAccountTokens
includes details that expose a signed JWT at a given path, for a specific
audience.
The same Kubernetes service account can be projected multiple times, each for a different audience.
This line in the main.go
file rereads this secret whenever it has expired and injects the value as an
Authorization
header in the form Authorization: Bearer [JWT_VALUE]
:
authTransport, err := transport.NewBearerAuthWithRefreshRoundTripper("", cfg.TokenPath, http.DefaultTransport)
Using Cloud Run Invoker IAM authorization with Workload Identity
Cloud Run offers a built-in authorization feature that acts as an alternative to the authorization middleware used in the previous sections of this tutorial.
When the Cloud Run Invoker IAM feature is employed, the Cloud Run service performs the following checks:
- Is the JWT in an
Authorization
header issued (signed) byhttps://accounts.google.com
? - Is the token un-expired?
- Does the audience claim in the token match the target service?
- Is the
sub
(subject), or principal, of the request a member of the service's IAM invoker role?
The last check—against the service's IAM invoker role—is an additional authorization check not present in the middleware discussed in previous sections of this tutorial, and it checks specific service accounts against a list or Google Group membership.
If all of these checks pass, then the request is passed on to the Cloud Run service. The code in the container now no longer needs to implement any authorization middleware.
Remove the allow-unauthenticated
option for the Cloud Run service
Cloud Run services are secure by default. There is a special member, allUsers
, which can be added to the list of invokers,
which will allow any request to reach the service.
Remove this option:
gcloud run services remove-iam-policy-binding --role=roles/run.invoker --member allUsers auth-server
It may take a minute or more for this change in policy to propagate.
Attempt to reach the "exposed" base path:
curl $SERVICE_URL
You should receive a
403
response.
Create a service account to associate with Workload Identity
A key use of GKE Workload Identity is to provide a Kubernetes service account with a way to act as a Google Cloud service account for accessing Google APIs.
Create the Google Cloud service account in the project:
gcloud iam service-accounts create gke-gsa
Allow workloads running as the
gke-sa
Kubernetes service account in theid-namespace
to generate tokens for this service account:gcloud iam service-accounts add-iam-policy-binding \ --role roles/iam.serviceAccountTokenCreator \ --member "serviceAccount:${PROJECT}.svc.id.goog[id-namespace/gke-sa]" \ gke-gsa@$PROJECT.iam.gserviceaccount.com
Note that this permission applies to a combination of a project, a namespace, and a service account ID. It will apply to any cluster that contains a matching namespace and service account in the project.
Annotate the namespace so that GKE knows that it can make use of the permission that you just granted:
kubectl annotate serviceaccount \ --namespace id-namespace \ gke-sa \ iam.gke.io/gcp-service-account=gke-gsa@$PROJECT.iam.gserviceaccount.com
Now, the Kubernetes workload has the ability to get credentials and act as the Google Cloud service account.
It can get access these credentials from the metadata server, which most Google Cloud client libraries will do transparently. It allows the workload to access both OAuth 2.0 access tokens, as well as the OIDC ID token for the associated Google service account.
Reconfigure the client to use the Google-issued identity
The main.go
client program contains a switch statement that builds an authenticated HTTP client using the credential
provided by the metadata server. In this section, you change an environment variable and then update the client
configuration.
In the
client/k-project/project-details.yaml
file, change the value of an environment variable fromIDENTITY: cluster
toIDENTITY: google
and save the change.Update the client configuration:
kubectl delete configmap client-config -n id-namespace kubectl delete pod auth-client -n id-namespace kubectl apply -k client/k-project
Add the Google Cloud service account to the invoker policy
Add the associated Google Cloud service account to the invoker policy for the Cloud Run service:
gcloud run services add-iam-policy-binding --role=roles/run.invoker --member serviceAccount:gke-gsa@$PROJECT.iam.gserviceaccount.com auth-server
In the client workload configured to use the Google-issued identity, the HTTP client does not use the projected token
as the header, but instead reads an ID token from the metadata server at this URL: http://metadata/computeMetadata/v1/instance/service-accounts/default/identity
To specify the audience, a query string is used: ?format=full&audience=[URL_of_target_service]
The response includes a Google-issued ID token for the associated Google Cloud service account, not the cluster-issued Kubernetes service account as was done above.
Because this is a Google-issued token, it can be checked against the Cloud Run invoker policy.
Check client access
After letting the permissions propagate for a moment, check the output of the client:
kubectl logs -f auth-client -n id-namespace
This time, you should see log lines under ##### mapped GCP SvcAccnt via Metadata server
.
The "exposed" path is reachable. This is the path that the service is exposing without middleware. However, as you saw in a previous section, this path is being guarded by Cloud Run. Since you are seeing a response, it means that the token that the workload retrieved from the metadata server is passing the validation check by Cloud Run.
You won't see a successful response to the private
endpoint because it is guarded twice: once by Cloud Run and
IAM, and once by the authorization middleware in the service that is still configured to check that the token was issued by
the cluster. You can't have a single token "dual issued". The server example here is being used to demonstrate several
scenarios.
Another scenario that is possible, but not illustrated here, is using Google-issued ID tokens (those with an issuer of
https://accounts.google.com
) and verifying them yourself in middleware, using one of the libraries provided by Google.
For information, see
Authenticate with a backend server.
Cleaning up
Delete the client:
kubectl delete -k client/k-project/
Delete the cluster:
gcloud beta container clusters delete $CLUSTER
Delete the Cloud Run service:
gcloud run services delete auth-server
Except as otherwise noted, the content of this page is licensed under the Creative Commons Attribution 4.0 License, and code samples are licensed under the Apache 2.0 License. For details, see our Site Policies. Java is a registered trademark of Oracle and/or its affiliates.