Enabling Workload Identity with Apigee hybrid

This topic explains how to enable Workload Identity for Apigee hybrid.


Workload Identity is a way for applications running within GKE (Google Kubernetes Engine) to access Google Cloud services. For overviews of Workload Identity, see:

A Google Cloud IAM service account is an identity that an application can use to make requests to Google APIs. These service accounts are referred to as GSA (Google Service Accounts) in the document. For more information about GSAs, see Service accounts.

Separately, Kubernetes also has the concept of service accounts. A service account provides an identity for processes that run in a Pod. Kubernetes service accounts are Kubernetes resources, while Google service accounts are specific to Google Cloud. For information on Kubernetes service accounts, see Configure Service Accounts for Pods in the Kubernetes documentation.

With Apigee hybrid 1.4 and later, Apigee creates and uses a Kubernetes service account for each type of component. Enabling Workload Identity allows the hybrid components to interact with the Kubernetes service accounts.


Before enabling Workload Identity for Apigee hybrid, Workload Identity must be enabled for the GKE cluster running Apigee.

If you followed the instructions for Anthos Service Mesh (ASM), Workload Identity is already enabled for the cluster.

You can confirm if Workload Identity is enabled by running the following command:

gcloud container clusters describe $CLUSTER_NAME

The output should include something like the following:

status: RUNNING
subnetwork: default
  workloadPool: PROJECT_ID.svc.id.goog

When running Apigee hybrid on GKE, the standard practice is to create and download private keys (.json files) for each of the service accounts. When using Workload Identity, you do not need to download service account private keys and add them to GKE clusters.

Enable Workload Identity for Apigee hybrid

To begin, follow the instructions in Prepare to enable Workload Identity to update node pools and initialize variables prior to enabling Workload Identity.

Then, follow one of these sections, depending on the steps you're performing:

Prepare to enable Workload Identity

Before starting the install process, follow the steps in this section.

Initialize variables

Initialize (or verify) the following variables:

export PROJECT_ID=my-project-id
export ENV_NAME=my-environment-name
export NAMESPACE=apigee #the namespace where apigee is installed

Set the project to the project you're modifying:

gcloud config set project $PROJECT_ID

Update node pools

Ensure Workload Identity is enabled per node pool with the following command:

gcloud container node-pools update $NODE_POOL_NAME \
  --cluster=$CLUSTER_NAME \

Enable Workload Identity for a fresh install

Follow these instructions when enabling Workload Identity during a new hybrid installation.

  1. Add the bolded line below to your overrides.yaml file under the gcp stanza:
      projectID: "my-project"
      name: "my-project"
      region: "us-west1"
      workloadIdentityEnabled: true
  2. Create Google Service Accounts. There are two methods you can use:
    • Use the create-service-account tool to create a Google service account per component with the following command:
      $APIGEECTL_HOME/tools/create-service-account --env prod --dir $APIGEECTL_HOME/../service-accounts

      This command will create the following service accounts:

      • apigee-logger
      • apigee-metrics
      • apigee-cassandra
      • apigee-udca
      • apigee-synchronizer
      • apigee-mart
      • apigee-watcher
      • apigee-distributed-trace

      The following instructions assume you used the create-service-account tool to generate the service accounts.

    • Define custom naming conventions and service accounts per environment (for advanced users).
  3. Create the Kubernetes Service Accounts.

    For org-level components:

    • Cassandra
      kubectl create sa -n $NAMESPACE apigee-cassandra-schema-setup-$(apigeectl encode --org $ORG_NAME)-sa \
        && kubectl label sa -n $NAMESPACE -l app=apigee-cassandra,org=$ORG_NAME \
        && kubectl annotate serviceaccount \
        --namespace $NAMESPACE apigee-cassandra-schema-setup-$(apigeectl encode --org $ORG_NAME)-sa iam.gke.io/gcp-service-account=GSA_NAME@$PROJECT_ID.iam.gserviceaccount.com
      kubectl create sa -n $NAMESPACE apigee-cassandra-user-setup-$(apigeectl encode --org $ORG_NAME)-sa \
        && kubectl label sa -n $NAMESPACE -l app=apigee-cassandra,org=$ORG_NAME \
        && kubectl annotate serviceaccount \
        --namespace $NAMESPACE apigee-cassandra-user-setup-$(apigeectl encode --org $ORG_NAME)-sa iam.gke.io/gcp-service-account=GSA_NAME@$PROJECT_ID.iam.gserviceaccount.com
    • MART
      kubectl create sa -n $NAMESPACE apigee-mart-$(apigeectl encode --org $ORG_NAME)-sa \
        && kubectl label sa -n $NAMESPACE -l app=apigee-mart,org=$ORG_NAME \
        && kubectl annotate serviceaccount \
        --namespace $NAMESPACE apigee-mart-$(apigeectl encode --org $ORG_NAME)-sa iam.gke.io/gcp-service-account=GSA_NAME@$PROJECT_ID.iam.gserviceaccount.com
    • Apigee Connect
      kubectl create sa -n $NAMESPACE apigee-connect-$(apigeectl encode --org $ORG_NAME)-sa \
        && kubectl label sa -n $NAMESPACE -l app=apigee-connect,org=$ORG_NAME \
        && kubectl annotate serviceaccount \
        --namespace $NAMESPACE apigee-connect-$(apigeectl encode --org $ORG_NAME)-sa iam.gke.io/gcp-service-account=GSA_NAME@$PROJECT_ID.iam.gserviceaccount.com
    • Apigee Watcher
      kubectl create sa -n $NAMESPACE apigee-watcher-$(apigeectl encode --org $ORG_NAME)-sa \
        && kubectl label sa -n $NAMESPACE -l app=apigee-watcher,org=$ORG_NAME \
        && kubectl annotate serviceaccount \
        --namespace $NAMESPACE apigee-watcher-$(apigeectl encode --org $ORG_NAME)-sa iam.gke.io/gcp-service-account=GSA_NAME@$PROJECT_ID.iam.gserviceaccount.com
    • Apigee metrics
      kubectl create sa -n $NAMESPACE apigee-metrics-sa \
        && kubectl label sa -n $NAMESPACE -l app=apigee-metrics,org=$ORG_NAME \
        && kubectl annotate serviceaccount \
        --namespace $NAMESPACE apigee-metrics-sa iam.gke.io/gcp-service-account=GSA_NAME@$PROJECT_ID.iam.gserviceaccount.com

    For each environment:

    • Runtime
      kubectl create sa -n $NAMESPACE apigee-runtime-$(apigeectl encode --org $ORG_NAME --env $ENV_NAME)-sa \
        && kubectl label sa -n $NAMESPACE -l app=apigee-runtime,org=$ORG_NAME,env=$ENV_NAME \
        && kubectl annotate serviceaccount \
        --namespace $NAMESPACE apigee-runtime-$(apigeectl encode --org $ORG_NAME --env $ENV_NAME)-sa iam.gke.io/gcp-service-account=GSA_NAME@$PROJECT_ID.iam.gserviceaccount.com
    • UDCA
      kubectl create sa -n $NAMESPACE apigee-udca-$(apigeectl encode --org $ORG_NAME --env $ENV_NAME)-sa \
        && kubectl label sa -n $NAMESPACE -l app=apigee-udca,org=$ORG_NAME,emv=$ENV_NAME \
        && kubectl annotate serviceaccount \
        --namespace $NAMESPACE apigee-udca-$(apigeectl encode --org $ORG_NAME --env $ENV_NAME)-sa iam.gke.io/gcp-service-account=GSA_NAME@$PROJECT_ID.iam.gserviceaccount.com
    • Synchronizer
      kubectl create sa -n $NAMESPACE apigee-synchronizer-$(apigeectl encode --org $ORG_NAME --env $ENV_NAME)-sa \
        && kubectl label sa -n $NAMESPACE -l app=apigee-synchronizer,org=$ORG_NAME,env=$ENV_NAME \
        && kubectl annotate serviceaccount \
        --namespace $NAMESPACE apigee-synchronizer-$(apigeectl encode --org $ORG_NAME --env $ENV_NAME)-sa iam.gke.io/gcp-service-account=GSA_NAME@$PROJECT_ID.iam.gserviceaccount.com
  4. Continue installing Apigee hybrid as you normally would.

Upgrade an install to use Workload Identity

Follow these instructions to add Workload Identity to an existing hybrid installation.

Here is a sample showing Google Service Accounts (GSA) created for Apigee:

gcloud iam service-accounts list | grep apigee
apigee-connect   apigee-connect@$PROJECT_ID.iam.gserviceaccount.com    False
apigee-runtime   apigee-runtime@$PROJECT_ID.iam.gserviceaccount.com    False
apigee-metrics   apigee-metrics@$PROJECT_ID.iam.gserviceaccount.com    False
apigee-mart      apigee-mart@$PROJECT_ID.iam.gserviceaccount.com       False
apigee-watcher   apigee-watcher@$PROJECT_ID.iam.gserviceaccount.com    False
apigee-sync      apigee-sync@$PROJECT_ID.iam.gserviceaccount.com       False
apigee-udca      apigee-udca@$PROJECT_ID.iam.gserviceaccount.com       False

Here is a sample Kubernetes Service Accounts (KSA) created for Apigee (assumes Apigee hybrid 1.4 or higher is installed):

kubectl get sa -n $NAMESPACE
apigee-cassandra-schema-setup-ORG_NAME-cb84b88-sa                1         xxd
apigee-cassandra-user-setup-ORG_NAME-cb84b88-sa                  1         xxd
apigee-connect-agent-ORG_NAME-cb84b88-sa                         1         xxd
apigee-init                                                        1         xxd
apigee-mart-ORG_NAME-cb84b88-sa                                  1         xxd
apigee-metrics-apigee-telemetry                                    1         xxd
apigee-runtime-ORG_NAME-ENV_NAME-1d0dc5e-sa                    1         xxd
apigee-synchronizer-ORG_NAME-ENV_NAME-1d0dc5e-sa               1         xxd
apigee-udca-ORG_NAME-ENV_NAME-1d0dc5e-sa                       1         xxd
apigee-watcher-ORG_NAME-cb84b88                                  1         xxd
  1. Add the Workload Identity role to each service account:
    gcloud iam service-accounts add-iam-policy-binding \
      --role roles/iam.workloadIdentityUser \
      --member "serviceAccount:$PROJECT_ID.svc.id.goog[$NAMESPACE/$KSA_NAME]" \

    For example, if you were setting the permissions for Apigee Synchronizer, you would run:

    export KSA_NAME=$(kubectl get sa -n apigee -l app=apigee-synchronizer,env=$ENV_NAME,org=$ORG_NAME --output=jsonpath={.items..metadata.name})
    gcloud iam service-accounts add-iam-policy-binding --role roles/iam.workloadIdentityUser --member "serviceAccount:$PROJECT_ID.svc.id.goog[apigee/$KSA_NAME]" apigee-sync@$PROJECT_ID.iam.gserviceaccount.com
  2. Annotate each KSA with the GSA details:
    kubectl annotate serviceaccount \
      --namespace $NAMESPACE \
      $KSA_NAME \

    For example, if you were setting the permissions for Apigee Synchronizer, you would run:

    export KSA_NAME=$(kubectl get sa -n apigee -l app=apigee-synchronizer,env=$ENV_NAME,org=$ORG_NAME --output=jsonpath={.items..metadata.name})
    kubectl annotate serviceaccount --namespace $NAMESPACE $KSA_NAME iam.gke.io/gcp-service-account=apigee-sync@$PROJECT_ID.iam.gserviceaccount.com
  3. Validate if the steps worked:
    gcloud config set project $PROJECT_ID
    kubectl run --rm -it --image google/cloud-sdk:slim --serviceaccount $KSA_NAME --namespace $NAMESPACE workload-identity-test -- gcloud auth list

    If you don't see a command prompt, try pressing Enter.

    If the steps were correctly run, you should see a response like the following:

                       Credentialed Accounts
    *       GSA@PROJECT_ID.iam.gserviceaccount.com
  4. If upgrading from a previous install, clean up secrets that contained service account private keys:
    kubectl delete secrets -n $NAMESPACE $(k get secrets -n $NAMESPACE | grep svc-account | awk '{print $1}')
  5. Check logs:
    kubectl logs -n $NAMESPACE -l app=apigee=synchronizer,env=$ENV_NAME,org=$ORG_NAME apigee-synchronizer