Enabling Workload Identity with Apigee hybrid

This topic explains how to enable Workload Identity for Apigee hybrid.

Overview

Workload Identity is a way for applications running within GKE (Google Kubernetes Engine) to access Google Cloud services. For overviews of Workload Identity, see:

A Google Cloud IAM service account is an identity that an application can use to make requests to Google APIs. These service accounts are referred to as GSA (Google Service Accounts) in the document. For more information about GSAs, see Service accounts.

Separately, Kubernetes also has the concept of service accounts. A service account provides an identity for processes that run in a Pod. Kubernetes service accounts are Kubernetes resources, while Google service accounts are specific to Google Cloud. For information on Kubernetes service accounts, see Configure Service Accounts for Pods in the Kubernetes documentation.

With Apigee hybrid 1.4 and later, Apigee creates and uses a Kubernetes service account for each type of component. Enabling Workload Identity allows the hybrid components to interact with the Kubernetes service accounts.

Google service accounts in Apigee hybrid without Workload Identity

Without using Workload Identity, you must link the Google service accounts to each component in your overrides.yaml file with either a reference to a cert file or a Kubernetes secret. For example:

Environment variables used in these procedures

These procedures make use of the following environment variables. Either set these in your command shell or replace them in the code samples with the actual values:

  • PROJECT_ID: The ID of your Google Cloud project.
  • ORG_NAME: The name of your Apigee organization.
  • ENV_NAME: The name of the Apigee environment.
  • NAMESPACE: The name of your Apigee namespace. This is usually apigee.
  • CLUSTER_NAME: The name of your cluster.
  • CLUSTER_LOCATION: The region or zone of your cluster, for example: us-west1.
  • APIGEECTL_HOME: The directory where you have installed apigeectl.
  • HYBRID_FILES: Your hybrid files directory, for example hybrid-base-directory/hybrid-files.

Verify the environment variables:

echo $PROJECT_ID
echo $ORG_NAME
echo $ENV_NAME
echo $NAMESPACE
echo $CLUSTER_NAME
echo $CLUSTER_LOCATION
echo $APIGEECTL_HOME
echo $HYBRID_FILES

Initialize any of the variables you need:

export PROJECT_ID=my-project-id
export ORG_NAME=$PROJECT_ID
export ENV_NAME=my-environment-name
export NAMESPACE=apigee
export CLUSTER_NAME=my-cluster-name
export CLUSTER_LOCATION=my-cluster-location
export APIGEECTL_HOME=hybrid-base-directory/apigeectl
export HYBRID_FILES=hybrid-base-directory/hybrid-files

Workload Identity and service account key files

When running Apigee hybrid on GKE, the standard practice is to create and download private keys (.json files) for each of the service accounts. When using Workload Identity, you do not need to download service account private keys and add them to GKE clusters.

If you have downloaded service account key files as part of your Apigee hybrid installation, you can delete them after enabling Workload Identity. In most installations, they reside in the hybrid-base-directory/hybrid-files/service-accounts/ directory.

Enable Workload Identity for Apigee hybrid

To begin, follow the instructions in Prepare to enable Workload Identity to update node pools and initialize variables prior to enabling Workload Identity.

Then, follow the instructions in Set workloadIdentityEnabled: true and create service accounts to enable Workload Identity on your Apigee hybrid installation.

Prepare to enable Workload Identity

Before starting the install process, follow the steps in this section.

  1. Set the project to the project you're modifying:
    gcloud config set project $PROJECT_ID
  2. Get the gcloud credentials of the cluster on which you are enabling Workload Identity with the following command:
    gcloud container clusters get-credentials ${CLUSTER_NAME} \
      --region ${CLUSTER_LOCATION} \
      --project ${PROJECT_ID}
  3. Verify that Workload Identity is enabled for the GKE cluster running Apigee with the following command:
    gcloud container clusters describe $CLUSTER_NAME --region $CLUSTER_LOCATION --project $PROJECT_ID

    The output should include something like the following:

    …
      …
      status: RUNNING
      subnetwork: default
      workloadIdentityConfig:
        workloadPool: my-project-id.svc.id.goog

    If needed, enable workload identity on the cluster. This operation can take up to 30 minutes:

      gcloud container clusters update $CLUSTER_NAME \
      --workload-pool=$PROJECT_ID.svc.id.goog \
      --project $PROJECT_ID \
      --region $CLUSTER_LOCATION

    For more information, see Enable Workload Identity.

  4. Ensure Workload Identity is enabled on each node pool.
    1. List your node pools with the following command:
      gcloud container node-pools list \
        --cluster $CLUSTER_NAME \
        --region $CLUSTER_LOCATION \
        --project $PROJECT_ID

      Your output should look something like:

        NAME            MACHINE_TYPE   DISK_SIZE_GB  NODE_VERSION
        apigee-runtime  e2-standard-4  100           1.23.12-gke.100
        apigee-data     e2-standard-4  100           1.23.12-gke.100
    2. Ensure Workload Identity is enabled per node pool using the following command for each node pool:
      gcloud container node-pools update NODE_POOL_NAME \
        --cluster=$CLUSTER_NAME \
        --region $CLUSTER_LOCATION \
        --project $PROJECT_ID \
        --workload-metadata=GKE_METADATA

      Where NODE_POOL_NAME is the name of each node pool.

Verify or create Google service accounts

Google service accounts are created for many Apigee hybrid components during installation. Use this procedure to verify the Google service accounts and create any that are needed.

  1. Check the names of the Google service accounts for your project with the following command:
    gcloud iam service-accounts list --project $PROJECT_ID

    Your output should look something like:

    Prod

    For non-production environments:

    DISPLAY NAME         EMAIL                                                      DISABLED
    apigee-cassandra     apigee-cassandra@my_project_id.iam.gserviceaccount.com     False
    apigee-logger        apigee-logger@my_project_id.iam.gserviceaccount.com        False
    apigee-mart          apigee-mart@my_project_id.iam.gserviceaccount.com          False
    apigee-metrics       apigee-metrics@my_project_id.iam.gserviceaccount.com       False
    apigee-runtime       apigee-runtime@my_project_id.iam.gserviceaccount.com       False
    apigee-synchronizer  apigee-synchronizer@my_project_id.iam.gserviceaccount.com  False
    apigee-udca          apigee-udca@my_project_id.iam.gserviceaccount.com          False
    apigee-watcher       apigee-watcher@my_project_id.iam.gserviceaccount.com       False
    

    Non-prod

    For non-production environments:

    DISPLAY NAME         EMAIL                                                      DISABLED
    apigee-non-prod      apigee-non-prod@my_project_id.iam.gserviceaccount.com      False
    

    If you need to create the Google service accounts for your project, there are two methods you can use:

    • Use the create-service-account tool included with apigee in the apigeectl/tools/ directory. This tool can create all the service accounts with a single command or let you create them individually.
    • Use the gcloud iam service-accounts create command to create the service accounts one at a time and the gcloud projects add-iam-policy-binding command to assign the appropriate roles to each service account. This method requires you to create each service account individually, but lets you avoid downloading the key files.

    create-service-account

    Use the following command to create a Google service account per component:

    Prod

    For non-production environments:

    $APIGEECTL_HOME/tools/create-service-account --env prod --dir $APIGEECTL_HOME/../service-accounts

    This command will create the following service accounts:

    • apigee-cassandra
    • apigee-logger
    • apigee-mart
    • apigee-metrics
    • apigee-runtime
    • apigee-synchronizer
    • apigee-udca
    • apigee-watcher

    Non-prod

    For non-production environments:

    $APIGEECTL_HOME/tools/create-service-account --env non-prod --dir $APIGEECTL_HOME/../service-accounts

    This command will create a single service accounts, apigee-non-prod, with all the roles assigned needed to manage all Apigee components.

    gcloud

    Create the following service accounts and assign roles to them.

    1. Service account: apigee-cassandra, role: roles/storage.objectAdmin

      Create the account:

      gcloud iam service-accounts create apigee-cassandra \
        --display-name="apigee-cassandra" \
        --project $PROJECT_ID
                  

      Assign the role:

      gcloud projects add-iam-policy-binding $PROJECT_ID \
        --member="serviceAccount:apigee-cassandra@$PROJECT_ID.iam.gserviceaccount.com" \
        --role="roles/storage.objectAdmin"
                  
    2. Service account: apigee-logger, role: roles/logging.logWriter

      Create the account:

      gcloud iam service-accounts create apigee-logger \
        --display-name="apigee-logger" \
        --project $PROJECT_ID
                  

      Assign the role:

      gcloud projects add-iam-policy-binding $PROJECT_ID \
        --member="serviceAccount:apigee-logger@$PROJECT_ID.iam.gserviceaccount.com" \
        --role="roles/logging.logWriter"
                  
    3. Service account: apigee-mart, role: roles/apigeeconnect.Agent

      Create the account:

      gcloud iam service-accounts create apigee-mart \
        --display-name="apigee-mart" \
        --project $PROJECT_ID
                  

      Assign the role:

      gcloud projects add-iam-policy-binding $PROJECT_ID \
        --member="serviceAccount:apigee-mart@$PROJECT_ID.iam.gserviceaccount.com" \
        --role="roles/apigeeconnect.Agent"
                  
    4. Service account: apigee-metrics, role: roles/monitoring.metricWriter

      Create the account:

      gcloud iam service-accounts create apigee-metrics \
        --display-name="apigee-metrics" \
        --project $PROJECT_ID
                  

      Assign the role:

      gcloud projects add-iam-policy-binding $PROJECT_ID \
        --member="serviceAccount:apigee-metrics@$PROJECT_ID.iam.gserviceaccount.com" \
        --role="roles/monitoring.metricWriter"
                  
    5. Service account: apigee-runtime, role: No role assigned.

      Create the account:

      gcloud iam service-accounts create apigee-runtime \
        --display-name="apigee-runtime" \
        --project $PROJECT_ID
                  
    6. Service account: apigee-synchronizer, role: roles/apigee.synchronizerManager

      Create the account:

      gcloud iam service-accounts create apigee-synchronizer \
        --display-name="apigee-synchronizer" \
        --project $PROJECT_ID
                  

      Assign the role:

      gcloud projects add-iam-policy-binding $PROJECT_ID \
        --member="serviceAccount:apigee-synchronizer@$PROJECT_ID.iam.gserviceaccount.com" \
        --role="roles/apigee.synchronizerManager"
                  
    7. Service account: apigee-udca, role: roles/apigee.analyticsAgent

      Create the account:

      gcloud iam service-accounts create apigee-udca \
        --display-name="apigee-udca" \
        --project $PROJECT_ID
                  

      Assign the role:

      gcloud projects add-iam-policy-binding $PROJECT_ID \
        --member="serviceAccount:apigee-udca@$PROJECT_ID.iam.gserviceaccount.com" \
        --role="roles/apigee.analyticsAgent"
                  
    8. Service account: apigee-watcher, role: roles/apigee.runtimeAgent

      Create the account:

      gcloud iam service-accounts create apigee-watcher \
        --display-name="apigee-watcher" \
        --project $PROJECT_ID
                  

      Assign the role:

      gcloud projects add-iam-policy-binding $PROJECT_ID \
        --member="serviceAccount:apigee-watcher@$PROJECT_ID.iam.gserviceaccount.com" \
        --role="roles/apigee.runtimeAgent"
                  

Set workloadIdentityEnabled: true and create service accounts

apigeectl creates Kubernetes service accounts for each Apigee hybrid component when you set workloadIdentityEnabled: true in your overrides file and apply the changes.

  1. Add the bolded line below to your overrides.yaml file under the gcp stanza. This enables Workload Identity for your installation and triggers apigeectl to create the Kubernetes service accounts when you apply the configuration:
    gcp:
      projectID: "my-project-id"
      name: "my-project-id"
      region: "analytics-region"
      workloadIdentityEnabled: true
  2. Add the bolded lines below to your overrides.yaml file under the cassandra stanza. This triggers the creation of the apigee-cassandra-backup Kubernetes service account:
    cassandra:
      ...
      backup:
        enabled: true
  3. Apply the changes with apigeectl with the --restore flag:
    $APIGEECTL_HOME/apigeectl apply -f overrides/overrides.yaml --restore
  4. Verify the service accounts with the following command:
    kubectl get sa -n $NAMESPACE

    Your output should look something like the following. The Kubernetes service accounts in bold are the ones you will need to annotate with your Google service accounts:

    NAME                                                         SECRETS   AGE
    apigee-cassandra-backup                                      1         11m
    apigee-cassandra-restore                                     1         11m
    apigee-cassandra-schema-setup-my-project-id-123abcd-sa       1         11m
    apigee-cassandra-schema-val-my-project-id-123abcd            1         11m
    apigee-cassandra-user-setup-my-project-id-123abcd-sa         1         11m
    apigee-connect-agent-my-project-id-123abcd-sa                1         11m
    apigee-datastore-default-sa                                  1         11m
    apigee-ingressgateway                                        1         11m
    apigee-ingressgateway-my-project-id-123abcd                  1         11m
    apigee-ingressgateway-manager                                1         11m
    apigee-init                                                  1         11m
    apigee-mart-my-project-id-123abcd-sa                         1         11m
    apigee-metrics-sa                                            1         11m
    apigee-mint-task-scheduler-my-project-id-123abcd-sa          1         11m
    apigee-redis-default-sa                                      1         11m
    apigee-redis-envoy-default-sa                                1         11m
    apigee-runtime-my-project-id-env-name-234bcde-sa             1         11m
    apigee-synchronizer-my-project-id-env-name-234bcde-sa        1         11m
    apigee-udca-my-project-id-123abcd-sa                         1         11m
    apigee-udca-my-project-id-env-name-234bcde-sa                1         11m
    apigee-watcher-my-project-id-123abcd-sa                      1         11m
    default                                                      1         11m
        

Annotate the Kubernetes service accounts with the Google service accounts

For each Apigee component annotate the corresponding Kubernetes service accounts with the Google service account for the component.

Org-level components

You will have only one instance of each Kubernetes service account for your Apigee org.

Component Kubernetes Service Account Google Service Account
Cassandra apigee-cassandra-backup apigee-cassandra
apigee-cassandra-restore apigee-cassandra
apigee-cassandra-schema-setup-my-project-id-num-id1-sa apigee-cassandra
apigee-cassandra-schema-val-my-project-id-num-id1 apigee-cassandra
apigee-cassandra-user-setup-my-project-id-num-id1-sa apigee-cassandra
apigee-datastore-default-sa apigee-cassandra
Apigee Connect apigee-connect-agent-my-project-id-num-id1-sa apigee-mart
MART apigee-mart-my-project-id-num-id1-sa apigee-mart
Metrics apigee-metrics-sa apigee-metrics
UDCA (org-level) apigee-udca-my-project-id-num-id1-sa apigee-udca
Watcher apigee-watcher-my-project-id-num-id1-sa apigee-watcher

Env-level components

You will have one instance of each Kubernetes service account for each Apigee environment.

Component Kubernetes Service Account Google Service Account
Apigee Runtime apigee-runtime-my-project-id-env-name-num-id2-sa apigee-runtime