A new version of GKE on AWS was released on November 2. See the release notes for more information.

Creating a user cluster with workload identity

Workload identity lets you bind Kubernetes service accounts to AWS IAM accounts with specific permissions. This topic shows you how to enable workload identity for your GKE on AWS workloads.

Workload identity uses AWS IAM permissions to block unwanted access to cloud resources. With workload identity, you can assign different IAM roles to each workload. This fine-grained control of permissions lets you follow the principle of least privilege.

Without workload identity, you assign AWS IAM roles to your GKE on AWS nodes. All workloads on those nodes then have complete access to the node's permissions.

To enable workload identity for your cluster, you must complete the following steps. The steps are separated by the personas that complete them: a cluster administrator or a developer.

Cluster administrator

  1. Create a Cloud Storage bucket to store OIDC discovery data.
  2. Create a user cluster with workload identity enabled.
  3. Create a webhook on your cluster that applies workload identity credentials to Pods on creation or else manually set environment variables.
  4. Configure the AWS OIDC provider.
  5. Create AWS IAM roles and policies.
Cluster administrator or developer
  1. Create Kubernetes service accounts, and bind AWS policies to them.
Developer
  1. Apply credentials to your Pods.

Personas

This document refers to the following personas:

  • Cluster administrator: This person creates one or more user clusters and creates authentication configuration files for developers who use the clusters.

  • Developer: This person runs workloads on one or more clusters.

Prerequisites

To complete the steps in this document, you must have the following setup:

  • A GKE on AWS management service
  • User clusters that are running a Kubernetes version greater than 1.17.9-gke.2801
  • The following permissions and tools

Permissions

To create a cluster with workload identity enabled, you need the following permissions:

Google Cloud

  • Create a publicly readable Cloud Storage bucket with uniform bucket-level access enabled.
  • Grant management-sa@PROJECT_NAME.iam.gserviceaccount.com read/write permissions to the bucket.

AWS

  • Create an AWS OIDC provider
  • Create AWS IAM roles

Tools

On your local machine, Google recommends having the jq tool installed.

Creating the OIDC discovery bucket

This section is for cluster administrators.

Your AWSCluster needs to store the OIDC discovery data in a publicly accessible Cloud Storage bucket. The bucket includes OIDC discovery configuration and public keys. AWS uses the contents to authenticate the requests from GKE on AWS clusters.

Your bucket must have the following attributes:

If you don't have a bucket with these attributes, create one by using the following gsutil commands:

BUCKET=BUCKET_NAME
gsutil mb -b on gs://${BUCKET}
gsutil iam ch allUsers:objectViewer gs://${BUCKET}

Replace BUCKET_NAME with the name of your new bucket.

Grant the management service account permissions

The Identity and Access Management service account for the GKE on AWS management service needs permissions to read and write objects into this bucket. Grant your management service account permissions by using the following gsutil command.

MANAGEMENT_SA=management-sa@PROJECT_NAME.iam.gserviceaccount.com
gsutil iam ch serviceAccount:${MANAGEMENT_SA}:admin gs://${BUCKET}

Replace PROJECT_NAME with your project name.

Creating a user cluster

This section is for cluster administrators.

Create a user cluster with workload identity enabled

Create a user cluster that contains details about your OIDC discovery bucket. You set this information in your AWSCluster's spec.controlPlane.workloadIdentity.oidcDiscoveryGCSBucket field.

In this example, you create a cluster manually from AWSCluster and AWSNodePool CRDs.

  1. Change to the directory with your GKE on AWS configuration. You created this directory when Installing the management service.

    cd anthos-aws

  2. From your anthos-aws directory, use anthos-gke to switch context to your management service.

    cd anthos-aws
    anthos-gke aws management get-credentials

  3. Open a text editor and copy the following AWSCluster definition into a file named staging-cluster.yaml.

    apiVersion: multicloud.cluster.gke.io/v1
    kind: AWSCluster
    metadata:
      name: CLUSTER_NAME
    spec:
      region: AWS_REGION
      networking:
        vpcID: VPC_ID
        podAddressCIDRBlocks: POD_ADDRESS_CIDR_BLOCKS
        serviceAddressCIDRBlocks: SERVICE_ADDRESS_CIDR_BLOCKS
        ServiceLoadBalancerSubnetIDs: SERVICE_LOAD_BALANCER_SUBNETS
      controlPlane:
        version: GKE_VERSION # Latest version is 1.17.9-gke.2801
        instanceType: AWS_INSTANCE_TYPE
        keyName: SSH_KEY_NAME
        subnetIDs:
        - CONTROL_PLANE_SUBNET_IDS
        securityGroupIDs:
        - CONTROL_PLANE_SECURITY_GROUPS
        iamInstanceProfile: CONTROL_PLANE_IAM_ROLE
        rootVolume:
          sizeGiB: ROOT_VOLUME_SIZE
        etcd:
          mainVolume.sizeGIB: ETCD_VOLUME_SIZE
        databaseEncryption:
          kmsKeyARN: ARN_OF_KMS_KEY
        hub: # Optional
          membershipName: ANTHOS_CONNECT_NAME
        workloadIdentity: # Optional
          oidcDiscoveryGCSBucket: WORKLOAD_IDENTITY_BUCKET
    

    Replace the following:

    • CLUSTER_NAME: the name of your cluster.
    • AWS_REGION: the AWS region where your cluster runs.
    • VPC_ID: the ID of the VPC where your cluster runs.
    • POD_ADDRESS_CIDR_BLOCKS: the CIDR range of IPv4 addresses used by the cluster's Pods. . The range must be within your VPC CIDR address range, but not part of a subnet. For example, 10.2.0.0/16.
    • SERVICE_ADDRESS_CIDR_BLOCKS: the range of IPv4 addresses used by the cluster's Services. . The range must be within your VPC CIDR address range, but not part of a subnet. For example, 10.1.0.0/16.
    • SERVICE_LOAD_BALANCER_SUBNETS: the subnet IDs where GKE on AWS can create public or private load balancers.
    • GKE_VERSION: a Kubernetes version supported by GKE on AWS. The most recent version is 1.17.9-gke.2801.
    • AWS_INSTANCE_TYPE: a supported EC2 instance type.
    • SSH_KEY_NAME: an AWS EC2 key pair.
    • CONTROL_PLANE_SUBNET_IDS: the subnet IDs in the AZs where your control plane instances run.
    • CONTROL_PLANE_SECURITY_GROUPS: a securityGroupID created during the management service installation. You can customize this by adding any securityGroupIDs required to connect to the control plane.
    • CONTROL_PLANE_IAM_PROFILE: name of the AWS EC2 instance profile assigned to control plane replicas.
    • ROOT_VOLUME_SIZE: the size, in gibibyte (GiB), of your control plane root volumes.
    • ETCD_VOLUME_SIZE: the size of your volumes used by etcd.
    • ARN_OF_KMS_KEY: the AWS KMS key used to encrypt cluster Secrets.
    • ANTHOS_CONNECT_NAME: the Connect membership name used to register your cluster. The membership name must be unique. For example, projects/YOUR_PROJECT/locations/global/memberships/CLUSTER-_AME, where YOUR_PROJECT is your Google Cloud project and CLUSTER-NAME is a unique name in your project. This field is optional.
    • WORKLOAD_IDENTITY_BUCKET: the Cloud Storage bucket name containing your workload identity discovery information. This field is optional.
  4. Create one or more AWSNodePools for your cluster. Open a text editor and copy the following AWSCluster definition into a file named staging-nodepools.yaml.

    apiVersion: multicloud.cluster.gke.io/v1
    kind: AWSNodePool
    metadata:
      name: NODE_POOL_NAME
    spec:
      clusterName: AWSCLUSTER_NAME
      version: GKE_VERSION # latest version is 1.17.9-gke.2801
      region: AWS_REGION
      subnetID: AWS_SUBNET_ID
      minNodeCount: MINIMUM_NODE_COUNT
      maxNodeCount: MAXIMUM_NODE_COUNT
      maxPodsPerNode: MAXIMUM_PODS_PER_NODE_COUNT
      instanceType: AWS_NODE_TYPE
      keyName: KMS_KEY_PAIR_NAME
      iamInstanceProfile: NODE_IAM_PROFILE
      rootVolume:
        sizeGiB: ROOT_VOLUME_SIZE 
    

    Replace the following:

    • NODE_POOL_NAME: a unique name for your AWSNodePool.
    • AWSCLUSTER_NAME: your AWSCluster's name. For example, staging-cluster.
    • GKE_VERSION: a GKE on AWS supported GKE version.
    • AWS_REGION: the same AWS region as your AWSCluster.
    • AWS_SUBNET_ID: an AWS subnet in the same region as your AWSCluster. MINIMUM_NODE_COUNT and MAXIMUM_NODE_COUNT with the minimum and maximum number of nodes in the node pool.
    • MINIMUM_NODE_COUNT: the minimum number of nodes in the node pool. See Scaling user clusters for more information.
    • MAXIMUM_NODE_COUNT: the maximum number of nodes in the node pool.
    • MAXIMUM_PODS_PER_NODE_COUNT: the maximum number of pods that GKE on AWS can allocate to a node.
    • AWS_NODE_TYPE: an AWS EC2 instance type.
    • KMS_KEY_PAIR_NAME: the AWS KMS key pair assigned to each node pool worker.
    • NODE_IAM_PROFILE: the name of the AWS EC2 instance profile assigned to nodes in the pool.
    • ROOT_VOLUME_SIZE: the size, in gibibyte (GiB), of your control plane root volumes.

Create a kubeconfig

While your user cluster starts, you can create a kubeconfig context for your new user cluster. You use the context to authenticate to a user or management cluster.

  1. Use anthos-gke aws clusters get-credentials to generate a kubeconfig for your user cluster in ~/.kube/config.

    env HTTP_PROXY=http://localhost:8118 \
      anthos-gke aws clusters get-credentials CLUSTER_NAME
    

    Replace CLUSTER_NAME with your cluster's name. For example, cluster-0.

  2. Use kubectl to authenticate to your new user cluster.

    env HTTP_PROXY=http://localhost:8118 \
      kubectl cluster-info
    

    If your cluster is ready, the output includes the URLs for Kubernetes components within your cluster.

Viewing your cluster's status

The management service provisions AWS resources when you apply an AWSCluster or AWSNodePool.

  1. From your anthos-aws directory, use anthos-gke to switch context to your management service.

    cd anthos-aws
    anthos-gke aws management get-credentials

  2. To list your clusters, use kubectl get AWSClusters.

    env HTTP_PROXY=http://localhost:8118 \
      kubectl get AWSClusters
    

The output includes each cluster's name, state, age, version, and endpoint.

For example, the following output includes only one AWSCluster named cluster-0:

NAME        STATE          AGE     VERSION         ENDPOINT
cluster-0   Provisioning   2m41s   1.17.9-gke.2801   gke-xyz.elb.us-east-1.amazonaws.com

View your cluster's events

To see recent Kubernetes Events from your user cluster, use kubectl get events.

  1. From your anthos-aws directory, use anthos-gke to switch context to your management service.

    cd anthos-aws
    anthos-gke aws management get-credentials

  2. Run kubectl get events.

    env HTTP_PROXY=http://localhost:8118 \
      kubectl get events
    

The output includes information, warning, and errors related to from your management service.

Creating the workload identity webhook or setting environment variables

This section is for cluster administrators.

To provide workload identity credentials to your workloads with no additional configuration, you can optionally create a webhook on your user clusters (AWSCluster). This webhook intercepts Pod creation requests and sets the following AWS IAM information as environment variables:

  • AWS_ROLE_ARN: the Amazon Resource Name (ARN) of the IAM role
  • aws-iam-token: the token exchanged for AWS IAM credentials
  • AWS_WEB_IDENTITY_TOKEN_FILE: the path where the token is stored

With these variables, the AWS command-line interface or SDK running inside the Pod can access the resources granted to the AWS role.

Creating the webhook is optional. If you decide not to create the webhook, you need to set the environment variables listed previously in the Pod. For information about not using a webhook, see Applying credentials without the webhook.

Create the YAML files for the webhook

To deploy the webhook, perform the following steps:

  1. From your anthos-aws directory, use anthos-gke to switch context to your management service.

    cd anthos-aws
    anthos-gke aws management get-credentials

  2. Get the user cluster name with kubectl:

    env HTTP_PROXY=http://localhost:8118 \
    kubectl get awscluster
    

    kubectl lists all your user clusters. Choose the user cluster you created with workload identity enabled.

    CLUSTER_NAME=CLUSTER_NAME
    

    Replace CLUSTER_NAME with the name of your cluster. For example, cluster-0.

  3. Set environment variables for the workload identity Pod image and namespace.

    IDENTITY_IMAGE=gcr.io/gke-multi-cloud-release/amazon-eks-pod-identity-webhook:release-0.2.2-gke.0
    
    WEBHOOK_NAMESPACE=workload-identity-webhook
    
  4. Generate the webhook YAML manifest in a file named aws-webhook.yaml by performing the following steps.

    env HTTP_PROXY=http://localhost:8118 \
    anthos-gke aws clusters get-credentials ${CLUSTER_NAME}
    
    CLUSTER_CA=$(env HTTP_PROXY=http://localhost:8118 \
    kubectl config view --raw -o json  | jq -r '.clusters[] | select(.name == "'$(kubectl config current-context)'") | .cluster."certificate-authority-data"')
    
    cat << EOF > aws-webhook.yaml
    apiVersion: v1
    kind: Namespace
    metadata:
      name: ${WEBHOOK_NAMESPACE}
    ---
    apiVersion: v1
    kind: ServiceAccount
    metadata:
      name: pod-identity-webhook
      namespace: ${WEBHOOK_NAMESPACE}
    ---
    apiVersion: rbac.authorization.k8s.io/v1
    kind: Role
    metadata:
      name: pod-identity-webhook
      namespace: ${WEBHOOK_NAMESPACE}
    rules:
      - apiGroups: ['']
        resources: ['secrets']
        verbs: ['create']
      - apiGroups: ['']
        resources: ['secrets']
        verbs: ['get', 'update', 'patch']
        resourceNames:
          - pod-identity-webhook
    ---
    apiVersion: rbac.authorization.k8s.io/v1
    kind: RoleBinding
    metadata:
      name: pod-identity-webhook
      namespace: ${WEBHOOK_NAMESPACE}
    roleRef:
      apiGroup: rbac.authorization.k8s.io
      kind: Role
      name: pod-identity-webhook
    subjects:
      - kind: ServiceAccount
        name: pod-identity-webhook
        namespace: ${WEBHOOK_NAMESPACE}
    ---
    apiVersion: rbac.authorization.k8s.io/v1
    kind: ClusterRole
    metadata:
      name: pod-identity-webhook
    rules:
      - apiGroups: ['']
        resources: ['serviceaccounts']
        verbs: ['get', 'watch',  'list']
      - apiGroups:  ['certificates.k8s.io']
        resources: ['certificatesigningrequests']
        verbs:  ['create', 'get', 'list', 'watch']
    ---
    apiVersion: rbac.authorization.k8s.io/v1
    kind: ClusterRoleBinding
    metadata:
      name: pod-identity-webhook
    roleRef:
      apiGroup: rbac.authorization.k8s.io
      kind: ClusterRole
      name: pod-identity-webhook
    subjects:
      - kind: ServiceAccount
        name: pod-identity-webhook
        namespace: ${WEBHOOK_NAMESPACE}
    ---
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: pod-identity-webhook
      namespace: ${WEBHOOK_NAMESPACE}
    spec:
      replicas: 1
      selector:
        matchLabels:
          app: pod-identity-webhook
      template:
        metadata:
          labels:
            app: pod-identity-webhook
        spec:
          serviceAccountName: pod-identity-webhook
          containers:
            - name: pod-identity-webhook
              image: ${IDENTITY_IMAGE}
              imagePullPolicy: Always
              command:
                - /webhook
                - --in-cluster
                - --namespace=${WEBHOOK_NAMESPACE}
                - --service-name=pod-identity-webhook
                - --tls-secret=pod-identity-webhook
                - --annotation-prefix=eks.amazonaws.com
                - --token-audience=sts.amazonaws.com
                - --logtostderr
              volumeMounts:
                - name: webhook-certs
                  mountPath: /var/run/app/certs
                  readOnly: false
          volumes:
            - name: webhook-certs
              emptyDir: {}
    ---
    apiVersion: v1
    kind: Service
    metadata:
      name: pod-identity-webhook
      namespace: ${WEBHOOK_NAMESPACE}
      annotations:
        prometheus.io/port: '443'
        prometheus.io/scheme: https
        prometheus.io/scrape: 'true'
    spec:
      ports:
        - port: 443
          targetPort: 443
      selector:
        app: pod-identity-webhook
    ---
    apiVersion: admissionregistration.k8s.io/v1beta1
    kind: MutatingWebhookConfiguration
    metadata:
      name: pod-identity-webhook
      namespace: ${WEBHOOK_NAMESPACE}
    webhooks:
      - name: pod-identity-webhook.amazonaws.com
        failurePolicy: Ignore
        clientConfig:
          service:
            name: pod-identity-webhook
            namespace: ${WEBHOOK_NAMESPACE}
            path: /mutate
          caBundle: ${CLUSTER_CA}
        rules:
          - operations: ['CREATE']
            apiGroups: ['']
            apiVersions: ['v1']
            resources: ['pods']
    EOF
    

    The contents of aws-webhook.yaml are ready to apply to your cluster.

Apply the webhook to your user cluster

To apply the webhook to your user cluster, perform the following steps.

  1. Apply the aws-webhook.yaml file to your user cluster.

    env HTTP_PROXY=http://localhost:8118 \
    kubectl apply -f aws-webhook.yaml
    
  2. When you apply the manifest, the webhook Pod generates Kubernetes certificate signing requests (CSR). Approve all requests from system:serviceaccount:${WEBHOOK_NAMESPACE}:pod-identity-webhook with kubectl certificate approve.

    kubectl certificate approve $(env HTTP_PROXY=http://localhost:8118 \
    kubectl get csr -o \
      jsonpath="{.items[?(@.spec.username==\"system:serviceaccount:${WEBHOOK_NAMESPACE}:pod-identity-webhook\")].metadata.name}")
    
  3. Verify that there are no remaining unapproved CSRs.

    Use kubectl get csr to check that all CSRs from the requestor system:serviceaccount:${WEBHOOK_NAMESPACE}:pod-identity-webhook are approved:

    env HTTP_PROXY=http://localhost:8118 \
    kubectl get csr
    

    Response:

    NAME        AGE   REQUESTOR                                            CONDITION
    csr-mxrt8   10s   system:serviceaccount:default:pod-identity-webhook   Approved,Issued
    

Configuring the AWS OIDC provider

This section is for cluster administrators.

To create an OIDC provider at AWS, AWS requires an intermediate Certificate Authority (CA) or server certificate thumbprint. Your OIDC discovery credentials are stored on storage.googleapis.com, with a certificate signed by an intermediate CA named GTS CA 1O1. This CA expires at 12/15/2021 at 00:00:42 UTC. The SHA-1 thumbprint of its intermediate CA GTS CA 1O1 is dfe2070c79e7ff36a925ffa327ffe3deecf8f9c2.

To register your OIDC discovery bucket as an OIDC provider with AWS, perform the following steps:

  1. From your anthos-aws directory, use anthos-gke to switch context to your management service.

    cd anthos-aws
    anthos-gke aws management get-credentials

  2. Save the OIDC issuer URL, issuer host path, and Cloud Storage thumbprint in environment variables.

    ISSUER_URL=$(env HTTP_PROXY=http://localhost:8118 \
    kubectl get awscluster ${CLUSTER_NAME} -o jsonpath='{.status.workloadIdentityInfo.issuerURL}')
    ISSUER_HOSTPATH=${ISSUER_URL#"https://"}
    CA_THUMBPRINT=dfe2070c79e7ff36a925ffa327ffe3deecf8f9c2
    
  3. Use the aws tool to create an OIDC provider on AWS.

    aws iam create-open-id-connect-provider \
      --url ${ISSUER_URL} \
      --thumbprint-list ${CA_THUMBPRINT} \
      --client-id-list sts.amazonaws.com
    

Update the thumbprint

If Google rotates the CA for storage.googleapis.com, run the following commands: 1. Check for the updated certificate thumbprint, dfe2070c79e7ff36a925ffa327ffe3deecf8f9c2.

  1. Update the thumbprint with the aws command. bash aws iam update-open-id-connect-provider-thumbprint

Creating AWS IAM roles and policies

This section is for cluster administrators.

Create an AWS IAM role to bind to a Kubernetes service account. The IAM role has permissions for sts:AssumeRoleWithWebIdentity.

To create the role, perform the following steps:

  1. Find or create an AWS IAM policy that grants the necessary permissions for your workloads.

    You need the policy's Amazon resource name (ARN) AWS IAM policy. For example, arn:aws:iam::aws:policy/AmazonS3ReadOnlyAccess.

  2. To create the Kubernetes service account and bind the AWS IAM policy to it, run the following script:

    KSA_NAME=KUBERNETES_SERVICE_ACCOUNT
    WORKLOAD_NAMESPACE=WORKLOAD_IDENTITY_NAMESPACE
    
    AWS_ROLE_NAME=AWS_ROLE_NAME
    AWS_POLICY=EXISTING_AWS_POLICY
    
    # Get the ID of the user cluster
    anthos-gke aws management get-credentials
    
    CLUSTER_ID=$(env HTTP_PROXY=http://localhost:8118 \
    kubectl get awscluster ${CLUSTER_NAME} -o jsonpath='{.status.clusterID}')
    
    # Get the ID Provider ARN
    PROVIDER_ARN=$(aws iam list-open-id-connect-providers  \
    | jq '.OpenIDConnectProviderList' \
    | jq ".[] | select(.Arn |  contains(\"${CLUSTER_ID}\"))"   \
    | jq  '.Arn' | tr -d '"')
    
    # Create AWS role and policy
    cat > irp-trust-policy.json << EOF
    {
      "Version": "2012-10-17",
      "Statement": [
        {
          "Effect": "Allow",
          "Principal": {
            "Federated": "${PROVIDER_ARN}"
          },
          "Action": "sts:AssumeRoleWithWebIdentity",
          "Condition": {
            "StringEquals": {
              "${ISSUER_HOSTPATH}:sub": "system:serviceaccount:${WORKLOAD_NAMESPACE}:${KSA_NAME}"
            }
          }
        }
      ]
    }
    EOF
    aws iam create-role \
      --role-name ${AWS_ROLE_NAME} \
      --assume-role-policy-document file://irp-trust-policy.json
    aws iam update-assume-role-policy \
      --role-name ${AWS_ROLE_NAME} \
      --policy-document file://irp-trust-policy.json
    aws iam attach-role-policy \
      --role-name ${AWS_ROLE_NAME} \
      --policy-arn ${AWS_POLICY}
    

    Replace the following:

    • KUBERNETES_SERVICE_ACCOUNT: the name of the new Kubernetes service account.
    • WORKLOAD_IDENTITY_NAMESPACE: the name of the namespace where workloads run.
    • AWS_ROLE_NAME: the name for a new AWS role.
    • EXISTING_AWS_POLICY: the Amazon resource name (ARN) of an existing AWS IAM policy. For example, arn:aws:iam::aws:policy/AmazonS3ReadOnlyAccess.

Creating Kubernetes service accounts for workloads

This section is for developers or cluster administrators.

To create Kubernetes service accounts bound to the AWS IAM role that was specified previously, perform the following steps:

  1. From your anthos-aws directory, use anthos-gke to switch context to your user cluster.

    cd anthos-aws
    env HTTP_PROXY=http://localhost:8118 \
    anthos-gke aws clusters get-credentials CLUSTER_NAME

  2. Create the Kubernetes service account by running the following commands:

    S3_ROLE_ARN=$(aws iam get-role \
      --role-name AWS_ROLE_NAME \
      --query Role.Arn --output text)
    
    cat << EOF  > k8s-service-account.yaml
    apiVersion: v1
    kind: ServiceAccount
    metadata:
      name: ${KSA_NAME}
      namespace: WORKLOAD_IDENTITY_NAMESPACE
    EOF
    
    env HTTP_PROXY=http://localhost:8118 \
    kubectl apply -f k8s-service-account.yaml
    
    env HTTP_PROXY=http://localhost:8118 \
    kubectl annotate sa --namespace ${WORKLOAD_NAMESPACE} ${KSA_NAME} eks.amazonaws.com/role-arn=${S3_ROLE_ARN}
    

    Replace the following:

    • AWS_ROLE_NAME: the name of the AWS IAM role to apply to your workloads
    • WORKLOAD_IDENTITY_NAMESPACE: the name of the namespace where workloads run

Applying credentials to your Pods

This section is for developers.

This section assumes that you have deployed the workload identity webhook. If you haven't deployed the webhook, skip to Applying credentials without the webhook.

Apply credentials with the webhook

This section describes how to configure your Pods to read credentials made available by the webhook.

Add the service account to the Pod

To use workload identity with a workload, add the Kubernetes service account to the following fields:

  • For a Deployment: spec.template.spec.serviceAccountName
  • For a Pod: spec.serviceAccount

The following Pod manifest launches a base CentOS image and contains the spec.serviceAccount field.

apiVersion: v1
kind: Pod
metadata:
  name: sample-centos-pod
  namespace: WORKLOAD_IDENTITY_NAMESPACE
spec:
  containers:
  - command:
    - /bin/bash
    - -ec
    - while :; do echo '.'; sleep 500 ; done
    image: centos:7
    name: centos
  serviceAccount: KUBERNETES_SERVICE_ACCOUNT

Replace the following:

  • WORKLOAD_IDENTITY_NAMESPACE: the name of the namespace where workloads run
  • KUBERNETES_SERVICE_ACCOUNT: the name of the Kubernetes service account you created previously

Check if Pods have the environment variables set

To check if Pods have the environment variables set, run the following command to get the Pod's information:

kubectl get pod --namespace WORKLOAD_IDENTITY_NAMESPACE POD_NAME -o yaml

Replace the following:

  • WORKLOAD_IDENTITY_NAMESPACE: the name of the namespace where workloads run
  • POD_NAME: the name of the Pod to check

The output contains the environment variable values in spec.containers.command.env and the mount point for the AWS IAM token. An example Pod manifest follows.

apiVersion: v1
kind: Pod
metadata:
  ...
spec:
  containers:
  - command:
    - /bin/bash
    - -ec
    - while :; do echo '.'; sleep 500 ; done
    env:
    - name: AWS_ROLE_ARN
      value: arn:aws:iam::1234567890:role/my-example-workload-role-1
    - name: AWS_WEB_IDENTITY_TOKEN_FILE
      value: /var/run/secrets/eks.amazonaws.com/serviceaccount/token
    image: centos:7
    imagePullPolicy: IfNotPresent
    name: centos
    resources: {}
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: my-k8s-serviceaccount-token-d4nz4
      readOnly: true
    - mountPath: /var/run/secrets/eks.amazonaws.com/serviceaccount
      name: aws-iam-token
      readOnly: true
  serviceAccount: my-k8s-serviceaccount
  serviceAccountName: my-k8s-serviceaccount
  volumes:
  - name: aws-iam-token
    projected:
      defaultMode: 420
      sources:
      - serviceAccountToken:
          audience: sts.amazonaws.com
          expirationSeconds: 86400
          path: token
  - name: my-k8s-serviceaccount-token-d4nz4
    secret:
      defaultMode: 420
      secretName: my-k8s-serviceaccount-token-d4nz4
   ...
status:
  ...

Apply credentials without the webhook

If you do not deploy the workload identity webhook, you need to do the following:

Create a Pod with credentials for workload identity

To create a Pod that includes the necessary credentials for workload identity, perform the following steps:

  1. Copy the following Pod manifest into a file named sample-pod-no-webhook.yaml. The configuration launches a base CentOS image with the necessary credentials.

    apiVersion: v1
    kind: Pod
    metadata:
      name: sample-centos-pod-no-webhook
      namespace: WORKLOAD_IDENTITY_NAMESPACE
    spec:
      containers:
      - command:
        - /bin/bash
        - -ec
        - while :; do echo '.'; sleep 500 ; done
        image: centos:7
        name: centos
        env:
        - name: AWS_ROLE_ARN
          value: IAM_ROLE_ARN
        - name: AWS_WEB_IDENTITY_TOKEN_FILE
          value: /var/run/secrets/eks.amazonaws.com/serviceaccount/token
        volumeMounts:
        - mountPath: /var/run/secrets/eks.amazonaws.com/serviceaccount
          name: aws-iam-token
          readOnly: true
      volumes:
      - name: aws-iam-token
        projected:
          defaultMode: 420
          sources:
          - serviceAccountToken:
              audience: sts.amazonaws.com
              expirationSeconds: 86400
              path: token
      serviceAccount: KUBERNETES_SERVICE_ACCOUNT
    

    Replace the following:

    • WORKLOAD_IDENTITY_NAMESPACE: the name of the namespace where workloads run.
    • IAM_ROLE_ARN: the ARN of the IAM role granted to the Pod. For example, arn:aws:iam::aws:policy/AmazonS3ReadOnlyAccess.
    • KUBERNETES_SERVICE_ACCOUNT: the name of the Kubernetes service account you created previously.
  2. Apply the Pod manifest to your cluster by using kubectl:

    env HTTP_PROXY=http://localhost:8118 \
    kubectl apply -f sample-pod-no-webhook.yaml
    

Check if Pods can access AWS resources

The following procedure describes how to check whether the Pod has received the credentials necessary for workload identity to function.

To complete the steps, you need to have the following:

  • bash shell access to the container; most production images don't have a shell available. The following example shows you how to use the Pod specified in the preceding section to access AWS S3.

  • Your Pod needs to have outbound access to the internet to download the AWS command-line interface.

To check if the Pod can access an S3 bucket, perform the following steps:

  1. Use kubectl exec to launch an interactive bash shell on the Pod sample-centos-pod-no-webhook:

    env HTTP_PROXY=http://localhost:8118 \
    kubectl exec -it --namespace ${WORKLOAD_NAMESPACE} sample-centos-pod-no-webhook -- bash
    

    Your terminal opens the bash shell on the Pod.

  2. Use pip to install the AWS command-line interface:

    curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py && python get-pip.py
    pip install awscli --upgrade
    
  3. Check the AWS IAM permissions and credentials by using the aws tool:

    aws sts assume-role-with-web-identity \
     --role-arn IAM_ROLE_ARN \
     --role-session-name mh9test \
     --web-identity-token file:///var/run/secrets/eks.amazonaws.com/serviceaccount/token \
     --duration-seconds 1000
    

    Replace IAM_ROLE_ARN with the ARN of the IAM role granted to the Pod. For example, arn:aws:iam::aws:policy/AmazonS3ReadOnlyAccess.

    The aws tool prints credentials information similar to the following:

    {
        "AssumedRoleUser": {
            "AssumedRoleId": "AROAR2ZZZLEXVSDCDJ37N:mh9test",
            "Arn": "arn:aws:sts::126285863215:assumed-role/my-example-workload-role-1/mh9test"
        },
        "Audience": "sts.amazonaws.com",
        "Provider": "arn:aws:iam::126285863215:oidc-provider/storage.googleapis.com/gke-issuer-cec6c353",
        "SubjectFromWebIdentityToken": "system:serviceaccount:default:my-s3-reader-ksa",
        "Credentials": {
            "SecretAccessKey": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
            "SessionToken": "MY_TOKEN",
            "Expiration": "2020-08-14T22:46:36Z",
            "AccessKeyId": "AKIAIOSFODNN7EXAMPLE"
        }
    }
    

    If you see the following message, check that the bucket is publicly accessible: An error occurred (InvalidIdentityToken) when calling the AssumeRoleWithWebIdentity operation: Couldn't retrieve verification key from your identity provider, please reference AssumeRoleWithWebIdentity documentation for requirements

Cleaning up

This section shows you how to remove resources that you created earlier in this document.

Clean up the service account and its associated IAM role

To delete the service account and its associated IAM role, perform the following steps:

  1. Clean up the service account:

    kubectl delete sa KUBERNETES_SERVICE_ACCOUNT --namespace WORKLOAD_IDENTITY_NAMESPACE
    

    Replace the following:

    • KUBERNETES_SERVICE_ACCOUNT: the name of the new Kubernetes service account
    • WORKLOAD_IDENTITY_NAMESPACE: the name of the namespace where workloads run
  2. Clean up the AWS IAM role. Choose from one of the following:

    • Delete the AWS IAM role with the AWS console.

    • Delete the role with the AWS command-line tool using the following commands:

      aws iam  detach-role-policy \
        --role-name=${AWS_ROLE_NAME} \
        --policy-arn=${AWS_POLICY}
      aws iam delete-role --role-name=${AWS_ROLE_NAME}
      

Delete your user cluster

To delete your user cluster, perform the steps in Uninstalling GKE on AWS.

Clean up the AWS OIDC provider

After the user cluster is deleted, unregister and delete the OIDC provider on AWS by using either the following bash shell command or the AWS console.

  1. From your anthos-aws directory, use anthos-gke to switch context to your management service.

    cd anthos-aws
    anthos-gke aws management get-credentials

  2. Delete the role with the AWS command-line tool with the following commands:

    CLUSTER_ID=$(env HTTP_PROXY=http://localhost:8118 \
    kubectl get awscluster ${CLUSTER_NAME} -o jsonpath='{.status.clusterID}')
    
    PROVIDER_ARN=$(aws iam list-open-id-connect-providers  \
    | jq '.OpenIDConnectProviderList' \
    | jq ".[] | select(.Arn |  contains(\"${CLUSTER_ID}\"))"   \
    | jq  '.Arn' | tr -d '"')
    
    aws iam delete-open-id-connect-provider \
        --open-id-connect-provider-arn=${PROVIDER_ARN}
    

    You receive confirmation that the AWS OIDC provider is deleted.

What's next