Anthos Service Mesh 1.6

Integrating IAP with Anthos Service Mesh

This guide describes how to integrate Identity-Aware Proxy (IAP) with Anthos Service Mesh. The IAP integration with Anthos Service Mesh enables you to safely access services based on Google's BeyondCorp principles. IAP verifies user identity and context of the request to determine if a user should be allowed to access an application or resource. The IAP integration with Anthos Service Mesh provides you with the following benefits:

  • Complete context-aware access control to the workloads running on Anthos Service Mesh. You can set fine-grained access policies based on attributes of the originating request, such as user identity, the IP address, and device type. You can combine your access policies with restrictions based on the hostname and path of a request URL.

  • Enable support for context-aware claims in Anthos Service Mesh authorization.

  • Scalable, secure, and highly available access to your application through a Google Cloud load balancer. High performance load balancing provides built-in protection of distributed denial-of-service (DDoS) attacks and support for global anycast IP addressing.


This guide assumes that you have:

Setting up a cluster with Anthos Service Mesh

You need an Anthos GKE on Google Cloud (GKE on Google Cloud) cluster with Anthos Service Mesh installed. Use one of the following guides to set up your project, a new or existing cluster, and install Anthos Service Mesh using istioctl:

Note the following steps that you need to do when you follow the other guides:

  1. When you set up your project, enable when you enable the other APIs that Anthos Service Mesh requires.

  2. The cluster that you are creating or updating must have the --addons=HttpLoadBalancing option set. The HttpLoadBalancing add-on enables an HTTP (L7) load balancing controller for the cluster.

    • If you are creating a new cluster, when you run gcloud beta container clusters create, include the --addons=HttpLoadBalancing option with the other options that Anthos Service Mesh requires.

    • If you are updating an existing cluster, run the following command when update the cluster with the options required by Anthos Service Mesh:

      gcloud container clusters update ${CLUSTER_NAME} \
  3. When you install Anthos Service Mesh, run the following command instead of the one in the guide you are following:

    istioctl install -f asm/cluster/istio-operator.yaml \
      --set values.gateways.istio-ingressgateway.type=NodePort

    You specify NodePort for the istio-ingressgateway, which configures Anthos Service Mesh to open a specific port on the service mesh. This allows you to set up a load balancer, which routes traffic sent to your domain name to this port.

Reserving a static IP address and configuring DNS

To integrate Identity-Aware Proxy with Anthos Service Mesh, you have to set up a Google Cloud HTTP(S) load balancer, which requires a domain name that points to a static IP address. You can reserve a static external IP address, which assigns the address to your project indefinitely until you explicitly release it.

  1. Reserve a static external IP address:

    gcloud compute addresses create example-static-ip --global
  2. Get the static IP address:

    gcloud compute addresses describe example-static-ip --global
  3. In your domain name registrar, configure a fully qualified domain name (FQDN) with the static IP address. Typically, you add an A record to your DNS settings. The configuration steps and terminology for adding an A record for a FQDN vary depending on your domain name registrar.

  4. Set the domain name in an environment variable:


    It can take 24 to 48 hours for the DNS setting to propagate. You can continue setting up everything in this guide, but you won't be able to test the setup until the DNS settings propagate.

Deploying a sample application

Before you enable IAP, you need an application running on your GKE cluster so you can verify that all requests have an identity. This guide uses the Bookinfo sample to demonstrate how to setup the HTTP(S) load balancer and enable IAP.

Follow the steps to deploy Bookinfo. Until you deploy the load balancer, the Bookinfo application isn't accessible outside of your GKE cluster (such as from a browser).

External requests

Bookinfo's Gateway resource (defined in samples/bookinfo/networking/bookinfo-gateway.yaml) uses the preconfigured istio-ingressgateway. Recall that when you deployed Anthos Service Mesh, you specified NodePort for the istio-ingressgateway, which opens a specific port on the service mesh. Although the nodes in your cluster have external IP addresses, requests coming from outside your cluster are blocked by Google Cloud firewall rules. With IAP, the correct way to expose applications to the public internet is by using a load balancer. Don't expose the node addresses using firewall rules, which would bypass IAP.

To route requests to Bookinfo, you set up an HTTP(S) load balancer in your Cloud project. Because the load balancer is in your project, it is inside of the firewall and can access the nodes in your cluster. After you configure the load balancer with the static IP address and your domain name, you can send requests to the domain name, and the load balancer forwards the requests to the nodes in the cluster.

Deploying the load balancer

You can use an Ingress resource to create an HTTP(S) load balancer with automatically configured SSL certificates. Google-managed SSL certificates are provisioned, renewed, and managed for your domain.

  1. Create a ManagedCertificate resource. This resource specifies the domain for the SSL certificate. The list must contain only one domain. Wildcard domains aren't supported.

    cat <<EOF | kubectl apply -f -
    kind: ManagedCertificate
      name: example-certificate
      namespace: istio-system
        - ${DOMAIN_NAME}
  2. Create the load balancer by defining the Ingress resource.

    • Set the annotation to the name of the certificate you created in the previous step, example-certificate.

    • Set the annotation to the name of the static IP address you reserved, example-static-ip.

    • Set the serviceName to istio-ingressgateway, which is used in the Gateway resource for the Bookinfo sample.

    cat <<EOF | kubectl create -f -
    apiVersion: extensions/v1beta1
    kind: Ingress
      name: example-ingress
      namespace: istio-system
      annotations: example-static-ip example-certificate
        serviceName: istio-ingressgateway
        servicePort: 80
  3. In the Cloud Console, go to the Kubernetes Engine > Services & Ingress page.

    Go to the Services & Ingress page

    You should see the "Creating ingress" message in the Status column. Wait for GKE to fully provision the Ingress before continuing. Refresh the page every few minutes to get the most up-to-date status on the Ingress. After the Ingress is provisioned, you might see the "Ok" status, or the error "All backend services are in UNHEALTHY state." One of the resources that GKE provisions is a default health check. If you see the error message, that indicates that the Ingress is provisioned and that the default health check ran. When you see either the "Ok" status or the error, continue with the next section to configure the health checks for the load balancer.

Configure health checks for the load balancer.

To configure the health checks, you need to obtain the ID of the default health check created by the Ingress and then update the health check to use istio-ingress's health check path and port.

  1. Get new user credentials to use for Application Default Credentials:

      gcloud auth application-default login

  2. Obtain the ID of the default health check created by the Ingress:

    1. Set the following environment variables:

      • Backend Service: Bridges various Instance Groups on a given Service NodePort.

        BACKEND_SERVICE=$(gcloud compute url-maps list | grep example-ingress | awk '{print $2}' | cut -d'/' -f 2)

      • Health check: This is the default health check that is created automatically when the Ingress is deployed.

        HC=$(gcloud compute backend-services describe ${BACKEND_SERVICE} --global | grep healthChecks | cut -d'/' -f 10 | tail -n 1)

      • Health check ingress port: This is the health check port of istio-ingress.

        export HC_INGRESS_PORT=$(kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.spec.ports[?("status-port")].nodePort}')

      • Health check ingress path: This is the health check path of istio-ingress.

        export HC_INGRESS_PATH=$(kubectl -n istio-system get deployments istio-ingressgateway -o jsonpath='{.spec.template.spec.containers[?("istio-proxy")].readinessProbe.httpGet.path}')

      • Health check API: This is the API that you call to configure the health check.
        export HC_API=${PROJECT_ID}/global/healthChecks/${HC}

    2. Get the default health check into a JSON file by calling the healthChecks API:

      curl --request GET  --header "Authorization: Bearer $(gcloud auth application-default print-access-token)" ${HC_API} > health_check.json
  3. Update the health check to use istio-ingress's health check path and port:

    1. Update the health_check.json file as follows:

      • Set httpHealthCheck.port to the value of ${HC_INGRESS_PORT}.
      • Set httpHealthCheck.requestPath to the value of ${HC_INGRESS_PATH}.
      • Add the following attribute and set it to an empty string: httpHealthCheck.portSpecification=""

      The easiest way to do this is to use jq, which comes preinstalled on Cloud Shell:

      jq ".httpHealthCheck.port=${HC_INGRESS_PORT} | .httpHealthCheck.requestPath=\"${HC_INGRESS_PATH}\" | .httpHealthCheck.portSpecification=\"\"" health_check.json > updated_health_check.json

      If you run cat on the resulting updated_health_check.json file, it looks similar to the following:

      "id": "5062913090021441698",
      "creationTimestamp": "2019-11-12T10:47:41.934-08:00",
      "name": "${HC}",
      "description": "Default kubernetes L7 Loadbalancing health check.",
      "checkIntervalSec": 60,
      "timeoutSec": 60,
      "unhealthyThreshold": 10,
      "healthyThreshold": 1,
      "type": "HTTP",
      "httpHealthCheck": {
        "port": 32394,
        "requestPath": "/healthz/ready",
        "proxyHeader": "NONE",
        "portSpecification": ""
      "selfLink": "${PROJECT_ID}/global/healthChecks/${HC}",
      "kind": "compute#healthCheck"

      If you edited the JSON file manually instead of using the jq command, save the file as updated_health_check.json so that it matches the filename in the next command.

    2. Update the health check:

      curl --request PATCH --header "Authorization: Bearer $(gcloud auth application-default print-access-token)" --header "Content-Type: application/json" --data @updated_health_check.json ${HC_API}

    It takes several minutes for GKE to update the health check. In the Cloud Console, refresh the Kubernetes Engine > Services & Ingress page every minute or so until the status for the Ingress changes to "Ok."

  4. Test the load balancer. Point your browser to:


    where YOUR_DOMAIN_NAME is the domain name that you configured with the external static IP address.

    You should see the Bookinfo application's productpage. If you refresh the page several times, you should see different versions of reviews, presented in a round robin style: red stars, black stars, no stars.

    You should also test https access to Bookinfo.

Enabling IAP

The following steps describe how to enable IAP.

  1. Check if you already have an existing brand by using the list command. You may only have one brand per project.

    gcloud alpha iap oauth-brands list

    The following is an example gcloud response, if the brand exists:

    name: projects/[PROJECT_NUMBER]/brands/[BRAND_ID]
    applicationTitle: [APPLICATION_TITLE]
    supportEmail: [SUPPORT_EMAIL]
    orgInternalOnly: true
  2. If no brand exists, use the create command:

    gcloud alpha iap oauth-brands create --applicationTitle=APPLICATION_TITLE --supportEmail=SUPPORT_EMAIL

    The above fields are required when calling this API:

    • supportEmail: The support email displayed on the OAuth consent screen. This email address can either be a user's address or a Google Groups alias. While service accounts also have an email address, they are not actual valid email addresses, and cannot be used when creating a brand. However, a service account can be the owner of a Google Group. Either create a new Google Group or configure an existing group and set the desired service account as an owner of the group.

    • applicationTitle: The application name displayed on OAuth consent screen.

    The response contains the following fields:

    name: projects/[PROJECT_NUMBER]/brands/[BRAND_NAME]
    applicationTitle: [APPLICATION_TITLE]
    supportEmail: [SUPPORT_EMAIL]
    orgInternalOnly: true

Creating an IAP OAuth Client

  1. Use the create command to create a client. Use the brand name from previous step.

    gcloud alpha iap oauth-clients create projects/PROJECT-ID/brands/BRAND-ID --display_name=NAME

    The response contains the following fields:

    name: projects/[PROJECT_NUMBER]/brands/[BRAND_NAME]/identityAwareProxyClients/[CLIENT_ID]
    secret: [CLIENT_SECRET]
    displayName: [NAME]

Turning on IAP for your service

Use the following command to turn on IAP for your service. Replace CLIENT_ID and CLIENT_SECRET with your OAuth client ID and client secret from the client you created previously.

gcloud beta iap web enable \
    --oauth2-client-id=CLIENT_ID \
    --oauth2-client-secret=CLIENT_SECRET \
    --resource-type=backend-services \

Configure the IAP access list

Add a user to the access policy for IAP:

gcloud beta iap web add-iam-policy-binding \
    --member=user:EMAIL_ADDRESS \
    --role=roles/iap.httpsResourceAccessor \
    --resource-type=backend-services \

where EMAIL_ADDRESS is the user's full email address such as

Enable RCToken support on the service mesh

By default, IAP generates a JSON Web Token (JWT) that is scoped to the OAuth client. For Anthos Service Mesh, you can configure IAP to generate a RequestContextToken (RCToken), which is a JWT but with a configurable audience. RCToken lets you configure the audience of the JWT to an arbitrary string, which can be used in the Anthos Service Mesh policies for fine-grained authorization.

To configure the RCToken:

  1. Create an environment variable for your project number. This is the number that was automatically generated and assigned to your project when you created it. (This isn't the same as the project ID.)

  2. Create an environment variable for the RCToken audience. This can be any string that you want.

    export RCTOKEN_AUD="your-rctoken-aud"
  3. Fetch the existing IAP settings

    gcloud beta iap settings get --format json \
    --project=${PROJECT_NUMBER} --resource-type=compute \
    --service=${BACKEND_SERVICE} > iapSettings.json
  4. Update IapSettings with the RCToken audience.

    cat iapSettings.json | jq --arg RCTOKEN_AUD_STR $RCTOKEN_AUD \
    '. + {applicationSettings: {csmSettings: {rctokenAud: $RCTOKEN_AUD_STR}}}' \
    > updatedIapSettings.json
    gcloud beta iap settings set updatedIapSettings.json --format json \
    --project=${PROJECT_NUMBER} --resource-type=compute --service=${BACKEND_SERVICE}
  5. Enable RCToken authentication on the Istio ingress gateway.

    cat <<EOF | kubectl apply -f -
    apiVersion: ""
    kind: "RequestAuthentication"
      name: "ingressgateway-jwt-policy"
      namespace: "istio-system"
          app: istio-ingressgateway
      - issuer: ""
        jwksUri: ""
        - $RCTOKEN_AUD
        - name: ingress-authorization
          prefix: "Istio "
        outputPayloadToHeader: "verified-jwt"
        forwardOriginalToken: true
  6. Make sure requests to the Bookinfo productpage are still successful:


To test the policy:

  1. Create an IapSettings request object, but set the rctokenAud to a different string:

    echo $(cat <<EOF
       "name": "projects/${PROJECT_NUMBER}/iap_web/compute/services/${BACKEND_SERVICE}",
       "applicationSettings": {
         "csmSettings": {
           "rctokenAud": "some-other-arbitrary-string"
    ) > request.txt
  2. Call the IapSettings API to set the RCtoken audience.

    curl --request PATCH --header "Authorization: Bearer $(gcloud beta auth application-default print-access-token)" ${IAP_SETTINGS_API}
  3. Make a request to the Bookinfo productpage and it should fail:


Cleaning up

After completing this tutorial, remove the following resources to prevent unwanted charges incurring on your account:

  1. Delete the managed certificate:

    kubectl delete managedcertificates example-certificate
  2. Delete the Ingress, which deallocates the load balancing resources:

    kubectl -n istio-system delete ingress example-ingress

  3. Delete the static IP address:

    gcloud compute addresses delete example-static-ip --global

    If you do this, be sure to delete the IP address from your domain registrar.

  4. Delete the cluster, which deletes the resources that make up the cluster, such as the compute instances, disks and network resources:

    gcloud container clusters delete ${CLUSTER_NAME}