Integrating IAP with Cloud Service Mesh
This guide describes how to integrate Identity-Aware Proxy (IAP) with Cloud Service Mesh. The IAP integration with Cloud Service Mesh enables you to safely access services based on Google's BeyondCorp principles. IAP verifies user identity and context of the request to determine if a user should be allowed to access an application or resource. The IAP integration with Cloud Service Mesh provides you with the following benefits:
Complete context-aware access control to the workloads running on Cloud Service Mesh. You can set fine-grained access policies based on attributes of the originating request, such as user identity, the IP address, and device type. You can combine your access policies with restrictions based on the hostname and path of a request URL.
Enable support for context-aware claims in Cloud Service Mesh authorization.
Scalable, secure, and highly available access to your application through a Google Cloud load balancer. High performance load balancing provides built-in protection of distributed denial-of-service (DDoS) attacks and support for global anycast IP addressing.
Prerequisites
Follow the steps in Install dependent tools and validate cluster to:- Install required tools
- Download
asmcli
- Grant cluster admin permissions
- Validate your project and cluster
Additionally, this guide assumes that you have:
Setting up a cluster with Anthos Service Mesh
This section explains how to get set up for the IAP integration for both new installations of Cloud Service Mesh and upgrades.
New installations
Enable
iap.googleapis.com
. In the following command, replacePROJECT_ID
with the project that you will install Cloud Service Mesh in:gcloud services enable \ --project=PROJECT_ID \ iap.googleapis.com
The cluster that you are updating must have the
--addons=HttpLoadBalancing
option set. TheHttpLoadBalancing
add-on enables an HTTP (L7) load balancing controller for the cluster. Run the following command to update the cluster with the options required by Cloud Service Mesh. Unless you've set a default zone or region, you need to supply the region (--region=REGION) or zone (--zone=ZONE) in the command.gcloud container clusters update CLUSTER_NAME \ --project=PROJECT_ID \ --update-addons=HttpLoadBalancing=ENABLED
By default, the
iap-operator.yaml
file has port 31223 set as the status port, and port 31224 set as the http port. If port 31223 is already in use in your cluster, run the following to set another status port:kpt cfg set asm gcloud.container.cluster.ingress.statusPort STATUS_PORT
If port 31224 is already in use in your cluster, run the following to set another http port:
kpt cfg set asm gcloud.container.cluster.ingress.httpPort HTTP_PORT
Follow the steps in Install default features and Mesh CA to use a Google-provided script to install Cloud Service Mesh. When you run the script, include the following option:
--option iap-operator
For example:
./asmcli install \ --project_id "PROJECT_ID" \ --cluster_name "CLUSTER_NAME" \ --cluster_location "CLUSTER_LOCATION" \ --fleet_id FLEET_PROJECT_ID \ --output_dir DIR_PATH \ --enable_all \ --option iap-operator
When you install Cloud Service Mesh, the
iap-operator.yaml
file sets thetype
field on theistio-ingressgateway
service toNodePort
, which configures the gateway to open a specific port on the service mesh. This allows you to set up a load balancer, which routes traffic sent to your domain name to this port.If you are installing managed Cloud Service Mesh, then also complete the following steps:
Add the revision label to the
istio-system
namespace.Download the Istio ingress gateway service spec for IAP and name it
iap_operator.yaml
.Install the ingress as a NodePort service. For more information, see Migrate from IstioOperator.
asmcli experimental mcp-migrate-check -f iap_operator.yaml istioctl install -f /asm-generated-configs/gateways-istiooperator/"GATEWAY_NAME".yaml
After installing Cloud Service Mesh, return to this guide and continue with the next section to set up the integration with IAP.
Upgrades
This section covers the following upgrade use cases:
You have already set up the IAP integration and you are upgrading Cloud Service Mesh. In this case, you have already enabled
iap.googleapis.com
on your project and theHttpLoadBalancing
add-on on your cluster. Skip to step 3, to download theasm
package and upgrade Cloud Service Mesh.You are upgrading Cloud Service Mesh and you want to set up the integration with IAP for the first time. In this case, you need to complete all of the following steps, upgrade Cloud Service Mesh, and return to this guide after the upgrade to complete the integration.
Enable
iap.googleapis.com
. In the following command, replacePROJECT_ID
with the project that you will install Cloud Service Mesh in.gcloud services enable \ --project=PROJECT_ID \ iap.googleapis.com
The cluster that you are updating must have the
--addons=HttpLoadBalancing
option set. TheHttpLoadBalancing
add-on enables an HTTP (L7) load balancing controller for the cluster. Run the following command to update the cluster with the options required by Cloud Service Mesh. Unless you've set a default zone or region, you need to supply the region (--region=REGION) or zone (--zone=ZONE) in the command.gcloud container clusters update CLUSTER_NAME \ --project=PROJECT_ID --update-addons=HttpLoadBalancing=ENABLED
If you are updating an existing, functioning HTTP Cloud Load Balancer, run the following to preserve your existing http and status ports:
kpt cfg set asm gcloud.container.cluster.ingress.httpPort $(kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.spec.ports[?(@.name=="http2")].nodePort}')
kpt cfg set asm gcloud.container.cluster.ingress.statusPort $(kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.spec.ports[?(@.name=="status-port")].nodePort}')
Follow the steps in Upgrading Cloud Service Mesh to use a Google-provided script to upgrade Cloud Service Mesh.
When you upgrade Cloud Service Mesh, the
iap-operator.yaml
file sets thetype
field on theistio-ingressgateway
service toNodePort
, which configures the gateway to open a specific port on the service mesh. This allows you to set up a load balancer, which routes traffic sent to your domain name to this port.By default, the
iap-operator.yaml
file has port 31223 set as the status port, and port 31224 set as the http port.When you run the script, include the following option:
--option iap-operator
For example:
./asmcli install \ --project_id "PROJECT_ID" \ --cluster_name "CLUSTER_NAME" \ --cluster_location "CLUSTER_LOCATION" \ --fleet_id FLEET_PROJECT_ID \ --output_dir DIR_PATH \ --enable_all \ --option iap-operator
Complete the upgrade by triggering automatic sidecar proxy injection on your workloads. For details, see Deploying and redeploying workloads.
After completing the upgrade, return to this guide and continue with the next section to set up the integration with IAP.
Reserving a static IP address and configuring DNS
To integrate Identity-Aware Proxy with Cloud Service Mesh, you have to set up a Google Cloud HTTP(S) load balancer, which requires a domain name that points to a static IP address. You can reserve a static external IP address, which assigns the address to your project indefinitely until you explicitly release it.
Reserve a static external IP address:
gcloud compute addresses create example-static-ip --global
Get the static IP address:
gcloud compute addresses describe example-static-ip --global
In your domain name registrar, configure a fully qualified domain name (FQDN) with the static IP address. Typically, you add an
A
record to your DNS settings. The configuration steps and terminology for adding anA
record for a FQDN vary depending on your domain name registrar.It can take 24 to 48 hours for the DNS setting to propagate. You can continue setting up everything in this guide, but you won't be able to test the setup until the DNS settings propagate.
Deploying a sample application
Before you enable IAP, you need an application running on your GKE cluster so you can verify that all requests have an identity. This guide uses the Bookinfo sample to demonstrate how to setup the HTTP(S) load balancer and enable IAP.
Follow the steps to deploy Bookinfo. Until you deploy the load balancer, the Bookinfo application isn't accessible outside of your GKE cluster (such as from a browser).
External requests
Bookinfo's Gateway resource (defined in
samples/bookinfo/networking/bookinfo-gateway.yaml
) uses the preconfigured
istio-ingressgateway
.
Recall that when you deployed Cloud Service Mesh, you specified NodePort
for the
istio-ingressgateway
, which opens a specific port on the service mesh.
Although the nodes in your cluster have external IP addresses, requests coming
from outside your cluster are blocked by Google Cloud firewall rules. With
IAP, the correct way to expose applications to the public
internet is by using a load balancer. Don't expose the node addresses using
firewall rules, which would bypass IAP.
To route requests to Bookinfo, you set up an HTTP(S) load balancer in your Google Cloud project. Because the load balancer is in your project, it is inside of the firewall and can access the nodes in your cluster. After you configure the load balancer with the static IP address and your domain name, you can send requests to the domain name, and the load balancer forwards the requests to the nodes in the cluster.
Enabling IAP
The following steps describe how to enable IAP.
Use the client ID (
CLIENT_ID
in the above step) andCLIENT_SECRET
to enable Cloud Service Mesh. Create a kubernetes secret with the materials from your OAuth Client:kubectl create secret generic -n istio-system my-secret --from-literal=client_id=CLIENT_ID \ --from-literal=client_secret=CLIENT_SECRET
Deploying the load balancer
You can use an Ingress resource to create an HTTP(S) load balancer with automatically configured SSL certificates. Managed SSL certificates are provisioned, renewed, and managed for your domain.
Create a ManagedCertificate resource. This resource specifies the domain for the SSL certificate. The
spec.domains
list must contain only one domain. Wildcard domains aren't supported. In the following YAML, replaceDOMAIN_NAME
with the domain name that you configured for the external static IP address.cat <<EOF | kubectl apply -f - apiVersion: networking.gke.io/v1 kind: ManagedCertificate metadata: name: example-certificate namespace: istio-system spec: domains: - DOMAIN_NAME EOF
Create a BackendConfig resource. This resource instructs GCLB how to perform Health Checks on the Ingress Gateway, as well as configure Identity-Aware Proxy. First, collect a few values from the Ingress Gateway about health checks:
Health check ingress port: This is the health check port of istio-ingress.
export HC_INGRESS_PORT=$(kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.spec.ports[?(@.name=="status-port")].nodePort}')
Health check ingress path: This is the health check path of istio-ingress.
export HC_INGRESS_PATH=$(kubectl get pods -n istio-system -l app=istio-ingressgateway -o jsonpath='{.items[0].spec.containers[?(@.name=="istio-proxy")].readinessProbe.httpGet.path}')
cat <<EOF | kubectl apply -n istio-system -f - apiVersion: cloud.google.com/v1 kind: BackendConfig metadata: name: http-hc-config spec: healthCheck: checkIntervalSec: 2 timeoutSec: 1 healthyThreshold: 1 unhealthyThreshold: 10 port: ${HC_INGRESS_PORT} type: HTTP requestPath: ${HC_INGRESS_PATH} iap: enabled: true oauthclientCredentials: secretName: my-secret EOF
Annotate the ingress service with your BackendConfig.
kubectl annotate -n istio-system service/istio-ingressgateway --overwrite \ cloud.google.com/backend-config='{"default": "http-hc-config"}' \ cloud.google.com/neg='{"ingress":false}'
Create the load balancer by defining the Ingress resource.
Set the
networking.gke.io/managed-certificates
annotation to the name of the certificate you created in the previous step,example-certificate
.Set the
kubernetes.io/ingress.global-static-ip-name
annotation to the name of the static IP address you reserved,example-static-ip
.Set the
serviceName
toistio-ingressgateway
, which is used in the Gateway resource for the Bookinfo sample.
cat <<EOF | kubectl apply -f - apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: example-ingress namespace: istio-system annotations: kubernetes.io/ingress.global-static-ip-name: example-static-ip networking.gke.io/managed-certificates: example-certificate spec: defaultBackend: service: name: istio-ingressgateway port: number: 80 EOF
In the Google Cloud console, go to the Kubernetes Engine > Services & Ingress page.
Go to the Services & Ingress page
You should see the "Creating ingress" message in the Status column. Wait for GKE to fully provision the Ingress before continuing. Refresh the page every few minutes to get the most up-to-date status on the Ingress. After the Ingress is provisioned, you might see the "Ok" status, or the error "All backend services are in UNHEALTHY state." One of the resources that GKE provisions is a default health check. If you see the error message, that indicates that the Ingress is provisioned and that the default health check ran. When you see either the "Ok" status or the error, continue with the next section to configure the health checks for the load balancer.
Configure the IAP access list
Add a user to the access policy for IAP:
gcloud beta iap web add-iam-policy-binding \ --member=user:EMAIL_ADDRESS \ --role=roles/iap.httpsResourceAccessor
where EMAIL_ADDRESS
is the user's full email address
such as alice@example.com
.
Test the load balancer. Point your browser to:
http://DOMAIN_NAME/productpage
where
DOMAIN_NAME
is the domain name that you configured with the external static IP address.You should see the Bookinfo application's
productpage
. If you refresh the page several times, you should see different versions of reviews, presented in a round robin style: red stars, black stars, no stars.You should also test
https
access to Bookinfo.
Enable RCToken support on the service mesh
By default, IAP generates a JSON Web Token (JWT) that is scoped to the OAuth client. For Cloud Service Mesh, you can configure IAP to generate a RequestContextToken (RCToken), which is a JWT but with a configurable audience. RCToken lets you configure the audience of the JWT to an arbitrary string, which can be used in the Cloud Service Mesh policies for fine-grained authorization.
To configure the RCToken:
Create an environment variable for the RCToken audience. This can be any string that you want.
export RCTOKEN_AUD="your-rctoken-aud"
Optional: The following step requires the
BACKEND_SERVICE_ID
. If you need to find out theBACKEND_SERVICE_ID
, run the following command:kubectl -n istio-system get Ingress example-ingress -o json | jq \ '.metadata.annotations."ingress.kubernetes.io/backends"'
The expected output is similar to
"{\"BACKEND_SERVICE_ID\":\"HEALTHY\"}"
. For example,"ingress.kubernetes.io/backends": "{\"k8s-be-31224--51f3b55cd1457fb6\":\"HEALTHY\"}"
. TheBACKEND_SERVICE_ID
in this example isk8s-be-31224--51f3b55cd1457fb6
.Fetch the existing IAP settings.
gcloud iap settings get --format json \ --project=${PROJECT_ID} --resource-type=compute --service=BACKEND_SERVICE_ID > iapSettings.json
Update
IapSettings
with the RCToken audience.cat iapSettings.json | jq --arg RCTOKEN_AUD_STR $RCTOKEN_AUD \ '. + {applicationSettings: {csmSettings: {rctokenAud: $RCTOKEN_AUD_STR}}}' \ > updatedIapSettings.json
gcloud iap settings set updatedIapSettings.json --format json \ --project=${PROJECT_ID} --resource-type=compute --service=BACKEND_SERVICE_ID
Enable RCToken authentication on the Istio ingress gateway.
cat <<EOF | kubectl apply -f - apiVersion: "security.istio.io/v1beta1" kind: "RequestAuthentication" metadata: name: "ingressgateway-jwt-policy" namespace: "istio-system" spec: selector: matchLabels: app: istio-ingressgateway jwtRules: - issuer: "https://cloud.google.com/iap" jwksUri: "https://www.gstatic.com/iap/verify/public_key-jwk" audiences: - $RCTOKEN_AUD fromHeaders: - name: ingress-authorization prefix: "Istio " outputPayloadToHeader: "verified-jwt" forwardOriginalToken: true EOF
Optional: Ensure requests that don't have valid JWTs are rejected:
cat <<EOF | kubectl apply -f - apiVersion: security.istio.io/v1beta1 kind: AuthorizationPolicy metadata: name: iap-gateway-require-jwt namespace: istio-system spec: selector: matchLabels: app: istio-iap-ingressgateway action: DENY rules: - from: - source: notRequestPrincipals: ["*"] EOF
Make sure requests to the Bookinfo
productpage
are still successful:http://DOMAIN_NAME/productpage
To test the policy:
Create an
IapSettings
request object, but set therctokenAud
to a different string:cat iapSettings.json | jq --arg RCTOKEN_AUD_STR wrong-rctoken-aud \ '. + {applicationSettings: {csmSettings: {rctokenAud: $RCTOKEN_AUD_STR}}}' \ > wrongIapSettings.json
Call the
IapSettings
API to set the RCtoken audience.gcloud beta iap settings set wrongIapSettings.json --project=PROJECT_ID --resource-type=compute --service=BACKEND_SERVICE
Make a request to the Bookinfo
productpage
and it should fail:http://DOMAIN_NAME/productpage
Cleaning up
After completing this tutorial, remove the following resources to prevent unwanted charges incurring on your account:
Delete the managed certificate:
kubectl delete managedcertificates example-certificate
Delete the Ingress, which deallocates the load balancing resources:
kubectl -n istio-system delete ingress example-ingress
Delete the static IP address:
gcloud compute addresses delete example-static-ip --global
If you do this, be sure to delete the IP address from your domain registrar.
Delete the cluster, which deletes the resources that make up the cluster, such as the compute instances, disks and network resources:
gcloud container clusters delete CLUSTER_NAME