Integrating HTTP(S) Load Balancing with Cloud Run for Anthos

This tutorial shows how to use HTTP(S) Load Balancing with Cloud Run for Anthos on Google Cloud services, in place of Network Load Balancing, to expose services on the internet.

HTTP(S) Load Balancing provides global load balancing and integrates with a number of Google Cloud products and features such as Google Cloud Armor, Cloud CDN, Identity-Aware Proxy (IAP), and managed TLS certificates for HTTPS traffic.

Kubernetes Ingress and Istio ingress gateway

This tutorial uses two similarly named and related concepts. It's important to understand the following distinctions when completing this tutorial:

• Istio ingress gateway defines rules for routing external HTTP/TCP traffic to services in a Kubernetes cluster. The Istio ingress gateway is implemented as a Kubernetes Service and a Kubernetes Deployment.
• Kubernetes Ingress defines rules for routing external HTTP(S) traffic to one or more Kubernetes Services in a cluster. When you create a Kubernetes Ingress object in a GKE cluster, GKE provisions the resources required for HTTP(S) Load Balancing.

In this tutorial, you use a Kubernetes Ingress resource to route external traffic to the Istio ingress gateway. The Istio ingress gateway then routes traffic to services in the cluster, as the following diagram illustrates.

You can use the Istio ingress gateway to route to multiple services inside the same GKE cluster.

Objectives

• Create a GKE cluster with Cloud Run enabled.
• Create an Istio virtual service to handle health check requests.
• Modify the Kubernetes Service object of the Istio ingress gateway so it can be exposed by a Kubernetes Ingress object.
• Configure HTTP(S) Load Balancing by creating a Kubernetes Ingress object.
• Deploy a sample service to verify the solution.

Costs

This tutorial uses the following billable components of Google Cloud:

• Cloud Run for Anthos
• Compute Engine
• Cloud Logging
• Cloud Monitoring

To generate a cost estimate based on your projected usage, use the pricing calculator. New Google Cloud users might be eligible for a free trial.

When you finish this tutorial, you can avoid continued billing by deleting the resources you created. For more information, see Cleaning up.

Before you begin

1. Select or create a Cloud project.

   Go to the project selector page

   Note: If you don't plan to keep the resources you create in this procedure, create a new Cloud project instead of selecting an existing Cloud project. After you finish following these steps, you can delete the Cloud project, removing all resources associated with the Cloud project.

2. Enable billing for your Cloud project.

3. Enable the GKE, Cloud Run, and Google Cloud APIs.

   Enable the APIs

4. In the Cloud Console, open Cloud Shell.

   Open Cloud Shell

   At the bottom of the Cloud Console, a Cloud Shell session opens and displays a command-line prompt. Cloud Shell is a shell environment with the Cloud SDK already installed, including the gcloud command-line tool, and with values already set for your current project. It can take a few seconds for the session to initialize.

   You use Cloud Shell to run all the commands in this tutorial.

Creating a GKE cluster with Cloud Run

• Create a GKE cluster with the Cloud Run add-on:

  CLUSTER=cloudrun-gke-gclb-tutorial
  ZONE=us-central1-f
  
  gcloud container clusters create $CLUSTER \
      --addons HorizontalPodAutoscaling,HttpLoadBalancing,CloudRun \
      --enable-ip-alias \
      --enable-stackdriver-kubernetes \
      --release-channel regular \
      --zone $ZONE
  

  This tutorial uses cloudrun-gke-gclb-tutorial as the cluster name and as the us-central1-f zone. You can change these values. For more information, see Geography and regions.

  To understand the options in the command, review the following:

  • The HttpLoadBalancing add-on lets Kubernetes Ingress objects configure HTTP(S) Load Balancing for the cluster.

  • The --enable-ip-alias option makes the cluster VPC-native. This option is required if you want to use container-native load balancing, which this tutorial uses.

  • The --enable-stackdriver-kubernetes option aggregates logs, events, and metrics from the cluster by using Cloud Operations for GKE. This option is required by the Cloud Run add-on.

Handling health check requests

• Create an Istio virtual service that allows the cluster to respond to load-balancing health check requests by forwarding the requests to the status endpoint on the Istio ingress gateway.

  Notice that the virtual service that you are creating is attached to the gke-system-gateway Istio gateway resource, which is installed by the Cloud Run add-on.

  kubectl apply -f - <<EOF
  apiVersion: networking.istio.io/v1alpha3
  kind: VirtualService
  metadata:
    name: health
    namespace: knative-serving
  spec:
    gateways:
    - gke-system-gateway
    hosts:
    - "*"
    http:
    - match:
      - headers:
          user-agent:
            prefix: GoogleHC
        method:
          exact: GET
        uri:
          exact: /
      rewrite:
        authority: istio-ingress.gke-system.svc.cluster.local:15020
        uri: /healthz/ready
      route:
      - destination:
          host: istio-ingress.gke-system.svc.cluster.local
          port:
            number: 15020
  EOF
  

Modifying the Istio ingress gateway for use with Kubernetes Ingress

1. Create a JSON Patch file to make changes to the Istio ingress gateway:

   cat <<EOF > istio-ingress-patch.json
   [
     {
       "op": "replace",
       "path": "/spec/type",
       "value": "NodePort"
     },
     {
       "op": "remove",
       "path": "/status"
     }
   ]
   EOF
   

2. Apply the patch file and add the Istio ingress gateway as a backend:

   kubectl -n gke-system patch svc istio-ingress \
       --type=json -p="$(cat istio-ingress-patch.json)" \
       --dry-run=true -o yaml | kubectl apply -f -
   kubectl annotate svc istio-ingress -n gke-system cloud.google.com/neg='{"exposed_ports": {"80":{}}}'
   

   This patch makes the following changes to the Kubernetes Service object of the Istio ingress gateway:

   • Adds the annotation cloud.google.com/neg: '{"ingress": true}'. This annotation creates a network endpoint group and enables container-native load balancing when the Kubernetes Ingress object is created.
   • Changes the Kubernetes Service type from LoadBalancer to NodePort. This change removes the Network Load Balancing resources.
   Note: When using the Cloud Run add-on for GKE, upgrades to the Cloud Run control plane components installed by the add-on are managed as part of the GKE master upgrade process. The upgrade process can remove changes applied to control plane components. If so, repeat the steps in this section to reapply the changes after the upgrade. Alternatively, you can install the open source Istio and Knative Serving components instead of using the GKE add-ons. If you choose to do so, you're responsible for installing Istio and Knative Serving updates.

Creating a Kubernetes Ingress object

1. Create a Kubernetes Ingress object called my-ingress:

   kubectl apply -f - <<EOF
   apiVersion: extensions/v1beta1
   kind: Ingress
   metadata:
     name: my-ingress
     namespace: gke-system
   spec:
     backend:
       serviceName: istio-ingress
       servicePort: 80
   EOF
   

   This Ingress object sends all incoming traffic to the istio-ingress Kubernetes Service in the gke-system namespace.

   When you create a Kubernetes Ingress object in a GKE cluster, GKE creates the resources required for HTTP(S) Load Balancing.

2. Watch the progress of the preceding command and wait until the my-ingress ADDRESS changes to an IP address:

   kubectl get ingress my-ingress -n gke-system --watch
   

   To stop waiting, press Control+C.

3. Create an environment variable to store the IP address of the Kubernetes Ingress object:

   INGRESS_IP=$(kubectl get ingress my-ingress -n gke-system \
       --output jsonpath='{.status.loadBalancer.ingress[0].ip}')
   

Note: In this tutorial you access the services using an ephemeral IP address and unencrypted HTTP. For a production environment, we recommend that you do all of the following:

• Reserve a global static external IP address and use it as the public IP address of the Kubernetes Ingress object.
• Create a DNS A record for your domain name and set its value to the public IP address of the Kubernetes Ingress object. If you use Cloud DNS, follow the Quickstart guide. If you use another provider, refer to their documentation.
• Configure the Kubernetes Ingress object to use HTTPS.

Deploying a service and verifying the solution

1. Deploy a sample service called my-service to Cloud Run for Anthos:

   gcloud run deploy my-service \
       --namespace default \
       --image gcr.io/knative-samples/simple-api \
       --platform gke \
       --cluster $CLUSTER \
       --cluster-location $ZONE
   

   It can take a few minutes for HTTP(S) Load Balancing to be ready to route traffic after the Kubernetes Ingress object is created. In the meantime, you might get errors such as HTTP/1.1 404 Not Found or HTTP/1.1 502 Bad Gateway when you send requests to services in the cluster.

2. Send an HTTP GET request to the sample service every two seconds:

   while sleep 2; do
     curl -siH "Host: my-service.default.example.com" $INGRESS_IP | head -n1
   done
   

   Monitor the output until you see the following:

   HTTP/1.1 200 OK
   

   To stop sending requests, press Control+C.

3. Send an HTTP GET request to the sample service:

   curl -s -w"\n" -H "Host: my-service.default.example.com" $INGRESS_IP
   

   The output is the following:

   OK
   

Troubleshooting

If you run into problems with this tutorial, review these documents:

• Container-native load balancing troubleshooting
• Cloud Run for Anthos troubleshooting
• GKE troubleshooting
• Istio Operations Guide
• Troubleshooting Kubernetes clusters

Cleaning up

To avoid incurring charges to your Google Cloud account for the resources used in this tutorial, you can delete the Cloud project that you created for this tutorial, or delete the resources associated with this tutorial.

Delete the Cloud project

The easiest way to eliminate billing is to delete the Cloud project you created for the tutorial.

Caution: Deleting a project has the following effects:

• Everything in the project is deleted. If you used an existing project for this tutorial, when you delete it, you also delete any other work you've done in the project.
• Custom project IDs are lost. When you created this project, you might have created a custom project ID that you want to use in the future. To preserve the URLs that use the project ID, such as an appspot.com URL, delete selected resources inside the project instead of deleting the whole project.

If you plan to explore multiple tutorials and quickstarts, reusing projects can help you avoid exceeding project quota limits.

1. In the Cloud Console, go to the Manage resources page.

   Go to Manage resources

2. In the project list, select the project that you want to delete, and then click Delete.
3. In the dialog, type the project ID, and then click Shut down to delete the project.

Delete the resources

If you want to keep the Cloud project you used in this tutorial, delete the GKE cluster:

```
gcloud container clusters delete $CLUSTER --zone $ZONE --async --quiet
```

What's next

• Learn how to authenticate end users of Cloud Run for Anthos services using Istio and Identity Platform.
• Learn how to authorize access to Cloud Run for Anthos services using Istio.
• Set up domain names and static IP addresses with HTTP(S) Load Balancing.
• Configure TLS certificates in HTTP(S) Load Balancing with Kubernetes Ingress.
• Add Google Cloud Armor to HTTP(S) Load Balancing using BackendConfig.
• Try out other Google Cloud features for yourself. Have a look at our tutorials.