Integrating HTTP(S) Load Balancing with Cloud Run for Anthos

This tutorial shows how to use HTTP(S) Load Balancing with Cloud Run for Anthos on Google Cloud services, in place of Network Load Balancing, to expose services on the internet.

HTTP(S) Load Balancing provides global load balancing and integrates with a number of Google Cloud products and features such as Google Cloud Armor, Cloud CDN, Identity-Aware Proxy (IAP), and managed TLS certificates for HTTPS traffic.

Kubernetes Ingress and Istio ingress gateway

This tutorial uses two similarly named and related concepts. It's important to understand the following distinctions when completing this tutorial:

  • Istio ingress gateway defines rules for routing external HTTP/TCP traffic to services in a Kubernetes cluster. The Istio ingress gateway is implemented as a Kubernetes Service and a Kubernetes Deployment.
  • Kubernetes Ingress defines rules for routing external HTTP(S) traffic to one or more Kubernetes Services in a cluster. When you create a Kubernetes Ingress object in a GKE cluster, GKE provisions the resources required for HTTP(S) Load Balancing.

In this tutorial, you use a Kubernetes Ingress resource to route external traffic to the Istio ingress gateway. The Istio ingress gateway then routes traffic to services in the cluster, as the following diagram illustrates.

Diagram that shows how external HTTP/TCP traffic is routed.

You can use the Istio ingress gateway to route to multiple services inside the same GKE cluster.

Objectives

  • Create a GKE cluster with Cloud Run enabled.
  • Create an Istio virtual service to handle health check requests.
  • Modify the Kubernetes Service object of the Istio ingress gateway so it can be exposed by a Kubernetes Ingress object.
  • Configure HTTP(S) Load Balancing by creating a Kubernetes Ingress object.
  • Deploy a sample service to verify the solution.

Costs

This tutorial uses the following billable components of Google Cloud:

To generate a cost estimate based on your projected usage, use the pricing calculator. New Google Cloud users might be eligible for a free trial.

When you finish this tutorial, you can avoid continued billing by deleting the resources you created. For more information, see Cleaning up.

Before you begin

  1. Select or create a Cloud project.

    Go to the project selector page

  2. Enable billing for your Cloud project.

  3. Enable the GKE, Cloud Run, and Google Cloud APIs.

    Enable the APIs

  4. In the Cloud Console, open Cloud Shell.

    Open Cloud Shell

    At the bottom of the Cloud Console, a Cloud Shell session opens and displays a command-line prompt. Cloud Shell is a shell environment with the Cloud SDK already installed, including the gcloud command-line tool, and with values already set for your current project. It can take a few seconds for the session to initialize.

    You use Cloud Shell to run all the commands in this tutorial.

Creating a GKE cluster with Cloud Run

  • Create a GKE cluster with the Cloud Run add-on:

    CLUSTER=cloudrun-gke-gclb-tutorial
    ZONE=us-central1-f
    
    gcloud beta container clusters create $CLUSTER \
        --addons HorizontalPodAutoscaling,HttpLoadBalancing,CloudRun \
        --enable-ip-alias \
        --enable-stackdriver-kubernetes \
        --machine-type n1-standard-2 \
        --zone $ZONE
    

    This tutorial uses cloudrun-gke-gclb-tutorial as the cluster name and as the us-central1-f zone. You can change these values. For more information, see Geography and regions.

    To understand the options in the command, review the following:

    • The HttpLoadBalancing add-on lets Kubernetes Ingress objects configure HTTP(S) Load Balancing for the cluster.

    • The --enable-ip-alias option makes the cluster VPC-native. This option is required if you want to use container-native load balancing, which this tutorial uses.

    • The --enable-stackdriver-kubernetes option aggregates logs, events, and metrics from the cluster by using Cloud Operations for GKE. This option is required by the Cloud Run add-on.

Handling health check requests

  • Create an Istio virtual service that allows the cluster to respond to load-balancing health check requests by forwarding the requests to the status endpoint on the Istio ingress gateway.

    Notice that the virtual service that you are creating is attached to the gke-system-gateway Istio gateway resource, which is installed by the Cloud Run add-on.

    kubectl apply -f - <<EOF
    apiVersion: networking.istio.io/v1alpha3
    kind: VirtualService
    metadata:
      name: health
      namespace: knative-serving
    spec:
      gateways:
      - gke-system-gateway
      hosts:
      - "*"
      http:
      - match:
        - headers:
            user-agent:
              prefix: GoogleHC
          method:
            exact: GET
          uri:
            exact: /
        rewrite:
          authority: istio-ingress.gke-system.svc.cluster.local:15020
          uri: /healthz/ready
        route:
        - destination:
            host: istio-ingress.gke-system.svc.cluster.local
            port:
              number: 15020
    EOF
    

Modifying the Istio ingress gateway for use with Kubernetes Ingress

  1. Create a JSON Patch file to make changes to the Istio ingress gateway:

    cat <<EOF > istio-ingress-patch.json
    [
      {
        "op": "replace",
        "path": "/spec/type",
        "value": "NodePort"
      },
      {
        "op": "remove",
        "path": "/status"
      }
    ]
    EOF
    
  2. Apply the patch file and add the Istio ingress gateway as a backend:

    kubectl -n gke-system patch svc istio-ingress \
        --type=json -p="$(cat istio-ingress-patch.json)" \
        --dry-run=true -o yaml | kubectl apply -f -
    kubectl annotate svc istio-ingress -n gke-system cloud.google.com/neg='{"exposed_ports": {"80":{}}}'
    

    This patch makes the following changes to the Kubernetes Service object of the Istio ingress gateway:

Creating a Kubernetes Ingress object

  1. Create a Kubernetes Ingress object called my-ingress:

    kubectl apply -f - <<EOF
    apiVersion: extensions/v1beta1
    kind: Ingress
    metadata:
      name: my-ingress
      namespace: gke-system
    spec:
      backend:
        serviceName: istio-ingress
        servicePort: 80
    EOF
    

    This Ingress object sends all incoming traffic to the istio-ingress Kubernetes Service in the gke-system namespace.

    When you create a Kubernetes Ingress object in a GKE cluster, GKE creates the resources required for HTTP(S) Load Balancing.

  2. Watch the progress of the preceding command and wait until the my-ingress ADDRESS changes to an IP address:

    kubectl get ingress my-ingress -n gke-system --watch
    

    To stop waiting, press Control+C.

  3. Create an environment variable to store the IP address of the Kubernetes Ingress object:

    INGRESS_IP=$(kubectl get ingress my-ingress -n gke-system \
        --output jsonpath='{.status.loadBalancer.ingress[0].ip}')
    

Deploying a service and verifying the solution

  1. Deploy a sample service called my-service to Cloud Run for Anthos:

    gcloud run deploy my-service \
        --namespace default \
        --image gcr.io/knative-samples/simple-api \
        --platform gke \
        --cluster $CLUSTER \
        --cluster-location $ZONE
    

    It can take a few minutes for HTTP(S) Load Balancing to be ready to route traffic after the Kubernetes Ingress object is created. In the meantime, you might get errors such as HTTP/1.1 404 Not Found or HTTP/1.1 502 Bad Gateway when you send requests to services in the cluster.

  2. Send an HTTP GET request to the sample service every two seconds:

    while sleep 2; do
      curl -siH "Host: my-service.default.example.com" $INGRESS_IP | head -n1
    done
    

    Monitor the output until you see the following:

    HTTP/1.1 200 OK
    

    To stop sending requests, press Control+C.

  3. Send an HTTP GET request to the sample service:

    curl -s -w"\n" -H "Host: my-service.default.example.com" $INGRESS_IP
    

    The output is the following:

    OK
    

Troubleshooting

If you run into problems with this tutorial, review these documents:

Cleaning up

To avoid incurring charges to your Google Cloud account for the resources used in this tutorial, you can delete the Cloud project that you created for this tutorial, or delete the resources associated with this tutorial.

Delete the Cloud project

The easiest way to eliminate billing is to delete the Cloud project you created for the tutorial.

  1. In the Cloud Console, go to the Manage resources page.

    Go to Manage resources

  2. In the project list, select the project that you want to delete, and then click Delete.
  3. In the dialog, type the project ID, and then click Shut down to delete the project.

Delete the resources

If you want to keep the Cloud project you used in this tutorial, delete the GKE cluster:

```
gcloud container clusters delete $CLUSTER --zone $ZONE --async --quiet
```

What's next