Setting up HTTP(S) Load Balancing with Ingress

This tutorial shows how to run a web application behind an external HTTP(S) load balancer by configuring the Ingress resource.

Note: This process does not apply to an NGINX Ingress controller.

Background

Google Kubernetes Engine (GKE) offers integrated support for two types of Cloud Load Balancing for a publicly accessible application:

• When you specify type:LoadBalancer in the resource manifest, GKE creates a Service of type LoadBalancer. GKE makes appropriate Google Cloud API calls to create either an external network load balancer or an internal TCP/UDP load balancer. GKE creates an internal TCP/UDP load balancer when you add the cloud.google.com/load-balancer-type: "Internal" annotation; otherwise, GKE creates an external network load balancer.

  Although you can use either of these types of load balancers for HTTP(S) traffic, they operate in OSI layers 3/4 and are not aware of HTTP connections or individual HTTP requests and responses. Another important characteristic is that the requests are not proxied to the destination.

Note: When GKE creates an external network load balancer, it does not configure a load balancer health check for the target pool. (External network load balancers using target pools do not require health checks.) When GKE creates an internal TCP/UDP load balancer, it creates a health check for the load balancer's backend service based on the readiness probe settings of the workload referenced by the GKE Service. For more information, see Internal TCP/UDP Load Balancing.

• When you specify kind:Ingress in the resource manifest, you instruct GKE to create an Ingress resource. By including annotations and supporting workloads and Services, you can create a custom Ingress controller. Otherwise, GKE makes appropriate Google Cloud API calls to create an external HTTP(S) load balancer. The load balancer's URL map's host rules and path matchers reference one or more backend services, where each backend service corresponds to a GKE Service of type NodePort, as referenced in the Ingress. The backends for each backend service are either instance groups or network endpoint groups (NEGs). NEGs are created when you configure container-native load balancing as part of the configuration for your Ingress. For each backend service, GKE creates a Google Cloud health check, based on the readiness probe settings of the workload referenced by the corresponding GKE Service.

  If you are exposing an HTTP(S) service hosted on GKE, HTTP(S) load balancing is the recommended method for load balancing.

Note: The load balancers created by GKE are billed according to the regular Cloud Load Balancing pricing.

Note: To use Ingress, you must have the HTTP(S) Load Balancing add-on enabled. GKE clusters have HTTP(S) Load Balancing enabled by default; you must not disable it.

Note: Google Kubernetes Engine relies on a health check mechanism to determine the health status of the backend service. This mechanism cannot be used to perform Blackbox monitoring. To perform Blackbox monitoring, create an uptime check. For more information, see (Optional) Monitoring the availability and latency of your service.

Objectives

• Create a GKE cluster.
• Deploy the sample web application to the cluster.
• Expose the sample app to the internet behind an external HTTP(S) load balancer.

Before you begin

Take the following steps to enable the Kubernetes Engine API:

1. Visit the Kubernetes Engine page in the Google Cloud Console.
2. Create or select a project.
3. Wait for the API and related services to be enabled. This can take several minutes.

4. Make sure that billing is enabled for your Cloud project. Learn how to confirm that billing is enabled for your project.

Install the following command-line tools used in this tutorial:

• gcloud is used to create and delete Kubernetes Engine clusters. gcloud is included in the Google Cloud SDK.
• kubectl is used to manage Kubernetes, the cluster orchestration system used by Kubernetes Engine. You can install kubectl using gcloud:

  gcloud components install kubectl

Clone the sample code from GitHub:

git clone https://github.com/GoogleCloudPlatform/kubernetes-engine-samples
cd kubernetes-engine-samples/load-balancing

Set defaults for the gcloud command-line tool

To save time typing your project ID and Compute Engine zone options in the gcloud command-line tool, you can set the defaults:

gcloud config set project project-id
gcloud config set compute/zone compute-zone

Create a container cluster

Create a container cluster named loadbalancedcluster by running:

gcloud container clusters create loadbalancedcluster

Note: If you are using an existing Google Kubernetes Engine cluster or if you have created a cluster through Google Cloud Console, you need to run the following command to retrieve cluster credentials and configure kubectl command-line tool with them:

gcloud container clusters get-credentials loadbalancedcluster

If you have already created a cluster with the gcloud container clusters create command listed above, this step is not necessary.

Deploying a web application

The following manifest describes a Deployment that runs the sample web application container image on an HTTP server on port 8080:

load-balancing/web-deployment.yaml

View on GitHub

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
  namespace: default
spec:
  selector:
    matchLabels:
      run: web
  template:
    metadata:
      labels:
        run: web
    spec:
      containers:
      - image: gcr.io/google-samples/hello-app:1.0
        imagePullPolicy: IfNotPresent
        name: web
        ports:
        - containerPort: 8080
          protocol: TCP

Apply the resource to the cluster:

kubectl apply -f web-deployment.yaml

Exposing your Deployment inside your cluster

The following manifest describes a Service that makes the web deployment accessible within your container cluster:

load-balancing/web-service.yaml

View on GitHub

apiVersion: v1
kind: Service
metadata:
  name: web
  namespace: default
spec:
  ports:
  - port: 8080
    protocol: TCP
    targetPort: 8080
  selector:
    run: web
  type: NodePort

1. Apply the resource to the cluster:

   kubectl apply -f web-service.yaml
   

   When you create a Service of type NodePort with this command, GKE makes your Service available on a randomly selected high port number (e.g. 32640) on all the nodes in your cluster.

2. Verify that the Service is created and a node port is allocated:

   kubectl get service web
   

   Output:

   NAME      TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE
   web       NodePort   10.35.245.219   <none>        8080:32640/TCP   5m
   

   In the sample output, the node port for the web Service is 32640. Also, note that there is no external IP allocated for this Service. Since the GKE nodes are not externally accessible by default, creating this Service does not make your application accessible from the internet.

To make your HTTP(S) web server application publicly accessible, you must create an Ingress resource.

Creating an Ingress resource

Ingress is a Kubernetes resource that encapsulates a collection of rules and configuration for routing external HTTP(S) traffic to internal services.

On GKE, Ingress is implemented using Cloud Load Balancing. When you create an Ingress in your cluster, GKE creates an HTTP(S) load balancer and configures it to route traffic to your application.

While the Kubernetes Ingress is a beta resource, meaning how you describe the Ingress object is subject to change, the Cloud Load Balancers that GKE provisions to implement the Ingress are production-ready.

The following manifest describes an Ingress resource that directs traffic to your web Service:

load-balancing/basic-ingress.yaml

View on GitHub

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: basic-ingress
spec:
  backend:
    serviceName: web
    servicePort: 8080

Apply the resource to the cluster:

kubectl apply -f basic-ingress.yaml

After you deploy this manifest, Kubernetes creates an Ingress resource on your cluster. The GKE Ingress controller creates and configures an HTTP(S) Load Balancer according to the information in the Ingress, routing all external HTTP traffic (on port 80) to the web NodePort Service you exposed.

Note: To use Ingress, you must have the HTTP(S) Load Balancing add-on enabled. GKE clusters have HTTP(S) Load Balancing enabled by default; you must not disable it.

Visiting your application

Find out the external IP address of the load balancer serving your application by running:

kubectl get ingress basic-ingress

Output:

NAME            HOSTS     ADDRESS         PORTS     AGE
basic-ingress   *         203.0.113.12    80        2m

Note: It might take a few minutes for GKE to allocate an external IP address and set up forwarding rules before the load balancer is ready to serve your application. You might get errors such as HTTP 404 or HTTP 500 until the load balancer configuration is propagated across the globe.

Open the external IP address of your application in a browser and see a plain text HTTP response like the following:

Hello, world!
Version: 1.0.0
Hostname: web-6498765b79-fq5q5

You can visit Load Balancing on Cloud Console and inspect the networking resources created by the GKE Ingress controller.

(Optional) Configuring a static IP address

When you expose a web server on a domain name, you need the external IP address of an application to be a static IP that does not change.

By default, GKE allocates ephemeral external IP addresses for HTTP applications exposed through an Ingress. Ephemeral addresses are subject to change. If you are planning to run your application for a long time, you must use a static external IP address.

Note that after you configure a static IP for the Ingress resource, deleting the Ingress does not delete the static IP address associated with it. Make sure to clean up the static IP addresses you configured when you no longer plan to use them again.

To configure a static IP address, complete the following steps:

1. Reserve a static external IP address named web-static-ip:

   gcloud

   gcloud compute addresses create web-static-ip --global
   

   Config Connector

   Note: This step requires Config Connector. Follow the installation instructions to install Config Connector on your cluster.

   View on GitHub

   apiVersion: compute.cnrm.cloud.google.com/v1beta1
   kind: ComputeAddress
   metadata:
     name: web-static-ip
   spec:
     location: global

   To deploy this manifest, download it to your machine as compute-address.yaml, and run:

   kubectl apply -f compute-address.yaml

2. The basic-ingress-static.yaml manifest adds an annotation on Ingress to use the static IP resource named web-static-ip:

   load-balancing/basic-ingress-static.yaml

   View on GitHub

   apiVersion: networking.k8s.io/v1beta1
   kind: Ingress
   metadata:
     name: basic-ingress
     annotations:
       kubernetes.io/ingress.global-static-ip-name: "web-static-ip"
   spec:
     backend:
       serviceName: web
       servicePort: 8080
   

   View the manifest:

   cat basic-ingress-static.yaml
   

3. Apply the resource to the cluster:

   kubectl apply -f basic-ingress-static.yaml
   

4. Check the external IP address:

   kubectl get ingress basic-ingress
   

   Wait until the IP address of your application changes to use the reserved IP address of the web-static-ip resource.

   It might take a few minutes to update the existing Ingress resource, re-configure the load balancer, and propagate the load balancing rules across the globe. After this operation completes, GKE releases the ephemeral IP address previously allocated to your application.

Note: Unused static external IP address are billed according to the regular IP address billing.

(Optional) Serving multiple applications on a load balancer

You can run multiple services on a single load balancer and public IP by configuring routing rules on the Ingress. By hosting multiple services on the same Ingress, you can avoid creating additional load balancers (which are billable resources) for every Service that you expose to the internet.

The following manifest describes a Deployment with version 2.0 of the same web application:

load-balancing/web-deployment-v2.yaml

View on GitHub

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web2
  namespace: default
spec:
  selector:
    matchLabels:
      run: web2
  template:
    metadata:
      labels:
        run: web2
    spec:
      containers:
      - image: gcr.io/google-samples/hello-app:2.0
        imagePullPolicy: IfNotPresent
        name: web2
        ports:
        - containerPort: 8080
          protocol: TCP

Apply the resource to the cluster:

kubectl apply -f web-deployment-v2.yaml

The following manifest describes a Service that exposes web2 internally to the cluster on a NodePort Service called web2:

load-balancing/web-service-v2.yaml

View on GitHub

apiVersion: v1
kind: Service
metadata:
  name: web2
  namespace: default
spec:
  ports:
  - port: 8080
    protocol: TCP
    targetPort: 8080
  selector:
    run: web2
  type: NodePort

Apply the resource to the cluster:

kubectl apply -f web-service-v2.yaml

The following manifest describes an Ingress resource that:

• routes the requests with path starting with /v2/ to the web2 Service
• routes all other requests to the web Service

load-balancing/fanout-ingress.yaml

View on GitHub

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: fanout-ingress
spec:
  rules:
  - http:
      paths:
      - path: /*
        backend:
          serviceName: web
          servicePort: 8080
      - path: /v2/*
        backend:
          serviceName: web2
          servicePort: 8080

Apply the resource to the cluster:

kubectl create -f fanout-ingress.yaml

After the Ingress is deployed, run kubectl get ingress fanout-ingress to find out the public IP address of the cluster.

Note: It might take a few minutes for GKE to allocate an external IP address and prepare the load balancer. You might get errors like HTTP 404 and HTTP 500 until the load balancer is ready to serve the traffic.

Then visit the IP address to see that both applications are reachable on the same load balancer:

• Visit http://<IP_ADDRESS>/ and note that the response contains Version: 1.0.0 (as the request is routed to the web Service)
• Visit http://<IP_ADDRESS>/v2/ and note that the response contains Version: 2.0.0 (as the request is routed to the web2 Service)

The only supported wildcard character for the path field of an Ingress is the * character. The * character must follow a forward slash (/) and must be the last character in the pattern. For example, /*, /foo/*, and /foo/bar/* are valid patterns, but *, /foo/bar*, and /foo/*/bar are not.

A more specific pattern takes precedence over a less specific pattern. If you have both /foo/* and /foo/bar/*, then /foo/bar/bat is taken to match /foo/bar/*.

For more information about path limitations and pattern matching, see the URL Maps documentation.

(Optional) Monitoring the availability and latency of your service

Google Cloud Uptime checks perform blackbox monitoring of applications from the viewpoint of the user, determining latency and availability from multiple external IPs to the IP address of the load balancer. In comparison, Google Cloud health checks perform an internal check against the Pod IPs, determining availability at the instance level. The checks are complementary and provide a holistic picture of application health.

You can create an uptime check by using the Google Cloud Console, the Cloud Monitoring API, or by using the Cloud Monitoring client libraries. For information, see Managing uptime checks. If you want to create an uptime check by using the Google Cloud Console, do the following:

1. In the Google Cloud Console, select Monitoring, or click the following button:

   Go to Monitoring

2. In the Monitoring navigation pane, select Uptime checks and then click Create uptime check.

3. For the target of your uptime check, set the following fields:

   • Select the type of protocol as TCP.
   • For the Resource type, select URL.
   • For the Hostname, enter the IP address of the load balancer.
   • Enter the Load Balancer port number in the Port field.

   For complete documentation on all the fields in an uptime check, see Creating an uptime check.

To monitor an uptime check, you can create an alerting policy or view the uptime check dashboard. An alerting policy can notify you by email or through a different channel if your uptime check fails. For general information about alerting policies, see Introduction to alerting.

Note: You can create an alerting policy for an uptime check as part of the process of creating the uptime check. Creating an alerting policy is optional, but it is recommended. For information on how to create an alerting policy as an independent action, see Alerting on uptime checks

Remarks

By default, Ingress performs a periodic health check by making a GET request on the / path to determine the health of the application, and expects a HTTP 200 response. If you want to check a different path or to expect a different response code, you can use a custom health check path.

Ingress supports more advanced use cases, such as:

• Name-based virtual hosting: You can use Ingress to reuse the load balancer for multiple domain names, subdomains, and to expose multiple Services on a single IP address and load balancer. Check out the simple fanout and name-based virtual hosting examples to learn how to configure Ingress for these tasks.

• HTTPS termination: You can configure the Ingress to terminate the HTTPS traffic using the Cloud Load Balancer.

Note: Always modify the properties of the Load Balancer via the Ingress object. Making changes directly on the load balancing resources might get lost or overridden by the GKE Ingress controller.

When an Ingress is deleted, the GKE Ingress controller cleans up the associated resources (except reserved static IP addresses) automatically.

Cleaning up

To avoid incurring charges to your Google Cloud account for the resources used in this tutorial, either delete the project that contains the resources, or keep the project and delete the individual resources.

1. Delete any manually created forwarding rules and target proxies that reference the Ingress:

   Note: This step is needed only if the load balancer associated with the Ingress is updated manually via GCP API or UI.

   A dangling target proxy that is referencing a GKE Ingress controller managed URL map will cause the deletion of the Ingress to fail in GKE versions 1.15.4-gke.22+. Inspect the Ingress resource to find an event with an error message similar to the following:

    Error during GC: error running load balancer garbage collection routine: googleapi: Error 400: The url_map resource 'projects/project-id/global/urlMaps/k8s2-um-tlw9rhgp-default-my-ingress-9ifnni82' is already being used by 'projects/project-id/global/targetHttpsProxies/k8s2-um-tlw9rhgp-default-my82-target-proxy', resourceInUseByAnotherResource
    

   In the preceding error message, k8s2-um-tlw9rhgp-default-my82-target-proxy is a manually created target https proxy that is still referencing the URL map k8s2-um-tlw9rhgp-default-my-ingress-9ifnni82 which was created and managed by an Ingress controller.

   These manually created frontend resources (both forwarding rule and target proxy) must be deleted before proceeding with the deletion of the Ingress.

2. Delete the Ingress: This step deallocates the ephemeral external IP address and the load balancing resources associated with your application:

   kubectl delete ingress basic-ingress

   If you followed the optional step to create an Ingress to route requests by path, then delete the Ingress:

   kubectl delete ingress fanout-ingress

3. Delete the static IP address: Complete this step only if you followed the optional step to create a static IP address.

   • If you followed "Option 1" to convert an existing ephemeral IP address to static IP, then visit Cloud Console to delete the static IP address.

   • If you followed "Option 2" to create a new static IP address, then run the following command to delete the static IP address:

     gcloud compute addresses delete web-static-ip --global

4. Delete the cluster: This step deletes the compute nodes of your container cluster and other resources such as the Deployments in the cluster:

   gcloud container clusters delete loadbalancedcluster

What's next

• Check out the Ingress user guide for details about Ingress features.
• Configure static IP and a domain name for your Ingress application using Ingress.
• Configure SSL certificates for your Ingress load balancer.

• If you have an application running on multiple Kubernetes Engine clusters in different regions, set up a multi-cluster Ingress to route traffic to a cluster in the region closest to the user.

• Explore other Kubernetes Engine tutorials.

• Try out other Google Cloud features for yourself. Have a look at our tutorials.