Configuring Ingress for external load balancing

This page shows you how to configure an external HTTP(S) load balancer by creating a Kubernetes Ingress object. An Ingress object must be associated with one or more Service objects, each of which is associated with a set of Pods.

A Service object has one or more servicePort structures. Each servicePort that is targeted by an Ingress is associated with a Google Cloud backend service resource.

Before you begin

Before you start, make sure you have performed the following tasks:

  • Ensure that you have enabled the Google Kubernetes Engine API.
  • Enable Google Kubernetes Engine API
  • Ensure that you have installed the Cloud SDK.
  • Set up default gcloud command-line tool settings for your project by using one of the following methods:
    • Use gcloud init, if you want to be walked through setting project defaults.
    • Use gcloud config, to individually set your project ID, zone, and region.

    gcloud init

    1. Run gcloud init and follow the directions:

      gcloud init

      If you are using SSH on a remote server, use the --console-only flag to prevent the command from launching a browser:

      gcloud init --console-only
    2. Follow the instructions to authorize the gcloud tool to use your Google Cloud account.
    3. Create a new configuration or select an existing one.
    4. Choose a Google Cloud project.
    5. Choose a default Compute Engine zone.
    6. Choose a default Compute Engine region.

    gcloud config

    1. Set your default project ID:
      gcloud config set project PROJECT_ID
    2. Set your default Compute Engine region (for example, us-central1):
      gcloud config set compute/region COMPUTE_REGION
    3. Set your default Compute Engine zone (for example, us-central1-c):
      gcloud config set compute/zone COMPUTE_ZONE
    4. Update gcloud to the latest version:
      gcloud components update

    By setting default locations, you can avoid errors in gcloud tool like the following: One of [--zone, --region] must be supplied: Please specify location.

You need to have a GKE cluster in your project with the HTTP(S) Load Balancing add-on enabled:

Console

  1. Go to the Google Kubernetes Engine page in Cloud Console.

    Go to Google Kubernetes Engine

  2. Click the name of the cluster you want to modify.

  3. Under Networking, in the HTTP Load Balancing field, click Edit HTTP Load Balancing.

  4. Select the Enable HTTP load balancing checkbox.

  5. Click Save Changes.

gcloud

gcloud container clusters update cluster-name --update-addons=HttpLoadBalancing=ENABLED

where cluster-name is the name of your cluster.

Creating an external HTTP(S) load balancer

An external HTTP(S) load balancer provides one stable IP address that you can use to route requests to a variety of backend services. If you want a permanent IP address, you must reserve a global static external IP address.

In this exercise, you configure an external HTTP(S) load balancer to route requests to different backend services depending on the URL path. Requests that have the path / are routed to one backend service, and requests that have the path /v2 are routed to a different backend service.

Here's the big picture of the steps in this exercise:

  1. Create a Deployment and expose it with a Service named hello-world-1.
  2. Create a second Deployment and expose it with a Service named hello-world-2.
  3. Create an Ingress that specifies rules for routing requests to one Service or the other, depending on the URL path in the request. When you create the Ingress, the GKE Ingress controller creates and configures an external HTTP(S) load balancer.
  4. Test the external HTTP(S) load balancer.

Creating a Deployment

Here's a manifest for the first Deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: hello-world-deployment-1
spec:
  selector:
    matchLabels:
      greeting: hello
      version: one
  replicas: 3
  template:
    metadata:
      labels:
        greeting: hello
        version: one
    spec:
      containers:
      - name: hello-app-1
        image: "us-docker.pkg.dev/google-samples/containers/gke/hello-app:1.0"
        env:
        - name: "PORT"
          value: "50000"

Copy the manifest to a file named hello-world-1.yaml, and create the Deployment:

kubectl apply -f hello-world-1.yaml

Creating a Service

Here's a manifest for a Service that exposes your first Deployment:

apiVersion: v1
kind: Service
metadata:
  name: hello-world-1
spec:
  type: NodePort
  selector:
    greeting: hello
    version: one
  ports:
  - protocol: TCP
    port: 60000
    targetPort: 50000

For the purpose of this exercise, these are the important points to understand about this Service:

  • Any Pod that has both the greeting: hello label and the version: one label is a member of the Service.

  • When a request is sent to the Service on TCP port 60000, it is forwarded to one of the member Pods on TCP port 50000.

  • The Service manifest includes type: NodePort. For container native load balancing, you must use the type: ClusterIP.

Copy the manifest to a file named hello-world-service-1.yaml, and create the Service:

kubectl apply -f hello-world-service-1.yaml

Creating a second Deployment

Here's a manifest for a second Deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: hello-world-deployment-2
spec:
  selector:
    matchLabels:
      greeting: hello
      version: two
  replicas: 3
  template:
    metadata:
      labels:
        greeting: hello
        version: two
    spec:
      containers:
      - name: hello-app-2
        image: "us-docker.pkg.dev/google-samples/containers/gke/hello-app:2.0"
        env:
        - name: "PORT"
          value: "8080"

Copy the manifest to a file named hello-world-deployment-2.yaml, and create the Deployment:

kubectl apply -f hello-world-deployment-2.yaml

Creating a second Service

Here's a manifest for a Service that exposes your second Deployment:

apiVersion: v1
kind: Service
metadata:
  name: hello-world-2
spec:
  type: NodePort
  selector:
    greeting: hello
    version: two
  ports:
  - protocol: TCP
    port: 80
    targetPort: 8080

For the purpose of this exercise, these are the important points to understand about this Service:

  • Any Pod that has both the greeting: hello label and the version: two label is a member of the Service.

  • When a request is sent to the Service on TCP port 80, it is forwarded to one of the member Pods on TCP port 8080.

Copy the manifest to a file named hello-world-service-2.yaml, and create the Service:

kubectl apply -f hello-world-service-2.yaml

Creating an Ingress

Here's a manifest for an Ingress:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: my-ingress
  annotations:
    # If the class annotation is not specified it defaults to "gce".
    kubernetes.io/ingress.class: "gce"
spec:
  rules:
  - http:
      paths:
      - path: /*
        pathType: ImplementationSpecific
        backend:
          service:
            name: hello-world-1
            port:
              number: 60000
      - path: /v2
        pathType: ImplementationSpecific
        backend:
          service:
            name: hello-world-2
            port:
              number: 80

For the purpose of this exercise, these are the important points to understand about this Ingress manifest:

  • There are two Ingress classes available for GKE Ingress. The gce class deploys an external load balancer and the gce-internal class deploys an internal load balancer. Ingress resources without a class specified default to gce.

  • The only supported wildcard character for the path field of an Ingress is the * character. The * character must follow a forward slash (/) and must be the last character in the pattern. For example, /*, /foo/*, and /foo/bar/* are valid patterns, but *, /foo/bar*, and /foo/*/bar are not.

    A more specific pattern takes precedence over a less specific pattern. If you have both /foo/* and /foo/bar/*, then /foo/bar/bat is taken to match /foo/bar/*. For more information about path limitations and pattern matching, see the URL Maps documentation.

  • The Ingress manifest has two (serviceName, servicePort) pairs. Each (serviceName, servicePort) is associated with a Google Cloud backend service.

  • For GKE clusters running versions earlier than 1.21.3-gke.1600, the only supported value for the pathType field is ImplementationSpecific. For clusters running version 1.21.3-gke.1600 or later, Prefix and Exact values are also supported for pathType.

Copy the manifest to a file named my-ingress.yaml, and create the Ingress:

kubectl apply -f my-ingress.yaml

When you create the Ingress, the GKE Ingress controller creates an external HTTP(S) load balancer, and configures the load balancer as follows:

  • When a client sends a request to the load balancer with URL path /, the request is forwarded to the hello-world-1 Service on port 60000.

  • When a client sends a request to the load balancer using URL path /v2, the request is forwarded to the hello-world-2 Service on port 80.

Wait about five minutes for the load balancer to be configured.

Testing the external HTTP(S) load balancer

To test the external HTTP(S) load balancer:

  1. View the Ingress:

    kubectl get ingress my-ingress --output yaml
    

    The output shows the IP address of the external HTTP(S) load balancer:

    status:
      loadBalancer:
        ingress:
        - ip: 203.0.113.1
    
  2. Test the / path:

    curl load-balancer-ip/
    

    Replace load-balancer-ip with the external IP address of your load balancer.

    The output includes Version: 1.0.0:

    Hello, world!
    Version: 1.0.0
    Hostname: ...
    
  3. Test the /v2 path:

    curl load-balancer-ip/v2
    

    The output includes Version: 2.0.0:

    Hello, world!
    Version: 2.0.0
    Hostname: ...
    

HTTPS between client and load balancer

An external HTTP(S) load balancer acts as a proxy between your clients and your application. If you want to accept HTTPS requests from your clients, the load balancer must have a certificate so it can prove its identity to your clients. The load balancer must also have a private key to complete the HTTPS handshake. For more information, see:

Disabling HTTP

If you want all traffic between the client and the load balancer to use HTTPS, you can disable HTTP. For more information, see Disabling HTTP.

HTTPS between load balancer and application

If your application, running in a GKE Pod, is capable of receiving HTTPS requests, you can configure the load balancer to use HTTPS when it forwards requests to your application. For more information, see HTTPS (TLS) between load balancer and your application.

HTTP/2 between client and load balancer

Clients can use HTTP/2 to send requests to the load balancer. No configuration is required.

HTTP/2 between load balancer and application

If your application, running in a GKE Pod, is capable of receiving HTTP/2 requests, you can configure the load balancer to use HTTP/2 when it forwards requests to your application. For more information, see HTTP/2 for load balancing with Ingress.

Network endpoint groups

If your cluster supports Container-native load balancing, it is recommended to use network endpoint groups (NEGs). For GKE clusters 1.17 and later and under certain conditions, container-native load balancing is default and does not require an explicit cloud.google.com/neg: '{"ingress": true}' Service annotation.

Shared VPC

If the GKE cluster in which you are deploying the Ingress resources is in a service project, and you want the GKE control plane to manage the firewall resources in your host project, then the service project's GKE service account must be granted the appropriate IAM permissions in the host project as per Managing firewall resources for clusters with Shared VPC. This lets the Ingress controller create firewall rules to allow both ingress traffic and traffic for Google Cloud health checks.

The following is an example of an event that might be present in the Ingress resource logs. This error occurs when the Ingress controller is unable to create a firewall rule to allow ingress traffic for Google Cloud health checks if the permissions are not configured correctly.

Firewall change required by security admin: `gcloud compute firewall-rules update <RULE_NAME> --description "GCE L7 firewall rule" --allow tcp:<PORT> --source-ranges 130.211.0.0/22,35.191.0.0/16 --target-tags <TARGET_TAG> --project <HOST_PROJECT>

If you prefer to manually provision firewall rules from the host project, then you can mute the firewallXPNError events by adding the networking.gke.io/suppress-firewall-xpn-error: "true" annotation to the Ingress resource.

Summary of external Ingress annotations

Ingress annotations

Annotation Description
kubernetes.io/ingress.allow-http Specifies whether to allow HTTP traffic between the client and the HTTP(S) load balancer. Possible values are "true" and "false". Default is "true". See Disabling HTTP.
ingress.gcp.kubernetes.io/pre-shared-cert You can upload certificates and keys to your Google Cloud project. Use this annotation to reference the certificates and keys. See Using multiple SSL certificates in HTTP(S) Load Balancing.
kubernetes.io/ingress.global-static-ip-name Use this annotation to specify that the load balancer should use a static external IP address that you previously created. See Static IP addresses for HTTP(S) load balancers.
networking.gke.io/v1beta1.FrontendConfig Use this annotation to customize the client-facing configuration of the load balancer. For more information, see Ingress features.
networking.gke.io/suppress-firewall-xpn-error For Ingress Load Balancers, if Kubernetes can't change the firewall rules due to insufficient permission, a firewallXPNError event is created every several minutes. In GLBC 1.4 and later, you can mute the firewallXPNError event by adding networking.gke.io/suppress-firewall-xpn-error: "true" annotation to the ingress resource. You can remove this annotation to unmute. Possible values are true and false. The default value is false.
Annotation Description
service.alpha.kubernetes.io/app-protocols Use this annotation to set the protocol for communication between the load balancer and the application. Possible protocols are HTTP, HTTPS, and HTTP/2. See HTTPS between load balancer and your application and HTTP/2 for load balancing with Ingress.
beta.cloud.google.com/backend-config Use this annotation to configure the backend service associated with a servicePort. For more information, see Ingress features.
cloud.google.com/neg Use this annotation to specify that the load balancer should use network endpoint groups. See Using Container-native Load Balancing.

What's next