Configure Ingress for external Application Load Balancers

This page shows you how to configure an external Application Load Balancer by creating a Kubernetes Ingress object.

Before reading this page, ensure that you're familiar with GKE networking concepts.

Before you begin

Before you start, make sure you have performed the following tasks:

  • Enable the Google Kubernetes Engine API.
    • Enable Google Kubernetes Engine API
  • If you want to use the Google Cloud CLI for this task, install and then initialize the gcloud CLI. If you previously installed the gcloud CLI, get the latest version by running gcloud components update.

Enable the HttpLoadBalancing add-on

Your cluster must have the HttpLoadBalancing add-on enabled. This add-on is enabled by default. In Autopilot clusters, you can't disable this add-on.

You can enable the HttpLoadBalancing add-on using the Google Cloud console or the Google Cloud CLI.

Console

  1. Go to the Google Kubernetes Engine page in the Google Cloud console.

    Go to Google Kubernetes Engine

  2. Click the name of the cluster you want to modify.

  3. Under Networking, in the HTTP Load Balancing field, click Edit HTTP Load Balancing.

  4. Select the Enable HTTP load balancing checkbox.

  5. Click Save Changes.

gcloud

gcloud container clusters update CLUSTER_NAME --update-addons=HttpLoadBalancing=ENABLED

Replace CLUSTER_NAME with the name of your cluster.

Create a static IP address

An external Application Load Balancer provides one stable IP address that you can use to route requests to one or more Services. If you want a permanent IP address, you must reserve a global static external IP address before you create an Ingress.

If you modify an existing Ingress to use a static IP address instead of an ephemeral IP address, GKE might change the IP address of the load balancer when GKE re-creates the forwarding rule of the load balancer.

Create an external Application Load Balancer

In this exercise, you configure an external Application Load Balancer to route requests to different Services depending on the URL path.

To follow step-by-step guidance for this task directly in the Google Cloud console, click Guide me:

Guide me

Create Deployments and Services

Create two Deployments with Services named hello-world-1 and hello-world-2:

  1. Save the following manifest as hello-world-deployment-1.yaml:

    apiVersion: apps/v1
kind: Deployment
metadata:
  name: hello-world-deployment-1
spec:
  selector:
    matchLabels:
      greeting: hello
      version: one
  replicas: 3
  template:
    metadata:
      labels:
        greeting: hello
        version: one
    spec:
      containers:
      - name: hello-app-1
        image: "us-docker.pkg.dev/google-samples/containers/gke/hello-app:1.0"
        env:
        - name: "PORT"
          value: "50000"

    This manifest describes a sample Deployment with three replicas.

  2. Apply the manifest to the cluster:

    kubectl apply -f hello-world-deployment-1.yaml

  3. Save the following manifest as hello-world-service-1.yaml:

    apiVersion: v1
kind: Service
metadata:
  name: hello-world-1
spec:
  type: NodePort
  selector:
    greeting: hello
    version: one
  ports:
  - protocol: TCP
    port: 60000
    targetPort: 50000

    This manifest describes a Service with the following properties:

    • Any Pod that has both the greeting: hello label and the version: one label is a member of the Service.
    • GKE forwards requests sent to the Service on TCP port 60000 to one of the member Pods on TCP port 50000.
    • The Service type is NodePort, which is required unless you're using container native load balancing. If using container-native load balancing, there is no restriction on type of service. Recommend to use type: ClusterIP.

  4. Apply the manifest to the cluster:

    kubectl apply -f hello-world-service-1.yaml

  5. Save the following manifest as hello-world-deployment-2.yaml:

    apiVersion: apps/v1
kind: Deployment
metadata:
  name: hello-world-deployment-2
spec:
  selector:
    matchLabels:
      greeting: hello
      version: two
  replicas: 3
  template:
    metadata:
      labels:
        greeting: hello
        version: two
    spec:
      containers:
      - name: hello-app-2
        image: "us-docker.pkg.dev/google-samples/containers/gke/hello-app:2.0"
        env:
        - name: "PORT"
          value: "8080"

    This manifest describes a sample Deployment with three replicas.

  6. Apply the manifest to the cluster:

    kubectl apply -f hello-world-deployment-2.yaml

  7. Save the following manifest as hello-world-service-2.yaml:

    apiVersion: v1
kind: Service
metadata:
  name: hello-world-2
spec:
  type: NodePort
  selector:
    greeting: hello
    version: two
  ports:
  - protocol: TCP
    port: 80
    targetPort: 8080

    This manifest describes a Service with the following properties:

    • Any Pod that has both the greeting: hello label and the version: two label is a member of the Service.
    • GKE forwards requests sent to the Service on TCP port 80 to one of the member Pods on TCP port 8080.

  8. Apply the manifest to the cluster:

    kubectl apply -f hello-world-service-2.yaml

Create an Ingress

Create an Ingress that specifies rules for routing requests depending on the URL path in the request. When you create the Ingress, the GKE Ingress controller creates and configures an external Application Load Balancer.

  1. Save the following manifest as my-ingress.yaml:

    apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: my-ingress
  annotations:
    # If the class annotation is not specified it defaults to "gce".
    kubernetes.io/ingress.class: "gce"
spec:
  rules:
  - http:
      paths:
      - path: /*
        pathType: ImplementationSpecific
        backend:
          service:
            name: hello-world-1
            port:
              number: 60000
      - path: /v2
        pathType: ImplementationSpecific
        backend:
          service:
            name: hello-world-2
            port:
              number: 80

    This manifest describes an Ingress with the following properties:

    • There are two GKE Ingress classes. To specify an Ingress class, you must use the kubernetes.io/ingress.class annotation. You cannot specify a GKE Ingress using spec.ingressClassName.

    • The gce class deploys an external Application Load Balancer.

    • The gce-internal class deploys an internal Application Load Balancer.

    • When you deploy an Ingress resource without the annotations spec.ingressClassName and kubernetes.io/ingress.class, GKE creates an external Application Load Balancer. This is the same behavior that occurs if you specify the kubernetes.io/ingress.class: gce annotation. For more information, see GKE Ingress controller behavior.

    • GKE creates a Google Cloud backend Service for each backend.service. Each of the backend services corresponds to a Kubernetes Service, and each backend service must reference a Google Cloud health check. This health check is different from a Kubernetes liveness or readiness probe because the health check is implemented outside of the cluster. To learn more, see Health checks

    • When a client sends a request to the load balancer with URL path /, GKE forwards the request to the hello-world-1 Service on port 60000. When a client sends a request to the load balancer using URL path /v2, GKE forwards the request to the hello-world-2 Service on port 80. For more information about the path and pathType properties, see URL Paths.

  2. Apply the manifest to the cluster:

    kubectl apply -f my-ingress.yaml

Test the external Application Load Balancer

Wait about five minutes for the load balancer to be configured, then test the external Application Load Balancer:

  1. View the Ingress:

    kubectl get ingress my-ingress --output yaml

    The output shows the IP address of the external Application Load Balancer:

    status:
  loadBalancer:
    ingress:
    - ip: 203.0.113.1

  2. Test the / path:

    curl LOAD_BALANCER_IP_ADDRESS/

    Replace LOAD_BALANCER_IP_ADDRESS with the external IP address of the load balancer.

    The output is similar to the following:

    Hello, world!
Version: 1.0.0
Hostname: ...

    If the output includes a 404 error, wait a few minutes.

  3. Test the /v2 path:

    curl load-balancer-ip/v2

    The output is similar to the following:

    Hello, world!
Version: 2.0.0
Hostname: ...

How Ingress for external load balancing works

An external Application Load Balancer acts as a proxy between your clients and your application. If you want to accept HTTPS requests from your clients, the load balancer must have a certificate so it can prove its identity to your clients. The load balancer must also have a private key to complete the HTTPS handshake. For more information, see:

URL Paths

The only supported wildcard character for the path field of an Ingress is the * character. The * character must follow a forward slash (/) and must be the last character in the pattern. For example, /*, /foo/*, and /foo/bar/* are valid patterns, but *, /foo/bar*, and /foo/*/bar are not.

A more specific pattern takes precedence over a less specific pattern. If you have both /foo/* and /foo/bar/*, then /foo/bar/bat is taken to match /foo/bar/*. For more information about path limitations and pattern matching, see the URL Maps documentation.

For GKE clusters running versions earlier than 1.21.3-gke.1600, the only supported value for the pathType field is ImplementationSpecific. For clusters running version 1.21.3-gke.1600 or later, Prefix and Exact values are also supported for pathType.

Disabling HTTP

If you want all traffic between the client and the load balancer to use HTTPS, you can disable HTTP. For more information, see Disabling HTTP.

HTTPS between load balancer and application

If your application, running in a GKE Pod, is capable of receiving HTTPS requests, you can configure the load balancer to use HTTPS when it forwards requests to your application. For more information, see HTTPS (TLS) between load balancer and your application.

HTTP/2 between client and load balancer

Clients can use HTTP/2 to send requests to the load balancer. No configuration is required.

HTTP/2 between load balancer and application

If your application, running in a GKE Pod, is capable of receiving HTTP/2 requests, you can configure the load balancer to use HTTP/2 when it forwards requests to your application. For more information, see HTTP/2 for load balancing with Ingress.

Network endpoint groups

If your cluster supports Container-native load balancing, it is recommended to use network endpoint groups (NEGs). For GKE clusters 1.17 and later and under certain conditions, container-native load balancing is default and does not require an explicit cloud.google.com/neg: '{"ingress": true}' Service annotation.

Shared VPC

If the GKE cluster in which you are deploying the Ingress resources is in a service project, and you want the GKE control plane to manage the firewall resources in your host project, then the service project's GKE service account must be granted the appropriate IAM permissions in the host project as per Managing firewall resources for clusters with Shared VPC. This lets the Ingress controller create firewall rules to allow both ingress traffic and traffic for Google Cloud health checks.

The following is an example of an event that might be present in the Ingress resource logs. This error occurs when the Ingress controller is unable to create a firewall rule to allow ingress traffic for Google Cloud health checks if the permissions are not configured correctly.

Firewall change required by security admin: `gcloud compute firewall-rules update <RULE_NAME> --description "GCE L7 firewall rule" --allow tcp:<PORT> --source-ranges 130.211.0.0/22,35.191.0.0/16 --target-tags <TARGET_TAG> --project <HOST_PROJECT>

If you prefer to manually provision firewall rules from the host project, then you can mute the firewallXPNError events by adding the networking.gke.io/suppress-firewall-xpn-error: "true" annotation to the Ingress resource.

Summary of external Ingress annotations

Ingress annotations

Annotation Description
kubernetes.io/ingress.allow-http Specifies whether to allow HTTP traffic between the client and the HTTP(S) load balancer. Possible values are "true" and "false". Default is "true". See Disabling HTTP.
ingress.gcp.kubernetes.io/pre-shared-cert Use this annotation to attach certificate resources to GKE Ingress resources. For more information, see Using multiple SSL certificates with external Application Load Balancers.
kubernetes.io/ingress.global-static-ip-name Use this annotation to specify that the load balancer should use a static external IP address that you previously created. See Static IP addresses for HTTP(S) load balancers.
networking.gke.io/v1beta1.FrontendConfig Use this annotation to customize the client-facing configuration of the load balancer. For more information, see Ingress configuration.
networking.gke.io/suppress-firewall-xpn-error For Ingress Load Balancers, if Kubernetes can't change the firewall rules due to insufficient permission, a firewallXPNError event is created every several minutes. In GLBC 1.4 and later, you can mute the firewallXPNError event by adding networking.gke.io/suppress-firewall-xpn-error: "true" annotation to the ingress resource. You can remove this annotation to unmute. Possible values are true and false. The default value is false.
Annotation Description
cloud.google.com/app-protocols Use this annotation to set the protocol for communication between the load balancer and the application. Possible protocols are HTTP, HTTPS, and HTTP2. See HTTPS between load balancer and your application and HTTP/2 for load balancing with Ingress.
cloud.google.com/backend-config Use this annotation to configure the backend service associated with a Service. For more information, see Ingress configuration.
cloud.google.com/neg Use this annotation to specify that the load balancer should use network endpoint groups. See Using Container-native Load Balancing.

What's next