This page shows you how to configure an external Application Load Balancer by creating a Kubernetes Ingress object.
Before reading this page, ensure that you're familiar with GKE networking concepts.
Before you begin
Before you start, make sure you have performed the following tasks:
- Enable the Google Kubernetes Engine API. Enable Google Kubernetes Engine API
- If you want to use the Google Cloud CLI for this task,
install and then
initialize the
gcloud CLI. If you previously installed the gcloud CLI, get the latest
version by running
gcloud components update
.
Enable the HttpLoadBalancing
add-on
Your cluster must have the HttpLoadBalancing
add-on enabled. This add-on is
enabled by default. In Autopilot clusters, you can't disable this
add-on.
You can enable the HttpLoadBalancing
add-on using the Google Cloud console or the
Google Cloud CLI.
Console
Go to the Google Kubernetes Engine page in the Google Cloud console.
Click the name of the cluster you want to modify.
Under Networking, in the HTTP Load Balancing field, click edit Edit HTTP Load Balancing.
Select the Enable HTTP load balancing checkbox.
Click Save Changes.
gcloud
gcloud container clusters update CLUSTER_NAME --update-addons=HttpLoadBalancing=ENABLED
Replace CLUSTER_NAME
with the name of your cluster.
Create a static IP address
An external Application Load Balancer provides one stable IP address that you can use to route requests to one or more Services. If you want a permanent IP address, you must reserve a global static external IP address before you create an Ingress.
If you modify an existing Ingress to use a static IP address instead of an ephemeral IP address, GKE might change the IP address of the load balancer when GKE re-creates the forwarding rule of the load balancer.
Create an external Application Load Balancer
In this exercise, you configure an external Application Load Balancer to route requests to different Services depending on the URL path.
To follow step-by-step guidance for this task directly in the Google Cloud console, click Guide me:
Create Deployments and Services
Create two Deployments with Services named hello-world-1
and hello-world-2
:
Save the following manifest as
hello-world-deployment-1.yaml
:apiVersion: apps/v1 kind: Deployment metadata: name: hello-world-deployment-1 spec: selector: matchLabels: greeting: hello version: one replicas: 3 template: metadata: labels: greeting: hello version: one spec: containers: - name: hello-app-1 image: "us-docker.pkg.dev/google-samples/containers/gke/hello-app:1.0" env: - name: "PORT" value: "50000"
This manifest describes a sample Deployment with three replicas.
Apply the manifest to the cluster:
kubectl apply -f hello-world-deployment-1.yaml
Save the following manifest as
hello-world-service-1.yaml
:apiVersion: v1 kind: Service metadata: name: hello-world-1 spec: type: NodePort selector: greeting: hello version: one ports: - protocol: TCP port: 60000 targetPort: 50000
This manifest describes a Service with the following properties:
- Any Pod that has both the
greeting: hello
label and theversion: one
label is a member of the Service. - GKE forwards requests sent to the Service on TCP port 60000 to one of the member Pods on TCP port 50000.
- The Service type is
NodePort
, which is required unless you're using container native load balancing. If using container-native load balancing, there is no restriction on type of service. Recommend to usetype: ClusterIP
.
- Any Pod that has both the
Apply the manifest to the cluster:
kubectl apply -f hello-world-service-1.yaml
Save the following manifest as
hello-world-deployment-2.yaml
:apiVersion: apps/v1 kind: Deployment metadata: name: hello-world-deployment-2 spec: selector: matchLabels: greeting: hello version: two replicas: 3 template: metadata: labels: greeting: hello version: two spec: containers: - name: hello-app-2 image: "us-docker.pkg.dev/google-samples/containers/gke/hello-app:2.0" env: - name: "PORT" value: "8080"
This manifest describes a sample Deployment with three replicas.
Apply the manifest to the cluster:
kubectl apply -f hello-world-deployment-2.yaml
Save the following manifest as
hello-world-service-2.yaml
:apiVersion: v1 kind: Service metadata: name: hello-world-2 spec: type: NodePort selector: greeting: hello version: two ports: - protocol: TCP port: 80 targetPort: 8080
This manifest describes a Service with the following properties:
- Any Pod that has both the
greeting: hello
label and theversion: two
label is a member of the Service. - GKE forwards requests sent to the Service on TCP port 80 to one of the member Pods on TCP port 8080.
- Any Pod that has both the
Apply the manifest to the cluster:
kubectl apply -f hello-world-service-2.yaml
Create an Ingress
Create an Ingress that specifies rules for routing requests depending on the URL path in the request. When you create the Ingress, the GKE Ingress controller creates and configures an external Application Load Balancer.
Save the following manifest as
my-ingress.yaml
:apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: my-ingress annotations: # If the class annotation is not specified it defaults to "gce". kubernetes.io/ingress.class: "gce" spec: rules: - http: paths: - path: /* pathType: ImplementationSpecific backend: service: name: hello-world-1 port: number: 60000 - path: /v2 pathType: ImplementationSpecific backend: service: name: hello-world-2 port: number: 80
This manifest describes an Ingress with the following properties:
There are two GKE Ingress classes. To specify an Ingress class, you must use the
kubernetes.io/ingress.class
annotation. You cannot specify a GKE Ingress usingspec.ingressClassName
.The
gce
class deploys an external Application Load Balancer.The
gce-internal
class deploys an internal Application Load Balancer.When you deploy an Ingress resource without the annotations
spec.ingressClassName
andkubernetes.io/ingress.class
, GKE creates an external Application Load Balancer. This is the same behavior that occurs if you specify thekubernetes.io/ingress.class: gce
annotation. For more information, see GKE Ingress controller behavior.GKE creates a Google Cloud backend Service for each
backend.service
. Each of the backend services corresponds to a Kubernetes Service, and each backend service must reference a Google Cloud health check. This health check is different from a Kubernetes liveness or readiness probe because the health check is implemented outside of the cluster. To learn more, see Health checksWhen a client sends a request to the load balancer with URL path
/
, GKE forwards the request to thehello-world-1
Service on port 60000. When a client sends a request to the load balancer using URL path/v2
, GKE forwards the request to thehello-world-2
Service on port 80. For more information about thepath
andpathType
properties, see URL Paths.
Apply the manifest to the cluster:
kubectl apply -f my-ingress.yaml
Test the external Application Load Balancer
Wait about five minutes for the load balancer to be configured, then test the external Application Load Balancer:
View the Ingress:
kubectl get ingress my-ingress --output yaml
The output shows the IP address of the external Application Load Balancer:
status: loadBalancer: ingress: - ip: 203.0.113.1
Test the
/
path:curl LOAD_BALANCER_IP_ADDRESS/
Replace
LOAD_BALANCER_IP_ADDRESS
with the external IP address of the load balancer.The output is similar to the following:
Hello, world! Version: 1.0.0 Hostname: ...
If the output includes a 404 error, wait a few minutes.
Test the
/v2
path:curl load-balancer-ip/v2
The output is similar to the following:
Hello, world! Version: 2.0.0 Hostname: ...
How Ingress for external load balancing works
An external Application Load Balancer acts as a proxy between your clients and your application. If you want to accept HTTPS requests from your clients, the load balancer must have a certificate so it can prove its identity to your clients. The load balancer must also have a private key to complete the HTTPS handshake. For more information, see:
- Setting up HTTPS (TLS) between client and load balancer.
- Using multiple SSL certificates in HTTPS Load Balancing with Ingress.
URL Paths
The only supported wildcard character for the path
field of an Ingress
is the *
character. The *
character must follow a forward slash (/
) and
must be the last character in the pattern. For example, /*
, /foo/*
, and
/foo/bar/*
are valid patterns, but *
, /foo/bar*
, and /foo/*/bar
are not.
A more specific pattern takes precedence over a less specific pattern. If you
have both /foo/*
and /foo/bar/*
, then /foo/bar/bat
is taken to match
/foo/bar/*
. For more information about path limitations and pattern matching,
see the
URL Maps documentation.
For GKE clusters running versions earlier than 1.21.3-gke.1600,
the only supported value for the pathType
field is ImplementationSpecific
.
For clusters running version 1.21.3-gke.1600 or later, Prefix
and Exact
values are also supported for pathType
.
Disabling HTTP
If you want all traffic between the client and the load balancer to use HTTPS, you can disable HTTP. For more information, see Disabling HTTP.
HTTPS between load balancer and application
If your application, running in a GKE Pod, is capable of receiving HTTPS requests, you can configure the load balancer to use HTTPS when it forwards requests to your application. For more information, see HTTPS (TLS) between load balancer and your application.
HTTP/2 between client and load balancer
Clients can use HTTP/2 to send requests to the load balancer. No configuration is required.
HTTP/2 between load balancer and application
If your application, running in a GKE Pod, is capable of receiving HTTP/2 requests, you can configure the load balancer to use HTTP/2 when it forwards requests to your application. For more information, see HTTP/2 for load balancing with Ingress.
Network endpoint groups
If your cluster supports
Container-native load balancing,
it is recommended to use network endpoint groups (NEGs). For GKE
clusters 1.17 and later and
under certain conditions,
container-native load balancing is default and does not require an explicit
cloud.google.com/neg: '{"ingress": true}'
Service annotation.
Shared VPC
If the GKE cluster in which you are deploying the Ingress resources is in a service project, and you want the GKE control plane to manage the firewall resources in your host project, then the service project's GKE service account must be granted the appropriate IAM permissions in the host project as per Managing firewall resources for clusters with Shared VPC. This lets the Ingress controller create firewall rules to allow both ingress traffic and traffic for Google Cloud health checks.
The following is an example of an event that might be present in the Ingress resource logs. This error occurs when the Ingress controller is unable to create a firewall rule to allow ingress traffic for Google Cloud health checks if the permissions are not configured correctly.
Firewall change required by security admin: `gcloud compute firewall-rules update <RULE_NAME> --description "GCE L7 firewall rule" --allow tcp:<PORT> --source-ranges 130.211.0.0/22,35.191.0.0/16 --target-tags <TARGET_TAG> --project <HOST_PROJECT>
If you prefer to manually provision firewall rules from the host project, then you can mute the firewallXPNError
events
by adding the networking.gke.io/suppress-firewall-xpn-error: "true"
annotation to the Ingress resource.
Summary of external Ingress annotations
Ingress annotations
Annotation | Description |
---|---|
kubernetes.io/ingress.allow-http | Specifies whether to allow HTTP traffic between the client and the HTTP(S) load balancer. Possible values are "true" and "false". Default is "true". See Disabling HTTP. |
ingress.gcp.kubernetes.io/pre-shared-cert | Use this annotation to attach certificate resources to GKE Ingress resources. For more information, see Using multiple SSL certificates with external Application Load Balancers. |
kubernetes.io/ingress.global-static-ip-name | Use this annotation to specify that the load balancer should use a static external IP address that you previously created. See Static IP addresses for HTTP(S) load balancers. |
networking.gke.io/v1beta1.FrontendConfig | Use this annotation to customize the client-facing configuration of the load balancer. For more information, see Ingress configuration. |
networking.gke.io/suppress-firewall-xpn-error | For Ingress Load Balancers, if Kubernetes can't change the firewall rules due to
insufficient permission, a firewallXPNError event is created every several
minutes. In GLBC 1.4
and later, you can mute the firewallXPNError event by
adding networking.gke.io/suppress-firewall-xpn-error: "true" annotation to the
ingress resource. You can remove this annotation to unmute. Possible values are true and false .
The default value is false .
|
Service annotations related to Ingress
Annotation | Description |
---|---|
cloud.google.com/app-protocols | Use this annotation to set the protocol for communication between the load balancer and the application. Possible protocols are HTTP, HTTPS, and HTTP2. See HTTPS between load balancer and your application and HTTP/2 for load balancing with Ingress. |
cloud.google.com/backend-config | Use this annotation to configure the backend service associated with a Service. For more information, see Ingress configuration. |
cloud.google.com/neg | Use this annotation to specify that the load balancer should use network endpoint groups. See Using Container-native Load Balancing. |
What's next
- Read a conceptual overview of Ingress for external Application Load Balancers in GKE.
- Perform the tutorial on Setting up an external Application Load Balancer with Ingress.
Read a conceptual overview of Services in GKE.
Implement a basic external Ingress.