GKE Ingress for HTTP(S) Load Balancing

This page provides a general overview of how Ingress for HTTP(S) Load Balancing works. Google Kubernetes Engine (GKE) provides a built-in and managed Ingress controller called GKE Ingress. This controller implements Ingress resources as Google Cloud load balancers for HTTP(S) workloads in GKE.

Overview

In GKE, an Ingress object defines rules for routing HTTP(S) traffic to applications running in a cluster. An Ingress object is associated with one or more Service objects, each of which is associated with a set of Pods. To learn more about how Ingress exposes applications using Services, see Service networking overview.

When you create an Ingress object, the GKE Ingress controller creates a Google Cloud HTTP(S) Load Balancer and configures it according to the information in the Ingress and its associated Services.

To use Ingress, you must have the HTTP load balancing add-on enabled. GKE clusters have HTTP load balancing enabled by default; you must not disable it.

Ingress for external and internal traffic

GKE Ingress resources come in two types:

• Ingress for external HTTP(S) load balancer deploys the Google Cloud external HTTP(S) load balancer. This internet-facing load balancer is deployed globally across Google's edge network as a managed and scalable pool of load balancing resources. Learn how to set up and use Ingress for external HTTP(S) load balancer.

• Ingress for Internal HTTP(S) Load Balancing deploys the Google Cloud internal HTTP(S) load balancer. These internal HTTP(S) load balancers are powered by Envoy proxy systems outside of your GKE cluster, but within your VPC network. Learn how to set up and use Ingress for Internal HTTP(S) Load Balancing.

Important: Whenever GKE creates an external HTTP(S) load balancer or internal HTTP(S) load balancer through an Ingress object, you should avoid changing the load balancer's configuration using methods outside of GKE. Your customizations to settings for the load balancer's objects – forwarding rules, target proxies, URL maps, backend services, and health checks – might be overwritten. If you need to manage an external HTTP(S) load balancer or internal HTTP(S) load balancer outside of GKE, use container native load balancing for standalone NEGs instead.

Features of HTTP(S) load balancing

HTTP(S) load balancing, configured by Ingress, includes the following features:

• Flexible configuration for Services. An Ingress defines how traffic reaches your Services and how the traffic is routed to your application. In addition, an Ingress can provide a single IP address for multiple Services in your cluster.
• Integration with Google Cloud network services
• Support for multiple TLS certificates. An Ingress can specify the use of multiple TLS certificates for request termination.

For a comprehensive list, see Ingress features.

Container-native load balancing

Container-native load balancing is the practice of load balancing directly to Pod endpoints in GKE using Network Endpoint Groups (NEGs).

When using Instance Groups, Compute Engine load balancers send traffic to VM IPs as backends. When running containers on VMs, in which containers share the same host interface, this introduces some limitations:

• It incurs two hops of load balancing - one hop from the load balancer to the VM NodePort and another hop through kube-proxy routing to the Pod IP (which may reside on a different VM).
• Additional hops add latency and make the traffic path more complex.
• The Compute Engine load balancer has no direct visibility to Pods resulting in suboptimal traffic balancing.
• Environmental events like VM or Pod loss are more likely to cause intermittent traffic loss due to the double traffic hop.

With NEGs, traffic is load balanced from the load balancer directly to the Pod IP as opposed to traversing the VM IP and kube-proxy networking. In addition, Pod readiness gates are implemented to determine the health of Pods from the perspective of the load balancer and not just the Kubernetes in-cluster health probes. This improves overall traffic stability by making the load balancer infrastructure aware of lifecycle events such as Pod startup, Pod loss, or VM loss. These capabilities resolve the above limitations and result in more performant and stable networking.

Container-native load balancing is enabled by default for Services when all of the following conditions are true:

• For Services created in GKE clusters 1.17.6-gke.7 and up
• Using VPC-native clusters
• Not using a Shared VPC
• Not using GKE Network Policy

In these conditions, Services will be annotated automatically with cloud.google.com/neg: '{"ingress": true}' indicating that a NEG should be created to mirror the Pod IPs within the Service. The NEG is what allows Compute Engine load balancers to communicate directly with Pods. Note that existing Services created prior to GKE 1.17.6-gke.7+ will not be automatically annotated by the Service controller.

For GKE 1.17.6-gke.7+ clusters where NEG annotation is automatic, it is possible to disable NEGs and force the Compute Engine load balancer to use an instance group as its backends if necessary. This can be done by explicitly annotating Services with cloud.google.com/neg: '{"ingress": false}'.

For clusters where NEGs are not the default, it is still strongly recommended to use container-native load balancing, but it must be enabled explicitly on a per-Service basis. The annotation should be applied to Services in the following manner:

kind: Service
...
  annotations:
    cloud.google.com/neg: '{"ingress": true}'
...

Caution: When you add this annotation, a new BackendService is created for the existing Service. This can result in a temporary outage for your Service.

Shared VPC

Ingress and MultiClusterIngress resources are supported in Shared VPC topologies, but require some upfront preparation to work. The GKE Ingress controllers use a Google Cloud service account to deploy and manage Google Cloud resources. When a GKE cluster resides in a service project of a Shared VPC, this service account does not have the rights to manage network resources owned by the host project. The Ingress controller actively manages firewall rules to provide access between load balancers and Pods and also between centralized health checkers and Pods.

However, in a Shared VPC, the GKE Ingress controller has no rights to manage firewall rules. You can add rights by manually provisioning firewall rules from the host project or by providing the GKE Ingress controller permission to manage host project firewall rules.

Manually provision firewall rules from the host project

If your security policies only allow firewall management from the host project, then you can provision these firewall rules manually. When deploying Ingress in a Shared VPC, the Ingress resource event provides the specific firewall rule you need to add necessary to provide access.

To manually provision a rule:

1. View the Ingress resource event:

   kubectl describe ingress INGRESS_NAME
   

   Replace INGRESS_NAME with the name of your Ingress.

   You should see output similar to the following example:

   Events:
   Type    Reason  Age                    From                     Message
   ----    ------  ----                   ----                     -------
   Normal  Sync    9m34s (x237 over 38h)  loadbalancer-controller  Firewall change required by network admin: `gcloud compute firewall-rules update k8s-fw-l7--6048d433d4280f11 --description "GCE L7 firewall rule" --allow tcp:30000-32767,tcp:8080 --source-ranges 130.211.0.0/22,209.85.152.0/22,209.85.204.0/22,35.191.0.0/16 --target-tags gke-l7-ilb-test-b3a7e0e5-node --project <project>`
   

   The suggested required firewall rule appears in the Message column.

2. Copy and apply the suggested firewall rules from the host project. Applying the rule provides access to your Pods from the load balancer and Google Cloud health checkers.

   Note: Different types of Ingress require different rules.

Providing the Ingress controller permission to manage host project firewall rules

An automated approach is to provide the GKE Ingress controller service account the permissions to update firewall rules. Create a custom IAM role which provides the ability to manage firewall rules directly and grant this role to the GKE Ingress service account.

1. In your host project, grant a custom role with the compute.firewalls.* and compute.networks.updatePolicy permissions to the GKE service account of the service project:

   gcloud iam roles create ROLE_NAME \
      --project PROJECT_ID \
      --title ROLE_TITLE \
      --description ROLE_DESCRIPTION \
      --permissions=compute.networks.updatePolicy, compute.firewalls.*\
      --stage GA
   

   Replace the following:

   • ROLE_NAME: add a name for the role.
   • PROJECT_ID: add the project ID of the host project.
   • ROLE_TITLE: add the title of the role you want to create.
   • ROLE_DESCRIPTION: add the description of the role you want to create.

2. Apply this custom role to the GKE Ingress service account:

   gcloud projects add-iam-policy-binding my-project \
       --member=user:SERVICE_ACCOUNT  \
       --role=roles/gke-ingress-fw-management
   

   The value of SERVICE_ACCOUNT differs between GKE Ingress and multi-cluster Ingress:

   • If you are using GKE Ingress, then SERVICE_ACCOUNT is: [e-mail].

   • If you are using multi-cluster Ingress, then SERVICE_ACCOUNT is: [e-mail].

   Replace PROJECT_NUMBER with your host project's project number.

Multiple backend services

Each external HTTP(S) load balancer or internal HTTP(S) load balancer uses a single URL map, which references one or more backend services. One backend service corresponds to each Service referenced by the Ingress.

For example, you can configure the load balancer to route requests to different backend services depending on the URL path. Requests sent to your-store.example could be routed to a backend service that displays full-price items, and requests sent to your-store.example/discounted could be routed to a backend service that displays discounted items.

You can also configure the load balancer to route requests according to the hostname. Requests sent to your-store.example could go to one backend service, and requests sent to your-experimental-store.example could go to another backend service.

In a GKE cluster, you create and configure an HTTP(S) load balancer by creating a Kubernetes Ingress object. An Ingress object must be associated with one or more Service objects, each of which is associated with a set of Pods.

Here is a manifest for an Ingress called my-ingress:

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: my-ingress
spec:
  rules:
  - http:
      paths:
      - path: /*
        backend:
          serviceName: my-products
          servicePort: 60000
      - path: /discounted
        backend:
          serviceName: my-discounted-products
          servicePort: 80

Note: To use Ingress, you must have the HTTP(S) Load Balancing add-on enabled. GKE clusters have HTTP(S) Load Balancing enabled by default; you must not disable it.

When you create the Ingress, the GKE ingress controller creates and configures an external HTTP(S) load balancer or internal HTTP(S) load balancer according to the information in the Ingress and the associated Services. Also, the load balancer is given a stable IP address that you can associate with a domain name.

In the preceding example, assume you have associated the load balancer's IP address with the domain name your-store.example. When a client sends a request to your-store.example, the request is routed to a Kubernetes Service named my-products on port 60000. And when a client sends a request to your-store.example/discounted, the request is routed to a Kubernetes Service named my-discounted-products on port 80.

The only supported wildcard character for the path field of an Ingress is the * character. The * character must follow a forward slash (/) and must be the last character in the pattern. For example, /*, /foo/*, and /foo/bar/* are valid patterns, but *, /foo/bar*, and /foo/*/bar are not.

A more specific pattern takes precedence over a less specific pattern. If you have both /foo/* and /foo/bar/*, then /foo/bar/bat is taken to match /foo/bar/*.

For more information about path limitations and pattern matching, see the URL Maps documentation.

The manifest for the my-products Service might look like this:

apiVersion: v1
kind: Service
metadata:
  name: my-products
spec:
  type: NodePort
  selector:
    app: products
    department: sales
  ports:
  - protocol: TCP
    port: 60000
    targetPort: 50000

In the Service manifest, you must use type: NodePort unless you're using container native load balancing. If using container native load balancing, use the type: ClusterIP.

In the Service manifest, the selector field says any Pod that has both the app: products label and the department: sales label is a member of this Service.

When a request comes to the Service on port 60000, it is routed to one of the member Pods on TCP port 50000.

Each member Pod must have a container listening on TCP port 50000.

The manifest for the my-discounted-products Service might look like this:

apiVersion: v1
kind: Service
metadata:
  name: my-discounted-products
spec:
  type: NodePort
  selector:
    app: discounted-products
    department: sales
  ports:
  - protocol: TCP
    port: 80
    targetPort: 8080

In the Service manifest, the selector field says any Pod that has both the app: discounted-products label and the department: sales label is a member of this Service.

When a request comes to the Service on port 80, it is routed to one of the member Pods on TCP port 8080.

Each member Pod must have a container listening on TCP port 8080.

Default backend

You can specify a default backend by providing a backend field in your Ingress manifest. Any requests that don't match the paths in the rules field are sent to the Service and port specified in the backend field. For example, in the following Ingress, any requests that don't match / or /discounted are sent to a Service named my-products on port 60001.

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: my-ingress
spec:
  backend:
    serviceName: my-products
    servicePort: 60001
  rules:
  - http:
      paths:
      - path: /
        backend:
          serviceName: my-products
          servicePort: 60000
      - path: /discounted
        backend:
          serviceName: my-discounted-products
          servicePort: 80

If you don't specify a default backend, GKE provides a default backend that returns 404.

Ingress to Compute Engine resource mappings

The GKE Ingress controller deploys and manages Compute Engine load balancer resources based on the Ingress resources that are deployed in the cluster. The mapping of Compute Engine resources depends on the structure of the Ingress resource. Awareness of these resources mappings helps you with planning, designing, and troubleshooting.

The my-ingress manifest shown in the Multiple backend services section specifies an external Ingress resource with two URL path matches that reference two different Kubernetes Services. Here are some of the Compute Engine resources created on behalf of my-ingress:

• A forwarding rule and IP address.
• Compute Engine firewall rules that permit traffic for load balancer health checks and application traffic from Google Front Ends or Envoy proxies.
• A target HTTP proxy and a target HTTPS proxy, if you configured TLS.
• A URL map which with a single host rule referencing a single path matcher. The path matcher has two path rules, one for /* and another for /discounted. Each path rule maps to a unique backend service.
• NEGs which hold a list of Pod IPs from each Service as endpoints. These are created as a result of the my-discounted-products and my-products Services. The following diagram provides an overview of the Ingress to Compute Engine resource mappings.

Options for providing SSL certificates

There are three ways to provide SSL certificates to an HTTP(S) load balancer:

Google-managed certificates

Google-managed SSL certificates are provisioned, deployed, renewed, and managed for your domains. Managed certificates do not support wildcard domains.

Self-managed certificates shared with Google Cloud

You can provision your own SSL certificate and create a certificate resource in your Google Cloud project. You can then list the certificate resource in an annotation on an Ingress to create an HTTP(S) load balancer that uses the certificate. Refer to instructions for pre-shared certificates for more information.

Self-managed certificates as Secret resources

You can provision your own SSL certificate and create a Secret to hold it. You can then refer to the Secret in an Ingress specification to create an HTTP(S) load balancer that uses the certificate. Refer to the instructions for using certificates in Secrets for more information.

Health checks

When you expose one or more Services through an Ingress using the default Ingress controller, GKE creates a Google Cloud external HTTP(S) load balancer or a Google Cloud internal HTTP(S) load balancer. Both of these load balancers support multiple backend services on a single URL map. Each of the backend services corresponds to a Kubernetes Service, and each backend service must reference a Google Cloud health check. This health check is different from a Kubernetes liveness or readiness probe because the health check is implemented outside of the cluster.

Note: Load balancer health checks are specified per backend service. While it's possible to use the same health check for all backend services of the load balancer, the health check reference isn't specified for the whole load balancer (at the Ingress object itself).

GKE uses the following procedure to create a health check for each backend service corresponding to a Kubernetes Service:

• If the Service references a BackendConfig CRD with healthCheck information, GKE uses that to create the health check. Both the Anthos Ingress controller and the GKE Ingress controller support creating health checks this way.

• If the Service does not reference a BackendConfig CRD:

  • GKE can infer some or all of the parameters for a health check if the Serving Pods use a Pod template with a container whose readiness probe has attributes that can be interpreted as health check parameters. See Parameters from a readiness probe for implementation details and Default and inferred parameters for a list of attributes that can be used to create health check parameters. Only the GKE Ingress controller supports inferring parameters from a readiness probe.

  • If the Pod template for the Service's serving Pods does not have a container with a readiness probe whose attributes can be interpreted as health check parameters, the default values are used to create the health check. Both the Anthos Ingress controller and the GKE Ingress controller can create a health check using only the default values.

Default and inferred parameters

The following parameters are used when you do not specify health check parameters for the corresponding Service using a BackendConfig CRD.

Health check parameter	Default value	Inferable value from readiness probe
Protocol	HTTP	if present in the serving Pod's spec: containers[].readinessProbe.httpGet.scheme
Request path	/	if present in the serving Pod's spec: containers[].readinessProbe.httpGet.path
Request Host header	Host: backend-ip-address	if present in the serving Pod's spec: containers[].readinessProbe.httpGet.httpHeaders
Expected response	HTTP 200 (OK)	HTTP 200 (OK) cannot be changed
Check interval	for zonal NEGs: 15 seconds for instance groups: 60 seconds	if present in the serving Pod's spec: for zonal NEGs: containers[].readinessProbe.periodSeconds for instance groups: containers[].readinessProbe.periodSeconds + 60 seconds
Check timeout	5 seconds	if present in the serving Pod's spec: containers[].readinessProbe.timeoutSeconds
Healthy threshold	1	1 cannot be changed
Unhealthy threshold	for zonal NEGs: 2 for instance groups: 10	same as default: for zonal NEGs: 2 for instance groups: 10
Port specified by number	for zonal NEGs: the Service's port for instance groups: the Service's nodePort	if the Ingress object's backend.servicePort references a Service's port, that port is used, provided the following are also defined: The serving Pod's readiness probe must specify a port: spec.containers[].readinessProbe.httpGet.port the Service's targetPort references the serving Pod's containers[].spec.ports.containerPort
Destination IP address	for zonal NEGs: the Pod's IP address for instance groups: the node's IP address	same as default: for zonal NEGs: the Pod's IP address for instance groups: the node's IP address

Note: A valid Kubernetes readiness probe supports setting multiple HTTP headers in readinessProbe.httpGet. If readinessProbe.httpGet.httpHeaders specifies more than just the Host header, the load balancer's health check parameters are set to default values instead of values inferred from the readiness probe. This limitation exists because health checks only support setting the Host header.

Parameters from a readiness probe

When GKE creates the health check for the Service's backend service, GKE can copy certain parameters from one container's readiness probe used by that Service's serving Pods. This option is only supported by the GKE Ingress controller.

Supported readiness probe attributes that can be interpreted as health check parameters are listed along with the default values in Default and inferred parameters. Default values are used for any attributes not specified in the readiness probe or if you don't specify a readiness probe at all.

If serving Pods for your Service contain multiple containers, or if you're using the Anthos Ingress controller, you should use a BackendConfig CRD to define health check parameters. For more information, see When to use a BackendConfig CRD instead.

Warning: When a backend service's health check parameters are inferred from a serving Pod's readiness probe, GKE does not keep the readiness probe and health check synchronized:

• If you change the readiness probe for the serving Pods of a Service referenced by an Ingress after GKE has created the external HTTP(S) load balancer or internal HTTP(S) load balancer for that Ingress, the changes you make to the readiness probe will not be copied to the health check for the corresponding backend service on the load balancer.
• Conversely, if you make changes to the health check used by a backend service of a load balancer created on behalf of an Ingress, GKE will not update any readiness probe, nor will it revert any changes you make to the health check.

When to use BackendConfig CRDs instead

Instead of relying on parameters from Pod readiness probes, you should explicitly define health check parameters for a backend service by creating a BackendConfig CRD for the Service in these situations:

• If you're using Anthos: The Anthos Ingress controller does not support obtaining health check parameters from the readiness probes of serving Pods. It can only create health checks using implicit parameters or as defined in a BackendConfig CRD.

• If serving Pods have multiple containers with unique readiness probes: If the serving Pods for a Service have more than one container, and each container has different readiness probe settings, you should define the health check for the corresponding backend service by referencing a BackendConfig CRD on the corresponding Service. GKE does not allow you to choose a specific readiness probe from which it infers health check parameters if multiple readiness probes are present in a serving Pod.

• If you need control over the port used for the load balancer's health checks: GKE only uses the readiness probe's containers[].readinessProbe.httpGet.port for the backend service's health check when that port matches the service port for the Service referenced in the Ingress spec.rules[].http.paths[].backend.servicePort.

Parameters from a BackendConfig CRD

You can specify the backend service's health check parameters using the healthCheck parameter of a BackendConfig CRD referenced by the corresponding Service. This gives you more flexibility and control over health checks for a Google Cloud external HTTP(S) load balancer or internal HTTP(S) load balancer created by an Ingress. See Ingress features for GKE version compatibility.

This example BackendConfig CRD defines the health check protocol (type), a request path, a port, and a check interval in its spec.healthCheck attribute:

apiVersion: cloud.google.com/v1
kind: BackendConfig
metadata:
  name: http-hc-config
spec:
  healthCheck:
    checkIntervalSec: 15
    port: 15020
    type: HTTPS
    requestPath: /healthz

Using multiple TLS certificates

Suppose you want an HTTP(S) load balancer to serve content from two hostnames: your-store.example and your-experimental-store.example. Also, you want the load balancer to use one certificate for your-store.example and a different certificate for your-experimental-store.example.

You can do this by specifying multiple certificates in an Ingress manifest. The load balancer chooses a certificate if the Common Name (CN) in the certificate matches the hostname used in the request. For detailed information on how to configure multiple certificates, see Using multiple SSL certificates in HTTPS Load Balancing with Ingress.

Kubernetes Service compared to Google Cloud backend service

A Kubernetes Service and a Google Cloud backend service are different things. There is a strong relationship between the two, but the relationship is not necessarily one to one. The GKE ingress controller creates a Google Cloud backend service for each (serviceName, servicePort) pair in an Ingress manifest. So it is possible for one Kubernetes Service object to be related to several Google Cloud backend services.

Limitations

• In clusters using versions earlier than 1.16, the total length of the namespace and name of an Ingress must not exceed 40 characters. Failure to follow this guideline may cause the GKE Ingress controller to act abnormally. For more information, see this issue on GitHub.

• In clusters using NEGs, ingress reconciliation time may be affected by the number of ingresses. For example, a cluster with 20 ingresses, each containing 20 distinct NEG backends, may result in a latency of more than 30 minutes for an ingress change to be reconciled. This especially impacts regional clusters due to the increased number of NEGs needed.

• Quotas for URL maps apply.

• Quotas for Compute Engine resources apply.

• If you're not using NEGs with the GKE ingress controller then GKE clusters have a limit of 1000 nodes. When services are deployed with NEGs, there is no GKE node limit. Any non-NEG Services exposed through Ingress do not function correctly on clusters above 1000 nodes.

• For the GKE Ingress controller to use your readinessProbes as health checks, the Pods for an Ingress must exist at the time of Ingress creation. If your replicas are scaled to 0, the default health check applies. For more information, see this issue comment.

• Changes to a Pod's readinessProbe do not affect the Ingress after it is created.

• An external HTTP(S) load balancer terminates TLS in locations that are distributed globally, to minimize latency between clients and the load balancer. If you require geographic control over where TLS is terminated, you should use a custom ingress controller exposed through a GKE Service of type LoadBalancer instead, and terminate TLS on backends that are located in regions appropriate to your needs.

• Combining multiple Ingress resources into a single Google Cloud load balancer is not supported.

Implementation details

• The Ingress controller performs periodic checks of service account permissions by fetching a test resource from your Google Cloud project. You will see this as a GET of the (non-existent) global BackendService with the name k8s-ingress-svc-acct-permission-check-probe. As this resource should not normally exist, the GET request will return "not found". This is expected; the controller is checking that the API call is not rejected due to authorization issues. If you create a BackendService with the same name, then the GET will succeed instead of returning "not found".

What's next

• Learn more about load balancing in Google Cloud.
• Read an overview of networking in GKE.
• Learn how to configure Ingress for internal HTTP(S) load balancer.
• Learn how to configure Ingress for external HTTP(S) load balancer.