Google Cloud Platform (GCP) provides health checking mechanisms that determine if backends – such as instance groups and network endpoint groups (NEGs) – properly respond to traffic. This document discusses health checking concepts specific to GCP and its load balancers.
GCP provides global and regional health check systems that connect to backends on a configurable, periodic basis. Each connection attempt is called a probe. GCP records the success or failure of each probe.
Health checks and load balancers work together. Based on a configurable number of sequential successful or failed probes, GCP computes an overall health state for each backend in the load balancer. Backends that respond successfully for the configured number of times are considered healthy. Backends that fail to respond successfully for a separate number of times are unhealthy.
GCP uses the overall health state of each backend to determine its eligibility for receiving new requests. In addition to being able to configure probe frequency and health state thresholds, you can configure the criteria that define a successful probe. This document describes how health checks work in detail.
GCP uses special routes not defined in your VPC network for health checks. For complete information on this, read Load balancer return paths.
Health check categories, protocols, and ports
GCP organizes health checks by category and protocol.
There are two health check categories: health checks and legacy health checks. Each category supports a different set of protocols and a means for specifying the port used for health checking. The protocol and port determine how GCP health check systems contact your backends. For example, you can create a health check that uses the HTTP protocol on TCP port 80, or you can create a health check that uses the TCP protocol for a named port configured on an instance group.
Most GCP load balancers require non-legacy health checks, but Network Load Balancing requires legacy health checks that use the HTTP protocol. Refer to Selecting a health check for specific guidance on selecting the category and the protocol, and specifying the ports.
You cannot convert a legacy health check to a health check or vice versa.
The term health check does not refer to legacy health checks. Legacy health checks are explicitly called legacy health checks in this document.
Selecting a health check
Health checks must be compatible with the type of load balancer and the types of backends (instance groups or network endpoint groups) it uses. The three factors you must specify when you create a health check are:
- Category: health check or legacy health check, which must be compatible with the load balancer
- Protocol: defines what protocol the GCP systems use to periodically probe your backends
- Port specification: defines which ports are used for the health check's protocol
The guide at the end of this section summarizes valid combinations of health check category, protocol, and port specification based on a given type of load balancer and backend type.
As used in this section, the term instance group refers to unmanaged instance groups, managed zonal instance groups, or managed regional instance groups.
Category and protocol
The type of load balancer and the types of backends that the load balancer uses determine the health check's category. Network Load Balancing requires legacy health checks that use the HTTP protocol. For all other load balancer types, use regular health checks.
You must select a protocol from the list of protocols supported by the health check's category. It's a best practice to use the same protocol as the load balancer itself; however, this is not a requirement, nor is it always possible. For example, network load balancers require legacy health checks, and they require that the legacy health checks use the HTTP protocol, despite the fact that Network Load Balancing supports TCP and UDP in general. For network load balancers, you must run an HTTP server on your VMs so that they can respond to health check probes.
The following table lists the health check categories and the protocols each category supports.
|Health check category||Supported protocols|
|Health check||• HTTP
|Legacy health check||• HTTP|
• HTTPS (Legacy HTTPS health checks are not supported for network load balancers and cannot be used with most other types of load balancers.)
Category and port specification
In addition to a protocol, you must select a port specification for your health check. Health checks provide three port specification methods, and legacy health checks provide one method. Not all port specification methods are applicable to each type of load balancer. The type of load balancer and the types of backends it uses determine which port specification method you can use.
|Health check category||Port specification methods and meanings|
|Legacy health check||
Note here and below: The flag
--use-serving-port is implemented with
gcloud beta compute health-checks create, but not with
gcloud beta compute health-checks update.
Load balancer guide
Use this table to choose the correct category and protocol of health check for a given load balancer.
|Load balancer||Backend type||Health Check Category and Scope||Port specification|
|Internal TCP/UDP||Instance Groups on a regional internal backend service||Health check (global)||Port number (
You cannot use the
|Internal HTTP(S)||Network Endpoint Groups
on a backend service
|Health check (regional)||Port number (
|Instance Groups on a backend service||Health check (regional)||Port number (
|Network||Instance Groups using target pools||Legacy health check (global)
using the HTTP protocol
|Legacy health checks only support port specification by port number
|Network Endpoint Groups
on a backend service
|Health check (global)||Port number (
|Instance Groups on a backend service||Health check (global)||Port number (
- The backends used by the backend service are instance groups, not network endpoint groups.
- The backend VMs can be probed using either
How health checks work
When you create a health check or create a legacy health check, you specify the following flags or accept their default values. These flags control how frequently each GCP health check system probes your instance group or NEG backends. GCP implements probes using multiple systems.
A health check's settings cannot be configured on a per-backend basis. Health checks are associated with a whole backend service, and legacy health checks are associated with a whole target pool or backend service, for certain HTTP(S) load balancers. Thus, the parameters for the probe are the same for all backends referenced by a given backend service or target pool.
|Configuration flag||Purpose||Default value|
||The check interval is the amount of time from the start of one probe issued by one probing system to the start of the next probe issued by the same system. Units are seconds.||If omitted, GCP uses
||The timeout is the amount of time that GCP will wait for a response to a probe. Its value must be less than or equal to the check interval. Units are seconds.||If omitted, GCP uses
Probe IP ranges
The source IP addresses for GCP probe systems depend on the type of load balancer. Use the following table to create ingress allow firewall rules that allow traffic from GCP probe systems to your backends.
|Load Balancer||Probe IP ranges||Firewall rule example|
• TCP Proxy
• SSL Proxy
• Internal HTTP(S)
|Firewall rules for all load balancers except network load balancers|
|Firewall rules for network load balancers|
Multiple probes and frequency
GCP sends health check probes from multiple redundant systems from the appropriate source IP ranges. No single probe system is responsible for all of the probes. Multiple systems issue probes simultaneously so that failure of one does not cause GCP to lose track of backend health states.
The interval and timeout settings you configure for a health check
are applied to each probe system. For a given backend,
software access logs and
tcpdump show more frequent health check probes than
your configured settings. Multiple probe systems simultaneously contacting your
backends result in more health check probes than the configuration for a
single probe system.
This is expected behavior, and you cannot configure the number of probe systems that GCP uses for health checks. However, you can estimate the effect of multiple simultaneous probes by considering the following factors:
To estimate the probe frequency per backend service, consider the following:
Base frequency per backend service: Each health check has an associated check frequency, inversely proportional to the configured check interval:
When you associate a health check with a backend service, you establish a base frequency used by each probe system for backends on that backend service.
Probe scale factor: The backend service's base frequency is multiplied by the number of simultaneous probe systems that GCP uses. This number can vary, but is generally between 5 and 10.
Multiple forwarding rules for internal TCP/UDP load balancers:: If you have configured multiple internal forwarding rules (each having a different IP address) pointing to the same regional internal backend service, GCP uses multiple probe systems to check each IP address. The probe frequency per backend service is multiplied by the number of configured forwarding rules.
Multiple forwarding rules for network load balancers:: If you have configured multiple forwarding rules that point to the same target pool, GCP uses multiple probe systems to check each IP address. The probe frequency as seen by each backend in the target pool is multiplied by the number of configured forwarding rules.
Multiple target proxies for HTTP(S) load balancers: If you have configured multiple target proxies for the same URL map for HTTP(S) Load Balancing, GCP uses multiple probe systems to check the IP address associated with each target proxy. The probe frequency per backend service is multiplied by the number of configured target proxies.
Multiple target proxies for SSL Proxy and TCP Proxy load balancers: If you have configured multiple target proxies for the same backend service for SSL Proxy or TCP Proxy Load Balancing, GCP uses multiple probe systems to check the IP address associated with each target proxy. The probe frequency per backend service is multiplied by the number of configured target proxies.
Sum over backend services: If a backend (such as an instance group) is used by multiple backend services, the backend instances are contacted as frequently as the sum of frequencies for each backend service's health check.
With network endpoint group backends (NEGs), it's more difficult to determine the exact number of health check probes. For example, the same endpoint can be in multiple NEGs, where those NEGs don't necessarily have the same set of endpoints, and different endpoints can point to the same backend.
Destination for health check packets
GCP health check probes send packets only to the primary network interface of each backend instance. The destination IP address of these packets depends on the type of load balancer:
- For internal TCP/UDP load balancers and network load balancers, the destination of health check packets is the IP address of the load balancer's forwarding rule. If multiple forwarding rules point to the same backend service or target pool, GCP sends probes to each forwarding rule's IP address. This can result in an increase in the number of probes, as described in the previous section.
- For HTTP(S), TCP Proxy, SSL Proxy, and internal HTTP(S) load balancers that use instance groups as backends, the destination of health check packets is the primary internal IP address associated with the primary network interface of each backend instance.
- For HTTP(S), TCP Proxy, SSL Proxy, and internal HTTP(S) load balancers that use network endpoint groups as backends, the destination of health check packets is the IP address of the endpoint, which can be either a primary or secondary (alias IP) address.
Content-based health checks
HTTP(S), HTTP/2, TCP, and SSL health checks can optionally be content-based (request/response). In a content-based health check, the health check prober sends a request string to the backend. The health check is configured to expect a particular response to the probe.
Success criteria for HTTP, HTTPS, and HTTP/2
Responses from probes for health checks using HTTP, HTTPS, or HTTP/2 protocols
are successful only if GCP receives an
HTTP 200 (OK) response
to the request it sends and that response is delivered before the probe
Requests are sent to a configurable request path, or
/, if unspecified. Any
response is accepted, unless you use content-based checking to provide an
expected response string. The flags available to control success criteria for
HTTP, HTTPS, and HTTP/2 health checks are:
||Specify the URL path to which GCP sends health check
If omitted, GCP sends probe requests to the root path,
||The optional response flag allows you to configure a content-based
health check. The expected response string must be less than or equal
to 1,024 ASCII (single byte) characters. When configured,
GCP expects this string within the first 1,024 bytes of the
response in addition to receiving
Use of a content-based health check is optional. Whether or not you specify
an expected response string, GCP expects the backend being
checked to respond with
HTTP 200 (OK). When you supply an expected response,
each GCP health check prober searches the response provided by
your backends, looking for the expected response string within the first 1,024 bytes
returned. A content-based HTTP health check is considered successful if both
HTTP 200 (OK) is received and the expected response is found in the first
1,024 bytes of the returned response.
Success criteria for SSL and TCP
By default, responses from probes for health checks using SSL and TCP protocols are successful only if GCP is able to successfully complete a SSL or TCP handshake to establish a session before the probe timeout.
Optionally, in addition to completing a handshake, you can provide a request string and an expected response string, each up to 1,024 ASCII (single byte) characters in length. When an expected response string is configured, GCP considers a probe successful only if the SSL or TCP handshake completes and the response string returned exactly matches the expected response string. The following combinations of request and response flags are available for health checks using the SSL and TCP protocols:
|Neither request nor response specified
Neither flag specified:
|GCP considers the probe successful if the TCP or SSL session is established before the probe timeout.|
|Both request and response specified
Both flags specified:
|GCP sends your configured request string and waits for the expected response string. The probe is successful if the TCP or SSL session is established and the response string returned exactly matches the expected response string before the probe timeout.|
|Only response specified
Flags specified: only
|GCP waits for the expected response string. The probe
is successful if the TCP or SSL session is established and the response
string returned exactly matches the expected response string
before the probe timeout.
You should only use
|Only request specified
Flags specified: only
|GCP sends your configured request string. The probe is successful if a TCP or SSL session is established before the probe timeout. The response, if any, is not checked.|
GCP uses the following configuration flags and whether or not probes were successful to determine the overall health state of each backend being load balanced:
|Configuration flag||Purpose||Default value|
||The healthy threshold specifies the number of sequential successful probe results for a backend to be considered healthy.||If omitted, GCP uses a threshold of
||The unhealthy threshold specifies the number of sequential failed probe results for a backend to be considered unhealthy.||If omitted, GCP uses a threshold of
GCP considers backends to be healthy once this healthy threshold has been met. Healthy backends are eligible to receive new connections.
GCP considers backends to be unhealthy when the unhealthy threshold has been met. Unhealthy backends are not eligible to receive new connections; however, existing connections are not immediately terminated. Instead, the connection remains open until a timeout occurs or until traffic is dropped. The specific behavior differs depending on the type of load balancer that you're using.
Existing connections might fail to return responses, depending on the cause for failing the probe. An unhealthy backend can become healthy if it is able to meet the healthy threshold again.
For information on configuring health checks, see Creating Health Checks.