An internal passthrough Network Load Balancer is a regional load balancer that is built on the Andromeda network virtualization stack.
Internal passthrough Network Load Balancers distribute traffic among internal virtual machine (VM) instances in the same region in a Virtual Private Cloud (VPC) network. It enables you to run and scale your services behind an internal IP address that is accessible only to systems in the same VPC network or systems connected to your VPC network.
Use an internal passthrough Network Load Balancer in the following circumstances:
- You need a high-performance, pass-through Layer 4 load balancer for TCP, UDP, ICMP, ICMPv6, SCTP, ESP, AH, and GRE protocols.
- If serving traffic through TLS (SSL), it is acceptable to have SSL traffic terminated by your backends instead of by the load balancer. The internal passthrough Network Load Balancer cannot terminate SSL traffic.
- You need to forward the original packets unproxied. For example, if you need the client source IP address to be preserved.
- You have an existing setup that uses a pass-through load balancer, and you want to migrate it without changes.
The internal passthrough Network Load Balancers address many use cases. For a few high-level examples, see Passthrough Network Load Balancer overview.
How internal passthrough Network Load Balancers work
An internal passthrough Network Load Balancer has a frontend (the forwarding rule) and a backend (the
backend service). You can use either instance groups or GCE_VM_IP
zonal NEGs
as backends on the backend service. This example shows instance group backends.
Unlike a proxy load balancer, an internal passthrough Network Load Balancer doesn't terminate connections from clients and then open new connections to backends. Instead, an internal passthrough Network Load Balancer routes connections directly from clients to the healthy backends, without any interruption. Responses from the healthy backend VMs go directly to the clients, not back through the load balancer. TCP responses use direct server return. For more information, see TCP and UDP request and return packets.
The load balancer monitors backend health by using health check probes. For more information, see the Health check section.
The Google Cloud Linux guest
environment, Windows guest
environment, or
an equivalent process configures each backend VM with the IP address of the load
balancer. For VMs created from Google Cloud images, the Guest
agent (formerly, the Windows Guest
Environment or Linux Guest Environment) installs the local route for the load
balancer's IP address. Google Kubernetes Engine instances based on Container-
Optimized OS implement this by using iptables
instead.
Google Cloud virtual networking manages traffic delivery and scaling as appropriate.
Protocols, scheme, and scope
Each internal passthrough Network Load Balancer supports the following:
- One backend service with load balancing scheme
INTERNAL
and a supported protocol. For more information, see backend service. - Backend VMs specified as either one of the following:
- Managed and unmanaged instance groups that are located in one region and VPC network.
- Backend zonal network endpoint groups (NEGs) with
GCE_VM_IP
type endpoints that are located in the same region and VPC network. Endpoints in the NEG must be primary internal IP addresses in the same subnet and zone used by the NEG.
- Support for IPv4 and IPv6 traffic when using instance group backends.
Zonal network endpoint groups (NEGs) with
GCE_VM_IP
endpoints only support IPv4 traffic. - One or more forwarding rules, each using either the
TCP
,UDP
, orL3_DEFAULT
protocol matching the backend service's protocol. - Each forwarding rule with its own unique IP address or multiple forwarding rules that share a common IP address.
- Each forwarding rule with up to five ports or all ports.
- If global access is enabled, clients in any region.
- If global access is disabled, clients in the same region as the load balancer.
An internal passthrough Network Load Balancer doesn't support:
- Backend VMs in multiple regions.
- Balancing traffic that originates from the internet, unless you're using it with an external load balancer.
- IPv6 packets with fragmented headers.
Client access
By default, the load balancer only supports clients that are in the same region as the load balancer. Clients can be in the same network as the load balancer or in a VPC network connected by using VPC Network Peering. You can enable global access to allow clients from any region to access your internal passthrough Network Load Balancer.
The following table summarizes client access.
Global access disabled | Global access enabled |
---|---|
Clients must be in the same region as the load balancer. They also must be in the same VPC network as the load balancer or in a VPC network that is connected to the load balancer's VPC network by using VPC Network Peering. | Clients can be in any region. They still must be in the same VPC network as the load balancer or in a VPC network that's connected to the load balancer's VPC network by using VPC Network Peering. |
On-premises clients can access the load balancer through Cloud VPN tunnels or VLAN attachments. These tunnels or attachments must be in the same region as the load balancer. | On-premises clients can access the load balancer through Cloud VPN tunnels or VLAN attachments. These tunnels or attachments can be in any region. |
IP addresses for request and return packets
When a backend VM receives a load-balanced packet from a client, the packet's source and destination are:
- Source: the client's internal IPv4, IPv6, or the IPv4 address from one of the client's alias IPv4 ranges.
- Destination: the IP address of the load balancer's forwarding rule. The forwarding rule uses either a single internal IPv4 address or an internal IPv6 range.
Because the load balancer is a pass-through load balancer (not a proxy), packets arrive bearing the destination IP address of the load balancer's forwarding rule. Software running on backend VMs should be configured to do the following:
- Listen on (bind to) the load balancer's forwarding rule IP address or any IP
address (
0.0.0.0
or::
) - If the load balancer forwarding rule's protocol supports ports: Listen on (bind to) a port that's included in the load balancer's forwarding rule
Return packets are sent directly from the load balancer's backend VMs to the client. The return packet's source and destination IP addresses depend on the protocol:
- TCP is connection-oriented so backend VMs must reply with packets whose source IP addresses match the forwarding rule's IP address so that the client can associate the response packets with the appropriate TCP connection.
- UDP is connectionless so backend VMs can send response packets whose source IP addresses either match the forwarding rule's IP address or match any assigned IP address for the VM. Practically speaking, most clients expect the response to come from the same IP address to which they sent packets.
The following table summarizes sources and destinations for response packets:
Traffic type | Source | Destination |
---|---|---|
TCP | The IP address of the load balancer's forwarding rule | The requesting packet's source |
UDP | For most use cases, the IP address of the load balancer's forwarding rule † | The requesting packet's source |
† It is possible to set the response packet's source to the VM NIC's primary internal IPv4 address or an alias IP address range. If the VM has IP forwarding enabled, arbitrary IP address sources can also be used. Not using the forwarding rule's IP address as a source is an advanced scenario because the client receives a response packet from an internal IP address that does not match the IP address to which it sent a request packet.
Architecture
An internal passthrough Network Load Balancer with multiple backends distributes connections among all of those backends. For information about the distribution method and its configuration options, see traffic distribution.
You can use either instance groups or zonal NEGs, but not a combination of both, as backends for an internal passthrough Network Load Balancer:
- If you choose instance groups, you can use unmanaged instance groups, zonal managed instance groups, regional managed instance groups, or a combination of instance group types.
- If you choose zonal NEGs, you must use
GCE_VM_IP
zonal NEGs.
High availability describes how to design an internal load balancer that is not dependent on a single zone.
Instances that participate as backend VMs for internal passthrough Network Load Balancers must be
running the appropriate Linux or Windows guest environment or other processes
that provide equivalent functionality. This guest environment must be able to
contact the metadata server (metadata.google.internal
, 169.254.169.254
) to
read instance metadata so that it can generate local routes to accept traffic
sent to the load balancer's internal IP address.
This diagram illustrates traffic
distribution among VMs located in two separate instance groups. Traffic sent
from the client instance to the IP address of the load balancer (10.10.10.9
)
is distributed among backend VMs in either instance group. Responses sent from
any of the serving backend VMs are delivered directly to the client VM.
You can use an internal passthrough Network Load Balancer with either a custom mode or auto mode VPC network. You can also create internal passthrough Network Load Balancers with an existing legacy network.
An internal passthrough Network Load Balancer consists of the following Google Cloud components.
Component | Purpose | Requirements |
---|---|---|
Internal IP address | This is the address for the load balancer. | The internal IP address must be in the same subnet as the internal forwarding rule. The subnet must be in the same region and VPC network as the backend service. |
Internal forwarding rule | An internal forwarding rule in combination with the internal IP address is the frontend of the load balancer. It defines the protocol and port(s) that the load balancer accepts, and it directs traffic to a regional internal backend service. | Forwarding rules for internal passthrough Network Load Balancers must do the following: • Have a load-balancing-scheme of INTERNAL .• Use an ip-protocol of either TCP or
UDP , matching the protocol of the backend
service.• Reference a subnet in the same VPC network
and region as the backend service. |
Regional internal backend service | The regional internal backend service defines the protocol used to
communicate with the backends, and it specifies a health
check. Backends can be unmanaged instance groups, managed zonal instance
groups, managed regional instance groups, or zonal NEGs with
GCE_VM_IP endpoints. |
The backend service must do the following: • Have a load-balancing-scheme of INTERNAL .• Use a protocol of either TCP or
UDP , matching the ip-protocol of the forwarding
rule.• Have an associated health check. • Have an associated region. The forwarding rule and all backends must be in the same region as the backend service • Be associated with a single VPC network. When not specified, the network is inferred based on the network used by each backend VM's default network interface ( nic0 ).Although the backend service is not tied to a specific subnet, the forwarding rule's subnet must be in the same VPC network as the backend service. |
Health check | Every backend service must have an associated health check. The
health check defines the parameters under which Google Cloud
considers the backends that it manages to be eligible to receive
traffic. Only healthy backend VMs receive traffic sent from client
VMs to the IP address of the load balancer. |
Even though the forwarding rule and backend service can use either
TCP or UDP , Google Cloud does not have
a health check for UDP traffic. For more information, see
health checks and UDP traffic.
|
An internal passthrough Network Load Balancer doesn't support:
- Backend VMs in multiple regions.
- Balancing traffic that originates from the internet, unless you're using it with an external load balancer.
- IPv6 packets with fragmented headers.
Internal IP address
Internal passthrough Network Load Balancers support IPv4 only (single-stack) subnets and IPv4 and IPv6 (dual-stack) subnets. For more information, see Types of subnet.
An internal passthrough Network Load Balancer requires at least one forwarding rule. The forwarding rule references the internal IP address:
- For IPv4 traffic, the forwarding rule references an IPv4 address from the primary IPv4 subnet range.
- For IPv6 traffic, the forwarding rule references a
/96
range of internal IPv6 addresses from the subnet's/64
internal IPv6 address range. The subnet must be a dual-stack subnet with an internal IPv6 address range (ipv6-access-type
set toINTERNAL
).
Firewall configuration
Internal passthrough Network Load Balancers require the following configuration for hierarchical firewall policies and VPC firewall rules:
- Allow ingress from IPv4 or IPv6 health check source ranges.
- Allow ingress from client IPv4 or IPv6 address source ranges.
For more information, see Configuring firewall rules.
Forwarding rules
A forwarding rule specifies the protocol and ports on which the load balancer accepts traffic. Because internal passthrough Network Load Balancers are not proxies, they pass traffic to backends on the same protocol and port.
An internal passthrough Network Load Balancer requires at least one internal forwarding rule. You can define multiple forwarding rules for the same load balancer.
If you want the load balancer to handle both IPv4 and IPv6 traffic, create two forwarding rules: one rule for IPv4 traffic that points to IPv4 (or dual-stack) backends, and one rule for IPv6 traffic that points only to dual-stack backends. It's possible to have an IPv4 and an IPv6 forwarding rule reference the same backend service, but the backend service must reference dual-stack backends.
The forwarding rule must reference a specific subnet in the same VPC network and region as the load balancer's backend components. This requirement has the following implications:
- The subnet that you specify for the forwarding rule doesn't need to be the same as any of the subnets used by backend VMs; however, the subnet must be in the same region as the forwarding rule.
- For IPv4 traffic, the internal forwarding rule references regional internal IPv4 address from the primary IPv4 address range of the subnet that you select. This IPv4 address can be selected automatically or you can use a static (reserved) IPv4 address.
- For IPv6 traffic, the forwarding rule references a
/96
range of IPv6 addresses from the subnet's internal IPv6 address range. The subnet must be a dual-stack subnet with theipv6-access-type
set toINTERNAL
. The/96
IPv6 address range is either assigned automatically or you can use a reserved internal IP address.
Forwarding rule protocols
Internal passthrough Network Load Balancers support the following IPv4 protocol options for
each forwarding rule: TCP
, UDP
, or L3_DEFAULT
.
Internal passthrough Network Load Balancers support the following IPv6 protocol options for
each forwarding rule: TCP
or UDP
.
The L3_DEFAULT
option enables you
to load balance
TCP, UDP, ICMP, ICMPv6, SCTP, ESP, AH, and GRE
protocols.
In addition to supporting protocols other than TCP and UDP, the
L3_DEFAULT
option makes
it possible for a single forwarding rule to simultaneously forward traffic for
multiple protocols. For example, in addition to making HTTP requests, you can
also ping the load balancer IP address.
Forwarding rules that use the TCP
or UDP
protocols can reference a backend
service by using either the same protocol as the forwarding rule or a backend
service using UNSPECIFIED
protocol.
If you're using the L3_DEFAULT
protocol, you must configure the forwarding
rule to accept traffic on all ports. To configure all ports, either set
--ports=ALL
by using
the Google Cloud CLI, or set allPorts
to
True
by using the API.
The following table summarizes how to use these settings for different protocols:
Traffic to be load-balanced | Forwarding rule protocol | Backend service protocol |
---|---|---|
TCP (IPv4 or IPv6) | TCP |
TCP or UNSPECIFIED |
UDP (IPv4 or IPv6) | UDP |
UDP or UNSPECIFIED |
TCP, UDP, ICMP, ICMPv6, SCTP, ESP, AH, and GRE | L3_DEFAULT |
UNSPECIFIED |
Forwarding rules and global access
An internal passthrough Network Load Balancer's forwarding rules are regional, even when global access
is enabled. After you enable global access, the regional internal forwarding
rule's allowGlobalAccess
flag is set to true
.
Forwarding rules and port specifications
When you create an internal forwarding rule, you must choose one of the following port specifications:
- Specify at least one and up to five ports, by number.
- Specify
ALL
to forward traffic on all ports.
An internal forwarding rule that supports either all TCP ports or all UDP ports allows backend VMs to run multiple applications, each on its own port. Traffic sent to a given port is delivered to the corresponding application, and all applications use the same IP address.
When you need to forward traffic on more than five specific ports, combine
firewall rules with forwarding rules. When you create the
forwarding rule, specify all ports, and then create ingress allow
firewall
rules that only permit traffic to the desired ports. Apply the firewall rules to
the backend VMs.
You cannot modify a forwarding rule after you create it. If you need to change the specified ports or the internal IP address for an internal forwarding rule, you must delete and recreate it.
Multiple forwarding rules for a single backend service
You can configure multiple internal forwarding rules that all reference the same internal backend service. An internal passthrough Network Load Balancer requires at least one internal forwarding rule.
Configuring multiple forwarding rules for the same backend service lets you do the following:
Assign multiple IP addresses to the load balancer. You can create multiple forwarding rules, each using a unique IP address. Each forwarding rule can specify all ports or a set of up to five ports.
Assign a specific set of ports, using the same IP address, to the load balancer. You can create multiple forwarding rules sharing the same IP address, where each forwarding rule uses a specific set of up to five ports. This is an alternative to configuring a single forwarding rule that specifies all ports.
For more information about scenarios involving two or more internal forwarding rules that share a common internal IP address, see Multiple forwarding rules with the same IP address.
When using multiple internal forwarding rules, make sure that you configure the
software running on your backend VMs to bind to all of the forwarding rule IP
addresses or to any address (0.0.0.0/0
for IPv4 or ::/0
for IPv6). The
destination IP address for a packet delivered through the load
balancer is the internal IP address associated with the corresponding internal
forwarding rule. For more information, see TCP and UDP request and return
packets.
Backend service
Each internal passthrough Network Load Balancer has one regional internal backend service that defines backend parameters and behavior. The name of the backend service is the name of the internal passthrough Network Load Balancer shown in the Google Cloud console.
Each backend service defines the following backend parameters:
Protocol. A backend service supports IPv4 and IPv6 traffic. If the protocol has a concept of a port (like
TCP
orUDP
), the backend service delivers packets to backend VMs on the same destination port to which traffic was sent.Backend services support the following IPv4 protocol options:
TCP
,UDP
, orUNSPECIFIED
.Backend services support the following IPv6 protocol options:
TCP
orUDP
. The backend service protocol must coordinate with the forwarding rule protocol.For a table with possible forwarding rule and backend service protocol combinations, see Forwarding rule protocol specification.
Traffic distribution. A backend service allows traffic to be distributed according to a configurable session affinity.
Health check. A backend service must have an associated health check.
Each backend service operates in a single region and distributes traffic for backend VMs in a single VPC network:
One type of backend, one region. Backends are either instance groups or zonal NEGs with
GCE_VM_IP
endpoints located in the same region as the backend service and forwarding rule. Instance group backends can be unmanaged instance groups, zonal managed instance groups, or regional managed instance groups. Zonal NEG backends must useGCE_VM_IP
endpoints.One VPC network. Each instance group or NEG backend has an associated VPC network, even if that instance group or NEG hasn't been connected to a backend service yet. For more information about how a network is associated with each type of backend, see Instance group backends and network interfaces and Zonal NEG backends and network interfaces. The backend service resource itself also has an associated VPC network, which can be defined explicitly or implicitly. For more information about the backend service's network, see Backend service network specification and Backend service network rules.
Instance group backends and network interfaces
The VPC network associated with an instance group is the
VPC network used by every member VM's nic0
network interface.
- For managed instance groups (MIGs), the VPC network for the instance group is defined in the instance template.
- For unmanaged instance groups, the VPC network for the instance
group is defined as the VPC network used by the
nic0
network interface of the first VM instance that you add to the unmanaged instance group.
Within a given (managed or unmanaged) instance group, all VM instances must
have their nic0
network interfaces in the same VPC network.
Each member VM can optionally have additional network interfaces (nic1
through nic7
), but each network interface must attach to a different
VPC network. These networks must also be different from the
VPC network associated with the instance group.
Zonal NEG backends and network interfaces
When you create a new zonal NEG with GCE_VM_IP
endpoints, you must explicitly
associate the NEG with a subnetwork of a VPC network before you
can add any endpoints to the NEG. Neither the subnet nor the VPC
network can be changed after the NEG is created.
Within a given NEG, each GCE_VM_IP
endpoint actually represents a network
interface. The network interface must be in the subnetwork associated with the
NEG. From the perspective of a Compute Engine instance, the network
interface can use any identifier (nic0
through nic7
). From the perspective
of being an endpoint in a NEG, the network interface is identified by using its
primary IPv4 address.
There are two ways to add a GCE_VM_IP
endpoint to a NEG:
- If you specify only a VM name (without any IP address) when adding an endpoint, Google Cloud requires that the VM has a network interface in the subnetwork associated with the NEG. The IP address that Google Cloud chooses for the endpoint is the primary internal IPv4 address of the VM's network interface in the subnetwork associated with the NEG.
- If you specify both a VM name and an IP address when adding an endpoint, the IP address that you provide must be a primary internal IPv4 address for one of the VM's network interfaces. That network interface must be in the subnetwork associated with the NEG. Note that specifying an IP address is redundant because there can only be a single network interface that is in the subnetwork associated with the NEG.
Backend service network specification
You can explicitly associate a VPC network with a backend
service by using the --network
flag when you create the backend service. The
backend service network rules go into effect
immediately.
If you omit the --network
flag when you create a backend service,
Google Cloud uses one of the following qualifying events to implicitly
set the backend service's associated VPC network. After it is set,
the backend service's associated VPC network cannot be changed:
If the backend service doesn't yet have any associated backends, and you configure the first forwarding rule to reference the backend service, Google Cloud sets the backend service's associated VPC network to the VPC network that contains the subnet used by this forwarding rule.
If the backend service doesn't yet have a forwarding rule referencing it, and you add the first instance group backend to the backend service, Google Cloud sets the backend service's VPC network to the VPC network associated with that first instance group backend. This means that the backend service's associated VPC network is the network used by the
nic0
network interface of each VM in the first instance group backend.If the backend service doesn't yet have a forwarding rule referencing it, and you add the first zonal NEG backend (with
GCE_VM_IP
endpoints) to the backend service, Google Cloud sets the backend service's VPC network to the VPC network associated with that first NEG backend.
After the backend service's VPC network has been set by a qualifying event, any additional forwarding rules, backend instance groups, or backend zonal NEGs that you add must follow the backend service network rules.
Backend service network rules
The following rules apply after a backend service is associated with a specific VPC network:
When configuring a forwarding rule to reference the backend service, the forwarding rule must use a subnet of the backend service's VPC network. The forwarding rule and backend service cannot use different VPC networks, even if those networks are connected by using VPC Network Peering.
When adding instance group backends, one of the following must be true:
- The VPC network associated with the instance
group—that is, the VPC network used by the
nic0
network interface of each VM in the instance group—must match the backend service's associated VPC network. - Each backend VM must have a non-
nic0
interface (nic1
throughnic7
) in the VPC network associated with the backend service.
- The VPC network associated with the instance
group—that is, the VPC network used by the
When adding zonal NEG backends with
GCE_VM_IP
endpoints, the VPC network associated with the NEG must match the VPC network associated with the backend service.
Backend subsetting
Backend subsetting is an optional feature that improves performance by limiting the number of backends to which traffic is distributed.
You should only enable subsetting if you need to support more than 250 backend VMs on a single load balancer. For more information, see Backend subsetting for internal passthrough Network Load Balancer.
Health check
The load balancer's backend service must be associated with a global or regional health check. Special routes outside of the VPC network facilitate communication between health check systems and the backends.
You can use an existing health check or define a new one. The internal passthrough Network Load Balancers use health check status to determine how to route new connections, as described in Traffic distribution.
You can use any of the following health check protocols; the protocol of the health check does not have to match the protocol of the load balancer:
- HTTP, HTTPS, or HTTP/2. If your backend VMs serve traffic by using HTTP, HTTPS, or HTTP/2, it's best to use a health check that matches that protocol because HTTP-based health checks offer options appropriate to that protocol. Serving HTTP-type traffic through an internal passthrough Network Load Balancer means that the load balancer's protocol is TCP.
- SSL or TCP. If your backend VMs do not serve HTTP-type traffic, you should use either an SSL or TCP health check.
Regardless of the type of health check that you create, Google Cloud sends
health check probes to the IP address of the internal passthrough Network Load Balancer's forwarding
rule, to the network interface in the VPC network selected by the load
balancer's backend service. This simulates how load balanced traffic is
delivered. Software running on your backend VMs must respond to both load
balanced traffic and health check probes sent to each forwarding rule's IP
address (the software must listen on 0.0.0.0:<port>
and not
on a specific IP address assigned to a network interface). For more information,
see Destination for probe packets.
Health checks and UDP traffic
Google Cloud does not offer a health check that uses the UDP protocol. When you use an internal passthrough Network Load Balancer with UDP traffic, you must run a TCP-based service on your backend VMs to provide health check information.
In this configuration, client requests are load balanced by using the UDP protocol, and a TCP service is used to provide information to Google Cloud health check probers. For example, you can run a simple HTTP server on each backend VM that returns an HTTP 200 response to Google Cloud. In this example, you should use your own logic running on the backend VM to ensure that the HTTP server returns 200 only if the UDP service is properly configured and running.
High availability architecture
The internal passthrough Network Load Balancer is highly available by design. There are no special steps to make the load balancer highly available because the mechanism doesn't rely on a single device or VM instance.
To ensure that your backend VM instances are deployed to multiple zones, follow these deployment recommendations:
Use regional managed instance groups if you can deploy your software by using instance templates. Regional managed instance groups automatically distribute traffic among multiple zones, providing the best option to avoid potential issues in any given zone.
If you use zonal managed instance groups or unmanaged instance groups, use multiple instance groups in different zones (in the same region) for the same backend service. Using multiple zones protects against potential issues in any given zone.
Shared VPC architecture
The following table summarizes the component requirements for internal passthrough Network Load Balancers used with Shared VPC networks. For an example, see creating an internal passthrough Network Load Balancer on the Provisioning Shared VPC page.
IP address | Forwarding rule | Backend components |
---|---|---|
An
internal IP address must be defined in the same project
as the backend VMs.
For the load balancer to be available in a Shared VPC network, the Google Cloud internal IP address must be defined in the same service project where the backend VMs are located, and it must reference a subnet in the desired Shared VPC network in the host project. The address itself comes from the primary IP range of the referenced subnet. If you create an internal IP address in a service project and the IP address subnet is in the service project's VPC network, your internal passthrough Network Load Balancer is local to the service project. It's not local to any Shared VPC host project. |
An internal
forwarding rule must be defined in the same project as the backend
VMs.
For the load balancer to be available in a Shared VPC network, the internal forwarding rule must be defined in the same service project where the backend VMs are located, and it must reference the same subnet (in the Shared VPC network) that the associated internal IP address references. If you create an internal forwarding rule in a service project and the forwarding rule's subnet is in the service project's VPC network, your internal passthrough Network Load Balancer is local to the service project. It's not local to any Shared VPC host project. |
In a Shared VPC scenario, backend VMs are located in a service project. A regional internal backend service and health check must be defined in that service project. |
Traffic distribution
The way that an internal passthrough Network Load Balancer distributes new connections depends on whether you have configured failover:
If you haven't configured failover, an internal passthrough Network Load Balancer distributes new connections to its healthy backend VMs if at least one backend VM is healthy. When all backend VMs are unhealthy, the load balancer distributes new connections among all backends as a last resort. In this situation, the load balancer routes each new connection to an unhealthy backend VM.
If you have configured failover, an internal passthrough Network Load Balancer distributes new connections among VMs in its active pool, according to a failover policy that you configure. When all backend VMs are unhealthy, you can choose from one of the following behaviors:
- (Default) The load balancer distributes traffic to only the primary VMs. This is done as a last resort. The backup VMs are excluded from this last-resort distribution of connections.
- The load balancer is configured to drop traffic.
The method for distributing new connections depends on the load balancer's session affinity setting.
The health check state controls the distribution of new connections. By default, TCP connections persist on unhealthy backends. For more details and how to change this behavior, see Connection persistence on unhealthy backends.
Backend selection and connection tracking
Internal passthrough Network Load Balancers use configurable backend selection and connection tracking algorithms to determine how traffic is distributed to backend VMs. Internal passthrough Network Load Balancers use the following algorithm to distribute packets among backend VMs (in its active pool, if you have configured failover):
- If the load balancer has an entry in its connection tracking table matching the characteristics of an incoming packet, the packet is sent to the backend specified by the connection tracking table entry. The packet is considered to be part of a previously established connection, so the packet is sent to the backend VM that the load balancer previously determined and recorded in its connection tracking table.
If the load balancer receives a packet for which it has no connection tracking entry, the load balancer does the following:
The load balancer selects a backend. The load balancer calculates a hash based on the configured session affinity. It uses this hash to select a backend:
- If at least one backend is healthy, the hash selects one of the healthy backends.
- If all backends are unhealthy, and there's no failover policy configured, the hash selects one of the backends.
- If all backends are unhealthy, there is a failover policy configured, and the failover policy is not configured to drop traffic in this situation, the hash selects one of the primary VM backends.
The default session affinity,
NONE
, uses a 5-tuple hash of the packet's source IP address, source port, destination IP address, destination port, and the protocolBackend selection can be customized by using a hash algorithm that uses fewer pieces of information. For all the supported options, see session affinity options.
The load balancer adds an entry to its connection tracking table. This entry records the selected backend for the packet's connection so that all future packets from this connection are sent to the same backend. Whether connection tracking is used depends on the protocol:
For TCP and UDP packets, connection tracking is always enabled, and cannot be turned off. By default, connection tracking is 5-tuple, but it can be configured to be less than 5-tuple.
When the connection tracking hash is 5-tuple, TCP SYN packets always create a new entry in the connection tracking table.
The default 5-tuple connection tracking is used when:- tracking mode is
PER_CONNECTION
(all session affinities), or, - tracking mode is
PER_SESSION
and the session affinity isNONE
, or, - tracking mode is
PER_SESSION
and the session affinity isCLIENT_IP_PORT_PROTO
.
For additional details about when connection tracking is enabled, and what tracking method is used when connection tracking is enabled, see connection tracking mode.
In addition, note the following:
- By default, an entry in the connection tracking table expires 600 seconds after the load balancer processes the last packet that matched the entry. For details about how to customize the idle timeout, see Idle timeout.
- Depending on the protocol, the load balancer might remove connection tracking table entries when backends become unhealthy. For details and how to customize this behavior, see Connection persistence on unhealthy backends.
- tracking mode is
Session affinity options
Session affinity controls the distribution of new connections from clients to the load balancer's backend VMs. You set session affinity when your backend VMs need to keep track of state information for their clients. This is a common requirement for web applications.
Session affinity works on a best-effort basis.
Internal passthrough Network Load Balancers support the following session affinity options, which you specify for the entire internal backend service, not per backend instance group.
- None (
NONE
). 5-tuple hash of source IP address, source port, protocol, destination IP address, and destination port - Client IP, no destination (
CLIENT_IP_NO_DESTINATION
). 1-tuple hash created from just the source IP address. - Client IP (
CLIENT_IP
). 2-tuple hash of source IP address and destination IP address. External passthrough Network Load Balancers call this session affinity option Client IP, Destination IP. - Client IP, Destination IP, Protocol (
CLIENT_IP_PROTO
). 3-tuple hash of source IP address, destination IP address, and protocol - Client IP, Client Port, Destination IP, Destination Port, Protocol
(
CLIENT_IP_PORT_PROTO
). 5-tuple hash of source IP address, source port, protocol, destination IP address, and destination port
Unless you use the load balancer as a next hop for a custom static route, a packet's destination IP address must match the IP address of the load balancer's forwarding rule for the packet to be delivered to the load balancer. For considerations when using the load balancer as a next hop for a custom static route, Session affinity and next hop internal passthrough Network Load Balancer.
Session affinity and next hop internal passthrough Network Load Balancer
When you configure a static route to use a next hop internal passthrough Network Load Balancer, the
load balancer uses the same backend selection and connection
tracking method previously discussed. Backend selection
is still accomplished by calculating a hash according to the configured session
affinity. Except for CLIENT_IP_NO_DESTINATION
session affinity, the backend
selection hash depends, in part, on the packet destination IP address.
When a internal passthrough Network Load Balancer load balancer is a next hop of a static route, the destination IP address is not limited to the forwarding rule IP address of the load balancer. Instead, the destination IP address of the packet can be any IP address that fits within the destination range of the static route.
If the number of configured and healthy backend VMs is constant (when failover is either not configured, or, when failover is configured but no failover or failback events occur), the load balancer behaves as follows:
If there's just one configured and healthy backend VM (in the active pool, if failover is configured), it doesn't matter what session affinity you use because all hashes are mapped to the one backend VM.
If there are two or more configured and healthy backend VMs (in the active pool, if failover is configured), your choice of session affinity is important:
If you need the same backend VM to process all packets from a client, based solely on the source IP address of the packet, regardless of the packet destination IP addresses, use
CLIENT_IP_NO_DESTINATION
session affinity. Depending on traffic patterns, some backend VMs might receive more packets or more connections than other backend VMs.If you use a session affinity option that's not
CLIENT_IP_NO_DESTINATION
, the load balancer selects a backend VM based on information that at least includes both the source IP address and the destination IP address of the packet. Packets sent from the same client, using the same source IP address, but different destination IP addresses, can be routed to different backend VMs.
Connection tracking mode
Tracking mode specifies the connection tracking algorithm to be used. There are
two tracking modes: PER_CONNECTION
(default) and PER_SESSION
.
PER_CONNECTION
(default). In this mode, TCP and UDP traffic are always tracked per 5-tuple, regardless of the session affinity setting. This implies that the key for connection tracking (5-tuple) can be different from the configured session affinity setting (for example, 3-tuple with theCLIENT_IP_PROTO
setting). As a result, the session affinity may be split and new connections for a session may select a different backend if the set of backends or their health changes.PER_SESSION
. In this mode, TCP and UDP traffic is tracked according to the configured session affinity. That is, if session affinity isCLIENT_IP
orCLIENT_IP_PROTO
, configuring this mode results in 2-tuple and 3-tuple connection tracking, respectively. This might be desirable in scenarios where breaking affinity is expensive and should be avoided even after more backends are added.
The connection tracking mode setting is redundant if session affinity is set to
NONE
or CLIENT_IP_PORT_PROTO
. To learn how these tracking modes work with
different session affinity settings for each protocol, see the following table.
Backend selection | Connection tracking mode | ||
---|---|---|---|
Session affinity setting | Hash method for backend selection | PER_CONNECTION (default) |
PER_SESSION |
Default: No session affinity
|
5-tuple hash | 5-tuple connection tracking | 5-tuple connection tracking |
Client IP, no destination
|
1-tuple hash | 5-tuple connection tracking | 1-tuple connection tracking |
Client IP
(same as Client IP, Destination IP for external passthrough Network Load Balancers) |
2-tuple hash | 5-tuple connection tracking | 2-tuple connection tracking |
Client IP, Destination IP, Protocol
|
3-tuple hash | 5-tuple connection tracking | 3-tuple connection tracking |
Client IP, Client Port, Destination IP, Destination Port, Protocol
|
5-tuple hash | 5-tuple connection tracking | 5-tuple connection tracking |
To learn how to change the connection tracking mode, see Configure a connection tracking policy.
Connection persistence on unhealthy backends
The connection persistence on unhealthy backends settings control whether an existing connection persists on a selected backend after that backend becomes unhealthy (as long as the backend remains in the load balancer's configured backend instance group).
The behavior described in this section does not apply to cases where you remove a backend VM from its instance group, or remove the instance group from the backend service. In such cases, established connections only persist as described in Enabling connection draining.
The following connection persistence options are available:
DEFAULT_FOR_PROTOCOL
(default)NEVER_PERSIST
ALWAYS_PERSIST
The following table summarizes connection persistence options and how connections persist for different protocols, session affinity options, and tracking modes.
Connection persistence on unhealthy backends option | Connection tracking mode | |
---|---|---|
PER_CONNECTION |
PER_SESSION |
|
DEFAULT_FOR_PROTOCOL |
TCP: connections persist on unhealthy backends (all session affinities) UDP: connections never persist on unhealthy backends |
TCP: connections persist on unhealthy backends if
session affinity is UDP: connections never persist on unhealthy backends |
NEVER_PERSIST |
TCP, UDP: connections never persist on unhealthy backends | |
ALWAYS_PERSIST
|
TCP, UDP: connections persist on unhealthy backends (all session affinities) This option should only be used for advanced use cases. |
Configuration not possible |
To learn how to change connection persistence behavior, see Configure a connection tracking policy.
Idle timeout
By default, an entry in the connection tracking table expires 600 seconds after
the load balancer processes the last packet that matched the entry. This
default idle timeout value can be modified only when the connection tracking is
less than 5-tuple (that is, when session affinity is configured to be either
CLIENT_IP
or CLIENT_IP_PROTO
, and the tracking mode is PER_SESSION
).
The maximum configurable idle timeout value is 57,600 seconds (16 hours).
To learn how to change the idle timeout value, see Configure a connection tracking policy.
Testing connections from a single client
When testing connections to the IP address of an internal passthrough Network Load Balancer from a single client system, you should keep the following in mind:
If the client system is not a VM being load balanced—that is, not a backend VM, new connections are delivered to the load balancer's healthy backend VMs. However, because all session affinity options rely on at least the client system's IP address, connections from the same client might be distributed to the same backend VM more frequently than you might expect.
Practically, this means that you cannot accurately monitor traffic distribution through an internal passthrough Network Load Balancer by connecting to it from a single client. The number of clients needed to monitor traffic distribution varies depending on the load balancer type, the type of traffic, and the number of healthy backends.
If the client VM is a backend VM of the load balancer, connections sent to the IP address of the load balancer's forwarding rule are always answered by the client/backend VM. This happens regardless of whether the backend VM is healthy. It happens for all traffic sent to the load balancer's IP address, not just traffic on the protocol and ports specified in the load balancer's internal forwarding rule.
For more information, see sending requests from load-balanced VMs.
UDP fragmentation
Internal passthrough Network Load Balancers can process both fragmented and unfragmented UDP packets. If your application uses fragmented UDP packets, keep the following in mind:- UDP packets might become fragmented before reaching a Google Cloud VPC network.
- Google Cloud VPC networks forward UDP fragments as they arrive (without waiting for all fragments to arrive).
- Non-Google Cloud networks and on-premises network equipment might forward UDP fragments as they arrive, delay fragmented UDP packets until all fragments have arrived, or discard fragmented UDP packets. For details, see the documentation for the network provider or network equipment.
If you expect fragmented UDP packets and need to route them to the same backends, use the following forwarding rule and backend service configuration parameters:
Forwarding rule configuration: Use only one
UDP
forwarding rule per load-balanced IP address, and configure the forwarding rule to accept traffic on all ports. This ensures that all fragments arrive at the same forwarding rule. Even though the fragmented packets (other than the first fragment) lack a destination port, configuring the forwarding rule to process traffic for all ports also configures it to receive UDP fragments that have no port information. To configure all ports, either use the Google Cloud CLI to set--ports=ALL
or use the API to setallPorts
toTrue
.Backend service configuration: Set the backend service's session affinity to
CLIENT_IP
(2-tuple hash) orCLIENT_IP_PROTO
(3-tuple hash) so that the same backend is selected for UDP packets that include port information and UDP fragments (other than the first fragment) that lack port information. Set the backend service's connection tracking mode toPER_SESSION
so that the connection tracking table entries are built by using the same 2-tuple or 3-tuple hashes.
Failover
An internal passthrough Network Load Balancer lets you designate some backends as failover backends. These backends are only used when the number of healthy VMs in the primary backend instance groups has fallen below a configurable threshold. By default, if all primary and failover VMs are unhealthy, as a last resort Google Cloud distributes new connections only among all the primary VMs.
When you add a backend to an internal passthrough Network Load Balancer's backend service, by default that backend is a primary backend. You can designate a backend to be a failover backend when you add it to the load balancer's backend service, or by editing the backend service later.
For a detailed conceptual overview of failover in internal passthrough Network Load Balancers, see Failover for internal passthrough Network Load Balancers.
Quotas and limits
For information about quotas and limits, see Load balancing resource quotas.
Limitations
- You cannot use the Google Cloud console to create an internal passthrough Network Load Balancer with zonal NEG backends.
What's next
- To configure and test an internal passthrough Network Load Balancer, see Set up an internal passthrough Network Load Balancer.
- To configure Cloud Monitoring for internal passthrough Network Load Balancers, see Internal passthrough Network Load Balancer logging and monitoring.
- To troubleshoot issues with your internal passthrough Network Load Balancer, see Troubleshoot internal passthrough Network Load Balancers.