Next-hop concepts for Internal TCP/UDP Load Balancing

Internal TCP/UDP Load Balancing is a regional load balancer that enables you to run and scale your services behind a private load balancing IP address. You can use an internal TCP/UDP load balancer as the next gateway to which packets are forwarded along the path to their final destination. To do this, you set the load balancer as the next hop in a custom static route. This configuration is useful in the following cases:

  • When you need an internal network address translation (NAT) gateway instance that can route traffic from internal-only virtual machine instances to the internet. Using a load balancer as the next hop allows you to use one external IP address to send traffic from multiple virtual machine instances but only expose a single virtual machine to the internet.

  • When you need a load balancer to serve as a next hop for a default route. When virtual machine instances in your VPC network send traffic to the internet, the traffic is routed through load-balanced gateway virtual appliances.

Overview

You can create a custom static route to pass TCP and UDP traffic to an internal TCP/UDP load balancer where the load balancer is the next hop for the static route. The route can be the default route (0.0.0.0/0), a publicly-routable (non-RFC 1918) CIDR prefix, or an internal (RFC 1918) CIDR prefix, if the prefix doesn't conflict with a subnet route. For example, you can replace your default route (0.0.0.0/0) with a route that directs traffic to third-party backend VMs for packet processing.

To use the custom static route, the instances in your VPC network must be in the same region as the load balancer.

You can advertise this route to other connected networks by using Cloud Router custom route advertisements, if the Cloud Router is in the same region as the load balancer. For more information, refer to Internal TCP/UDP Load Balancing and Connected Networks.

Benefits of using your internal TCP/UDP load balancer as a next hop

When the load balancer is a next hop for a static route, you don't need to explicitly configure your clients to send traffic to the load balancer or to each backend instance. You can integrate your backend instances in a bump-in-the-wire fashion.

Using an internal TCP/UDP load balancer as a next hop for a static route provides the same benefits as Internal TCP/UDP Load Balancing. For example, backends scale appropriately with traffic demands and are highly available.

The load balancer's health check ensures that new connections are routed to healthy backend instances. By adding your backend instances to a managed instance group, you can use autoscaling to grow or shrink the set of instances based on service demand. Because Internal TCP/UDP Load Balancing uses internal addresses, you don't need to expose public IP addresses to the internet.

Specifications

Client IP session affinity

  • Client IP session affinity is a two-tuple affinity (with source and destination IP address in the hash function). For regular internal TCP load balancer scenarios, the destination IP address is the load balancer's IP address, and all packets from a single client VM source go to the same backend, since the destination address doesn't vary. When the internal TCP load balancer is a next hop instead of the destination, the destination IP addresses can vary. This means that even if you configure client IP session affinity, traffic from a single client can go to multiple backends.

Destination range

  • If the route's destination is an internal (RFC 1918) IP range, the destination cannot be equal to or more specific than a subnet route. Also, the internal IP address associated with the load balancer's forwarding rule must be in a different subnet than the route's destination.

Same VPC network and region

Internal TCP/UDP load balancers are regional resources.

  • The internal TCP/UDP load balancer and the custom static route that uses it as a next hop must be in the same VPC network.

  • The custom static route that uses an internal TCP/UDP load balancer as a next hop is only available to resources in the same region as the load balancer. This regional restriction is enforced even though the route itself is part of the routing table for the whole VPC network.

Load balancer must be created before route

You can only create a custom static route whose next hop is an internal TCP/UDP load balancer if the load balancer already exists. Attempting to create a route that refers to a nonexistent load balancer will result in an error message indicating that the load balancer resource cannot be found.

After you've created a route whose next hop is an internal TCP/UDP load balancer, that load balancer cannot be deleted unless you first delete the route. Specifically, deletion of the load balancer's internal forwarding rule is prohibited unless no routes exist that use it as a next hop.

Route limitations

  • A custom static route cannot use network tags if the route has an internal TCP/UDP load balancer as the route's next hop.

  • As is true for any GCP route, the internal TCP/UDP load balancer route must have only one next hop specified. For example, you can't create a route with both --next-hop-ilb and next-hop-address.

  • You cannot create multiple custom static routes with the same priority, the same destination, and the same internal TCP/UDP load balancer next hop. Consequently, each custom static route with the same next hop load balancer must have at least either a unique destination or a unique priority.

  • You cannot distribute traffic among multiple internal TCP/UDP load balancers using equal-cost multi-path (ECMP) routing. This means that you cannot create routes with the same destination and same priority but with different internal TCP/UDP load balancers as next hops.

Refer to Routes overview for more information.

Backend requirement

  • The internal TCP/UDP load balancer's backend VMs must be configured to allow IP forwarding (--can-ip-forward = True). Refer to Instances as next hops for more information.
  • GKE backends aren't supported with internal TCP/UDP load balancers set as the next hop.

All TCP and UDP traffic

When using an internal TCP/UDP load balancer as a next hop, all TCP and UDP traffic is sent to backend VMs. This is true regardless of the backend service's protocol, the forwarding rule's protocol, and the forwarding rule's port specification.

Use cases

There are multiple deployments and topologies in which you can use an internal TCP/UDP load balancer as a next hop.

Some virtual appliance use cases involve North-South traffic, and some involve East-West traffic.

North-South means a GCP VPC network connected to an on-premises network or a non-GCP VPC network.

East-West means GCP VPC networks connected to each other.

You might connect your VPC network to an on-premises network using Cloud Interconnect (Dedicated or Partner) or through a Cloud VPN tunnel.

For the following examples, note:

  • The firewall instances are VMs with multiple NICs. Each interface must be in a separate VPC network, not just a separate subnet of the same VPC network. You cannot use backend VMs or load balancers to route traffic between subnets in the same VPC network because subnet routes cannot be overridden.

  • The primary network interface (nic0) of each firewall instance must be in the same VPC network as the internal TCP/UDP load balancer. Internal TCP/UDP load balancers only distribute traffic to the primary network interface of a backend VM.

  • The two diagrams in each use case are paired:

    • One diagram shows North-to-South traffic flow and load balancer 1.
    • The other diagram shows South-to-North traffic flow and load balancer 2.

    Two load balancers are required because internal TCP/UDP load balancing only distributes traffic to the first network interface (nic0) of each backend VM.

  • The internal TCP/UDP load balancer is a software-defined pass-through load balancer. NAT and proxying are only performed by the backend VMs — the firewall virtual appliances.

Instance as a NAT gateway

This use case load balances traffic from internal VMs to multiple NAT gateway instances that route traffic to the internet.

NAT use case (click to enlarge)
NAT use case (click to enlarge)

Some North-South traffic through load-balanced virtual appliances

In this use case, the backend VMs operate as firewall instances with NICs in multiple VPC networks. These firewall instances can be provided by a third-party as virtual appliances. On-premises systems connect to the transit VPC network using Cloud VPN or Cloud Interconnect.

Example: TCP traffic from an on-premises network and the established response

When an on-premises node at 10.0.5.100 sends a packet to a Compute Engine serving instance at 10.0.3.100, the packet is routed as shown in the following figure.

Some North-to-South traffic  (click to enlarge)
Some North-to-South traffic (click to enlarge)

Request path

When sending a request packet to the VM instances, the packet is routed as follows:

  1. The on-premises router sends the packet to the VPC network named transit, through a Cloud VPN tunnel or interconnect attachment (VLAN). The Cloud Router provides the route using a custom route advertisement for 10.0.3.0/24.

  2. When the packet arrives in the transit VPC network, a custom static route for the 10.0.3.0/24 destination sends the packet to the load balancer that uses the fr-ilb1 forwarding rule.

  3. GCP Internal TCP/UDP Load Balancing delivers the packet to the primary interface (nic0) of a healthy backend VM. In this example, the backend VMs are firewall virtual appliances.

  4. In addition to performing its filtering function, the firewall instance uses routing configured within its operating system to modify the packet, performing Source Network Address Translation (SNAT). SNAT changes the packet's source address to the IP address of the firewall instance's secondary network interface, nic1, which is located in the VPC network named production. For this example, if firewall-instance-a processes the packet, the firewall instance changes the source address of the packet to 10.0.3.2.

  5. The firewall instance consults its internal routing table, and sends the packet from its secondary interface.

  6. A subnet route in the production VPC network delivers the packet to the serving instance at 10.0.3.100. The received packet has a source IP address belonging to nic1 of a firewall instance (for example, 10.0.3.2), and the destination address10.0.3.100.

Response path

When the serving instance at 10.0.3.100 sends an established response packet back to the on-premises system, the packet is routed in the following way:

  1. Because the serving instance receives the incoming packet after SNAT is performed, the response packet has a source of 10.0.3.100 and a destination IP address belonging to the secondary network interface, nic1, of a firewall instance. For this example, the IP address is 10.0.3.2. The response packet is sent to the same firewall instance that delivered the corresponding incoming packet. A subnet route in the production VPC network delivers the packet to that firewall instance.

  2. Routing configured within the operating system of the firewall instance modifies the packet, performing Destination Network Address Translation (DNAT), changing the destination to 10.0.5.100. The firewall instance consults its internal routing table, and then delivers the packet to the instance's primary network interface (nic0), which is located in the transit VPC network.

  3. A custom dynamic route in the transit VPC network, learned by the Cloud Router, sends the packet to the on-premises network through the Cloud VPN tunnel or Cloud Interconnect VLAN.

  4. Routing within the on-premises network delivers the packet to the node at 10.0.5.100.

Example: TCP traffic from GCP and established response

When the serving GCP VM at 10.0.3.100 sends a packet to an on-premises system at 10.0.5.100, the packet is routed in the following way:

Some South-to-North traffic (click to enlarge)
Some South-to-North traffic (click to enlarge)

Request path

  1. In the production VPC network, a custom static route for the 10.0.5.0/24 destination sends the packet to the load balancer that uses the fr-ilb2 forwarding rule.

  2. GCP Internal TCP/UDP Load Balancing delivers the packet to the primary interface,nic0, of a healthy backend VM that is a firewall instance.

  3. Routing configured within the operating system of the firewall instance modifies the packet, performing Source Network Address Translation (SNAT), changing the packet's source address to the IP address of the firewall instance's secondary network interface, nic1, which is located in the transit VPC network. For this example, if the packet is processed by firewall-instance-e, the source IP address is changed to 10.0.1.6.

  4. The firewall instance consults its internal routing table and sends the packet out the secondary interface.

  5. A custom dynamic route in the transit VPC network, learned by the Cloud Router, sends the packet to the on-premises network through the Cloud VPN tunnel or Cloud Interconnect VLAN.

  6. Routing within the on-premises network delivers the packet to 10.0.5.100.

Response path

When the on-premises system at 10.0.5.100 sends an established response packet back to the serving GCP VM, the packet is routed in the following way:

  1. Because the on-premises system received the incoming packet after SNAT was performed, the response packet has a source IP address of 10.0.5.100 and a destination IP address belonging to the secondary network interface, nic1, of a firewall instance. For this example, the IP address is 10.0.1.6. The on-premises router sends the response packet to the same firewall instance that delivered the corresponding incoming packet, through a Cloud VPN tunnel or interconnect attachment (VLAN). The route is provided by the Cloud Router advertising subnet routes for the transit VPC network, including 10.0.1.0/24.

  2. When the packet arrives in the transit VPC network, a subnet route for the 10.0.1.0/24 destination sends the packet to the IP address belonging to the secondary network interface, nic1, of a firewall instance. For this example, the IP address is 10.0.1.6. The response packet is sent to the same firewall instance that delivered the corresponding incoming packet.

  3. Routing configured within the operating system of the firewall instance modifies the packet, performing Destination Network Address Translation (DNAT), changing the packet's destination to 10.0.3.100. The firewall instance consults its internal routing table and delivers the packet to the instance's primary network interface, nic0, which is located in the production VPC network.

  4. A subnet route in the production VPC network delivers the packet to the serving instance at 10.0.3.100.

VPC network-to-VPC network connectivity through load-balanced virtual appliances

Similar to the previous scenario, a route with an RFC 1918 destination prefix is added to VM instances in the source VPC network transit with the internal TCP/UDP load balancer as the next hop.

East-West traffic (click to enlarge)
East-West traffic (click to enlarge)

For traffic that originates from VPC network production, you must use a different set of load-balanced VM instances behind another internal TCP/UDP load balancer because internal TCP/UDP load-balancing only supports the primary interface (nic0) of backend VM instances.

West-East traffic (click to enlarge)
West-East traffic (click to enlarge)

Hub and spoke – Exchanging next-hop routes through VPC Peering

With VPC Network Peering, GCP can export and import both static and dynamic routes, if they don't have tags and if they don't have the internet gateway defined as the next hop.

You can configure a hub-and-spoke topology with your next-hop firewall virtual appliances located in the Hub VPC network peered with Spoke VPC networks.

You can then configure the internal TCP/UDP load balancer as the next-hop route in the Hub VPC network and export it to the peered Spoke VPC networks. The internal TCP/UDP load balancer next-hop resource and the internal TCP/UDP load balancer forwarding rule resource are owned by the Hub VPC network.

Hub and spoke (click to enlarge)
Hub and spoke (click to enlarge)

For information about routing priority, refer to Routing order.

What's next

Was this page helpful? Let us know how we did:

Send feedback about...

Load Balancing