External TCP proxy load balancer overview

This document introduces the concepts that you need to understand to configure a Google Cloud external TCP proxy load balancer.

An external TCP proxy load balancer is a reverse proxy load balancer that distributes TCP traffic coming from the internet to the backends in your Google Cloud Virtual Private Cloud (VPC) network. Traffic coming over a TCP connection is terminated at the load balancing layer and then forwarded to the closest available backend by using TCP.

An external TCP proxy load balancer lets you use a single IP address for all users worldwide.

Modes of operation

You can configure an external TCP proxy load balancer in the following modes:

  • An external TCP proxy load balancer is implemented on globally distributed GFEs. This load balancer is intended for TCP traffic on specific well-known ports, such as port 25 for Simple Mail Transfer Protocol (SMTP). For client traffic that is encrypted on these same ports, use the external TCP proxy load balancer. With the Network Service Tiers Premium Tier, this load balancer can be configured as a global load balancing service. With Standard Tier, this load balancer is configured as a regional load balancing service.
  • An external regional TCP proxy load balancer is implemented by the open source Envoy proxy software stack. This load balancer is configured as a regional load balancing service that uses the Standard Tier.

Identify the mode

To determine the mode of a load balancer, complete the following steps.

Console

  1. In the Google Cloud console, go to the Load balancing page.

    Go to Load balancing

  2. On the Load Balancers tab, the load balancer type, protocol, and region are displayed. If the region is blank, then the load balancer is global.

    The following table summarizes how to identify the mode of the load balancer.

    Load balancer mode Load balancing type Region Network tier
    External TCP proxy load balancer SSL/TCP (Proxy) Standard or Premium
    External regional TCP proxy load balancer TCP Proxy (External) Specifies a region Standard

gcloud

  1. Use the gcloud compute forwarding-rules describe command:

    gcloud compute forwarding-rules describe FORWARDING_RULE_NAME

  2. In the command output, check the load balancing scheme, region, and network tier. The following table summarizes how to identify the mode of the load balancer.

    Load balancer mode Load balancing scheme Forwarding rule Network tier
    External TCP proxy load balancer EXTERNAL Global Standard or Premium
    External regional TCP proxy load balancer EXTERNAL_MANAGED Regional Standard

Benefits

External TCP proxy load balancers provide the following benefits:

  • IPv6 termination. External TCP proxy load balancers support both IPv4 and IPv6 addresses for client traffic. Client IPv6 requests are terminated at the load balancing layer and then proxied over IPv4 addresses to your backends.
  • Intelligent routing. The load balancer can route requests to backend locations where there is capacity. In contrast, a Layer 3/Layer 4 load balancer must route requests to regional backends without considering capacity. The use of smarter routing allows provisioning at N+1 or N+2 instead of x*N.
  • Security patching. If vulnerabilities arise in the TCP stack, Cloud Load Balancing applies patches at the load balancer automatically to keep your backends safe.
  • Support for all ports. External TCP proxy load balancers allow any valid port from 1-65535.
  • Integration with Google Cloud Armor. You can use Google Cloud Armor security policies to protect your infrastructure from distributed denial-of-service (DDoS) attacks and other targeted attacks.

Architecture

The following diagrams compare the components of global and regional load balancers.

Global

This diagram shows the components of an external TCP proxy load balancer deployment.

External TCP proxy load balancer components
External TCP proxy load balancer components

Regional

This diagram shows the components of an external regional TCP proxy load balancer deployment.

External regional TCP proxy load balancer components
External regional TCP proxy load balancer components

Proxy-only subnet

The proxy-only subnet provides a set of IP addresses that Google uses to run Envoy proxies on your behalf. You must create one proxy-only subnet in each region of a VPC network where you use load balancers. The --purpose flag for this proxy-only subnet is set to REGIONAL_MANAGED_PROXY. All regional Envoy-based load balancers in the same region and VPC network share a pool of Envoy proxies from the same proxy-only subnet.

Proxy-only subnets are only used for Envoy proxies, not your backends.

Backend VMs or endpoints of all load balancers in a region and VPC network receive connections from the proxy-only subnet.

The IP address of the load balancer is not located in the proxy-only subnet. The load balancer's IP address is defined by its external managed forwarding rule.

Forwarding rules and IP addresses

Forwarding rules route traffic by IP address, port, and protocol to a load balancing configuration that consists of a target proxy and a backend service.

Each forwarding rule provides a single IP address that you can use in DNS records for your application. No DNS-based load balancing is required. You can either reserve a static IP address that you can use or let Cloud Load Balancing assign one for you. We recommend that you reserve a static IP address; otherwise, you must update your DNS record with the newly-assigned ephemeral IP address whenever you delete a forwarding rule and create a new one.

External forwarding rules can reference exactly one of the ports listed in Port specifications. They can specify a single port in the port range (contiguous).

To support multiple consecutive ports, you have to configure multiple forwarding rules. Multiple forwarding rules can be configured with the same virtual IP address and different ports. Therefore, you can proxy multiple applications with separate custom ports to the same TCP proxy virtual IP address.

The following table compares the two types of load balancers.

Load balancer mode Network Service Tier Forwarding rule, IP address, and load balancing scheme Routing from the internet to the load balancer frontend
External TCP proxy load balancer Premium Tier

Global external forwarding rule

Global external IP address

Load balancing scheme: EXTERNAL

 Requests routed to the Google Front Ends (GFEs) that are closest to the client on the internet.
Standard Tier

Regional external forwarding rule

Regional external IP address

Load balancing scheme: EXTERNAL

 Requests routed to a GFE in the load balancer's region.
External regional TCP proxy load balancer Standard Tier

Regional external forwarding rule

Regional external IP address

Load balancing scheme: EXTERNAL_MANAGED

 Requests routed to the Envoy proxies in the same region as the load balancer.

Target proxies

An external TCP proxy load balancer terminates TCP connections from the client and creates new connections to the backends. The target proxy routes these new connections to the backend service.

By default, the original client IP address and port information is not preserved. You can preserve this information by using the PROXY protocol.

Load balancer mode Network Service Tier Target proxy
External TCP proxy load balancer Premium Tier targetTcpProxies
Standard Tier targetTcpProxies
External regional TCP proxy load balancer Standard Tier regionTargetTcpProxies

Backend services

Backend services direct incoming traffic to one or more attached backends. Each backend is composed of an instance group or network endpoint group and information about the backend's serving capacity. Backend serving capacity can be based on CPU or requests per second (RPS).

Each load balancer has a single backend service resource. Backend service changes are not instantaneous; it can take several minutes for changes to propagate to the load balancer.

Each backend service specifies the health checks to perform for the available backends.

To ensure minimal interruptions to your users, you can enable connection draining on backend services. Such interruptions might happen when a backend is terminated, removed manually, or removed by an autoscaler. To learn more about using connection draining to minimize service interruptions, see Enable connection draining.

The following table specifies the backend features supported.

Load balancer mode Supported backends on a backend service
Instance groups Zonal NEGs Internet NEGs Serverless NEGs Hybrid NEGs Private Service Connect NEGs GKE
External TCP proxy load balancer Use standalone zonal NEGs
External regional TCP proxy load balancer GCE_VM_IP_PORT type endpoints Add a Private Service Connect NEG

Backends and VPC networks

All backends must be located in the same project but can be located in different VPC networks. The different VPC networks do not need to be connected by using VPC Network Peering because GFE proxy systems communicate directly with backends in their respective VPC networks.

Protocol for communicating with the backends

When you configure a backend service for the load balancer, you set the protocol that the backend service uses to communicate with the backends. You can choose TCP. The load balancer uses only the protocol that you specify and does not attempt to negotiate a connection with the other protocol.

Firewall rules

The following firewall rules are required:

  • For external TCP proxy load balancers: an ingress allow firewall rule to permit traffic from GFEs to reach your backends.
  • For external regional TCP proxy load balancers: an ingress firewall rule to permit traffic from the proxy-only subnet to reach your backends.
  • An ingress allow firewall rule to permit traffic from the health check probe ranges to reach your backends. For more information about health check probes and why it's necessary to allow traffic from them, see Probe IP ranges and firewall rules.

Firewall rules are implemented at the VM instance level, not at the GFE proxies level. You cannot use Google Cloud firewall rules to prevent traffic from reaching the load balancer.

The ports for these firewall rules must be configured as follows:

  • Allow traffic to the destination port for each backend service's health check.
  • For instance group backends: determine the ports to be configured by the mapping between the backend service's named port and the port numbers associated with that named port on each instance group. Port numbers can vary between instance groups assigned to the same backend service.
  • For GCE_VM_IP_PORT NEG backends: allow traffic to the port numbers of the endpoints.

The following table summarizes the required source IP address ranges for the firewall rules.

Load balancer mode Health check source ranges Request source ranges
External TCP proxy load balancer
  • 35.191.0.0/16
  • 130.211.0.0/22
 These ranges apply to health checks and requests from the GFE.
External regional TCP proxy load balancer
  • 35.191.0.0/16
  • 130.211.0.0/22

Source IP addresses

The source IP address for packets, as seen by the backends, is not the Google Cloud external IP address of the load balancer. In other words, there are two TCP connections.

  • Connection 1, from original client to the load balancer (GFE):

    • Source IP address: the original client (or external IP address if the client is behind NAT or a forward proxy).
    • Destination IP address: your load balancer's IP address.

  • Connection 2, from the load balancer (GFE) to the backend VM or endpoint:

    • Source IP address: an IP address in one of the ranges specified in Firewall rules.

    • Destination IP address: the internal IP address of the backend VM or container in the VPC network.

For the external regional TCP proxy load balancers

  • Connection 1, from the original client to the load balancer (proxy-only subnet):

    • Source IP address: the original client (or external IP address if the client is behind NAT or a forward proxy).
    • Destination IP address: your load balancer's IP address.

  • Connection 2, from the load balancer (proxy-only subnet) to the backend VM or endpoint:

    • Source IP address: an IP address in the proxy-only subnet that is shared among all the Envoy-based load balancers deployed in the same region and network as the load balancer.

    • Destination IP address: the external IP address of the backend VM or container in the VPC network.

Open ports

External TCP proxy load balancers are reverse proxy load balancers. The load balancer terminates incoming connections and then opens new connections from the load balancer to the backends. These load balancers are implemented by using GFE proxies worldwide.

GFEs have several open ports to support other Google services that run on the same architecture. To see a list of some of the ports likely to be open on GFEs, see Forwarding rule: Port specifications. There might be other open ports for other Google services running on GFEs.

Running a port scan on the IP address of a GFE-based load balancer is not useful from an auditing perspective for the following reasons:

  • A port scan (for example, with nmap) generally expects no response packet or a TCP RST packet when performing TCP SYN probing. GFEs will send SYN-ACK packets in response to SYN probes only for ports on which you have configured a forwarding rule and on ports 80 and 443 if your load balancer uses a Premium Tier IP address. GFEs only send packets to your backends in response to packets sent to your load balancer's IP address and the destination port configured on its forwarding rule. Packets sent to different load balancer IP addresses or your load balancer's IP address on a port not configured in your forwarding rule do not result in packets being sent to your load balancer's backends.

  • Packets sent to the IP address of your load balancer could be answered by any GFE in Google's fleet; however, scanning a load balancer IP address and destination port combination only interrogates a single GFE per TCP connection. The IP address of your load balancer is not assigned to a single device or system. Thus, scanning the IP address of a GFE-based load balancer does not scan all the GFEs in Google's fleet.

With that in mind, the following are some more effective ways to audit the security of your backend instances:

  • A security auditor should inspect the forwarding rules configuration for the load balancer's configuration. The forwarding rules define the destination port for which your load balancer accepts packets and forwards them to the backends. For GFE-based load balancers, each external forwarding rule can only reference a single destination TCP port.

  • A security auditor should inspect the firewall rule configuration applicable to backend VMs. The firewall rules that you set block traffic from the GFEs to the backend VMs, but do not block incoming traffic to the GFEs. For best practices, see the firewall rules section.

Shared VPC architecture

External TCP proxy load balancers and external regional TCP proxy load balancers support networks that use Shared VPC. Shared VPC lets you maintain a clear separation of responsibilities between network administrators and service developers. Your development teams can focus on building services in service projects, and the network infrastructure teams can provision and administer load balancing. If you're not already familiar with Shared VPC, see the Shared VPC overview.

IP address Forwarding rule Target proxy and URL map Backend components
An external IP address must be defined in the same project as the load balancer. The external forwarding rule must be defined in the same project as the backend instances (the service project). The target TCP proxy must be defined in the same project as the backend instances.

For external TCP proxy load balancers, a global backend service must be defined in the same project as the backend instances. These instances must be in instance groups attached to the backend service as backends. Health checks associated with backend services must be defined in the same project as the backend service.

For external regional TCP proxy load balancers, the backend VMs are typically located in a service project. A regional internal backend service and health check must be defined in that service project.

Traffic distribution

When you add a backend instance group or NEG to a backend service, you specify a balancing mode, which defines a method that measures the backend load and target capacity. How traffic is distributed among backends depends on the mode of the load balancer.

External TCP proxy load balancer

The way an external TCP proxy load balancer distributes traffic to its backends depends on the balancing mode and the hashing method selected to choose a backend (session affinity).

How connections are distributed

An external TCP proxy load balancer can be configured as a global load balancing service in Premium Tier and as a regional service in Standard Tier.

For Premium Tier:

  • Google advertises your load balancer's IP address from all points of presence, worldwide. Each load balancer IP address is global anycast.
  • If you configure a backend service with backends in multiple regions, Google Front Ends (GFEs) attempt to direct requests to healthy backend instance groups or NEGs in the region closest to the user. Details for the process are documented on this page.

For Standard Tier:

  • Google advertises your load balancer's IP address from points of presence associated with the forwarding rule's region. The load balancer uses a regional external IP address.

  • You can configure backends in the same region as the forwarding rule. The process documented here still applies, but the load balancer only directs requests to healthy backends in that one region.

Request distribution process:

  1. The forwarding rule's external IP address is advertised by edge routers at the borders of Google's network. Each advertisement lists a next hop to a Layer 3/4 load balancing system (Maglev).
  2. The Maglev systems route traffic to a first-layer Google Front End (GFE). The first-layer GFE terminates TLS if required and then routes traffic to second-layer GFEs according to this process:
    1. If a backend service uses instance group or GCE_VM_IP_PORT NEG backends, the first layer-GFEs prefer second-layer GFEs that are located in or near the region that contains the instance group or NEG.
    2. For backend buckets and backend services with hybrid NEGs, serverless NEGs, and internet NEGs, the first-layer GFEs choose second-layer GFEs in a subset of regions such that the round trip time between the two GFEs is minimized.

      Second-layer GFE preference is not a guarantee, and it can dynamically change based on Google's network conditions and maintenance.

      Second-layer GFEs are aware of health check status and actual backend capacity usage.

  3. The second-layer GFE directs requests to backends in zones within its region.
  4. For Premium Tier, sometimes second-layer GFEs send requests to backends in zones of different regions. This behavior is called spillover.

    5. Spillover is governed by two principles:

    • Spillover is possible when all backends known to a second-layer GFE are at capacity or are unhealthy.
    • The second-layer GFE has information for healthy, available backends in zones of a different region.

    The second-layer GFEs are typically configured to serve a subset of backend locations.

    Spillover behavior does not exhaust all possible Google Cloud zones. If you need to direct traffic away from backends in a particular zone or in an entire region, you must set the capacity scaler to zero. Configuring backends to fail health checks does not guarantee that the second-layer GFE spills over to backends in zones of a different region.

  5. When distributing requests to backends, GFEs operate at a zonal level.

Balancing mode

When you add a backend to the backend service, you set a load balancing mode.

For external TCP proxy load balancers, the balancing mode can be CONNECTION or UTILIZATION:

  • If the load balancing mode is CONNECTION, the load is spread based on how many concurrent connections the backend can handle. You must also specify exactly one of the following parameters: maxConnections (except for regional managed instance groups), maxConnectionsPerInstance, or maxConnectionsPerEndpoint.

  • If the load balancing mode is UTILIZATION, the load is spread based on the utilization of instances in an instance group.

For information about comparing the load balancer types and the supported balancing modes, see Load balancing methods.

External regional TCP proxy load balancer

The way an external regional TCP proxy load balancer distributes traffic to its backends depends on the balancing mode and the hashing method selected to choose a backend (session affinity).

An external regional TCP proxy load balancer distributes traffic to its backends as follows:

  • Connections that originate from a single client are sent to the same zone as long as healthy backends (instance groups or NEGs) within that zone are available and have capacity, as determined by the balancing mode. For external regional TCP proxy load balancers, the balancing mode can be CONNECTION (instance group or NEG backends) or UTILIZATION (instance group backends only).
  • Connections from a client are sent to the same backend if you have configured session affinity.

Session affinity

Session affinity sends all requests from the same client to the same backend if the backend is healthy and has capacity.

External TCP proxy load balancers and external regional TCP proxy load balancers offer the following types of session affinity:

  • NONE. Session affinity is not set for the load balancer.
  • Client IP affinity, which forwards all requests from the same client IP address to the same backend.

Failover

If a backend becomes unhealthy, traffic is automatically redirected to healthy backends within the same region. If all backends within a region are unhealthy, traffic is distributed to healthy backends in other regions (Premium Tier only). If all backends are unhealthy, the load balancer drops traffic.

Load balancing for GKE applications

If you are building applications in Google Kubernetes Engine, you can use standalone NEGs to load balance traffic directly to containers. With standalone NEGs, you are responsible for creating the Service object that creates the NEG, and then associating the NEG with the backend service so that the load balancer can connect to the Pods.

For related documentation, see Container-native load balancing through standalone zonal NEGs.

What's next