Proxy-only subnets for Envoy-based load balancers

This page describes how to work with proxy-only subnets used by Envoy-based load balancers. A proxy-only subnet provides a pool of IP addresses that are reserved exclusively for Envoy proxies used by Google Cloud load balancers. It cannot be used for any other purposes.

The proxies terminate incoming connections and then evaluate where each request should go based on the URL map, the backend service's session affinity, the balancing mode of each backend instance group or NEG, and other factors.

1. A client makes a connection to the IP address and port of the load balancer's forwarding rule.

2. Each proxy listens on the IP address and port specified by the corresponding load balancer's forwarding rule. One of the proxies receives and terminates the client's network connection.

3. The proxy establishes a connection to the appropriate backend VM or endpoint in a NEG, as determined by the load balancer's URL map and backend services.

Each of the load balancer's proxies is assigned an internal IP address. Packets sent from a proxy to a backend VM or endpoint has a source IP address from the proxy-only subnet.

The proxy-only subnet cannot be used for any other purpose. The IP address for the load balancer's forwarding rule doesn't come from the proxy-only subnet. Also, the IP addresses of the backend VMs and endpoints don't come from the proxy-only subnet.

Supported load balancers

The following Envoy-based load balancers require proxy-only subnets:

• Proxy-only subnet with purpose GLOBAL_MANAGED_PROXY:

  • Cross-region internal Application Load Balancer
  • Cross-region internal proxy Network Load Balancer

  In a given network and region, a single proxy-only subnet of purpose GLOBAL_MANAGED_PROXY is shared between all the cross-region internal Application Load Balancers in that region. At any point, only one subnet with purpose GLOBAL_MANAGED_PROXY can be active in each region of a VPC network.

• Proxy-only subnet with purpose REGIONAL_MANAGED_PROXY:

  • Regional external Application Load Balancer
  • Regional internal Application Load Balancer
  • Regional external proxy Network Load Balancer
  • Regional internal proxy Network Load Balancer

  In a given network and region, a single proxy-only subnet of purpose REGIONAL_MANAGED_PROXY is shared between all the regional Envoy-based load balancers. At any point, only one subnet with purpose REGIONAL_MANAGED_PROXY can be active in each region of a VPC network.

Note:

If the client sends packets with the port explicitly specified, Envoy-based load balancers don't remove the port number by default. For example, traffic with example.net:443 are first forwarded to an Envoy-based load balancing service and then routed to the backend by the URL map.

If there is no matching routing rule in the URL map, the traffic is routed to the default backend. You can specify URL map routing rules to specify the port number by setting the hostname as example.net:443 and path /*. For more information, see URL Map components.

gRPC library for golang mandates addition of port at the end of the hostname, so if a request is initiated to the load balancer, ensure that we have a matching host rule in the load balancer configuration.

How proxy-only subnets fit into the load balancer's architecture

The following diagram shows the Google Cloud resources required for a regional internal Application Load Balancer.

Regional internal Application Load Balancer numbered components (click to enlarge).

As shown in the diagrams, an Envoy-based load balancer deployment requires at least two subnets:

• The load balancer's backend VMs and backend endpoints use a single subnet whose primary IP address range is 10.1.2.0/24 (in this example). This subnet isn't the proxy-only subnet. You can use multiple subnets for your backend VMs and endpoints if the subnets are in the same region as the load balancer. For internal Application Load Balancers, the load balancer's IP address associated with the forwarding rule can also be in this subnet (but does not need to be).
• The proxy-only subnet is 10.129.0.0/23 (in this example).

Proxy-only subnet size recommendations

A proxy-only subnet must provide 64 or more IP addresses. This corresponds to a prefix length of /26 or shorter. We recommend that you start with a proxy-only subnet with a /23 prefix (512 proxy-only addresses) and change the size as your traffic needs change.

Note: You can use the non-RFC 1918 address range for the proxy-only subnet. This can be helpful if you have insufficient available addresses in your RFC 1918 address range.

Proxies are allocated at the VPC level, not the load balancer level. You must create one proxy-only subnet in each region of a VPC network in which you use Envoy-based load balancers. If you deploy multiple load balancers in the same region and same VPC network, they share the same proxy-only subnet for load balancing. Envoy-based load balancers automatically scale the number of proxies available to handle your traffic based on your traffic needs.

Each additional proxy incurs an additional hourly charge. To learn about how proxy-only subnets are billed, see Proxy instance charge in the Cloud Load Balancing pricing documentation.

Create a proxy-only subnet

You must create proxy-only subnets for Envoy-based load balancers regardless of whether your network is auto-mode or custom. Creating a proxy-only subnet is essentially the same procedure as creating any subnet, except with the addition of some flags.

For a proxy-only subnet, the --purpose must be set to either REGIONAL_MANAGED_PROXY or GLOBAL_MANAGED_PROXY, depending on your load balancer.

You can't reuse an existing subnet as the proxy-only subnet; you must create a new subnet in each region that has an Envoy-based load balancer. This is in part because the subnets update command doesn't allow modifying a subnet's --purpose field.

Note: For regional internal Application Load Balancers, set the purpose of the subnet to --purpose=REGIONAL_MANAGED_PROXY. If you previously created a proxy-only subnet with --purpose=INTERNAL_HTTPS_LOAD_BALANCER, you will need to migrate the subnet's purpose to REGIONAL_MANAGED_PROXY before you can create other Envoy-based load balancers in the same region of the VPC network.

You must create a proxy-only subnet for use by the load balancers' proxies, before creating forwarding rules for your regional load balancers. If you try to configure a load balancer without first creating a proxy-only subnet for the region, the load balancer creation process fails.

Console

1. In the Google Cloud console, go to the VPC networks page.
   Go to the VPC networks page
2. Click the name of the Shared VPC network that you want to add a proxy-only subnet to.
3. Click Add subnet.
4. Enter a Name.
5. Select a Region.
6. Set Purpose to one of the following:
   • For regional load balancers: Regional Managed Proxy
   • For cross-region load balancers: Cross-region Managed Proxy
   • Enter an IP address range.
   • Click Add.

gcloud

The gcloud compute networks subnets create command creates a proxy-only subnet.

gcloud compute networks subnets create SUBNET_NAME \
    --purpose=SUBNET_PURPOSE \
    --role=ACTIVE \
    --region=REGION \
    --network=VPC_NETWORK_NAME \
    --range=CIDR_RANGE

The fields are defined as follows:

• SUBNET_NAME is the name of the proxy-only subnet.
• SUBNET_PURPOSE is the purpose of the subnet. Set this to either REGIONAL_MANAGED_PROXY or GLOBAL_MANAGED_PROXY, depending on your load balancer.
• REGION is the region of the proxy-only subnet.
• VPC_NETWORK_NAME is the name of the VPC network that contains the subnet.
• CIDR_RANGE is the primary IP address range of the subnet. You must use a subnet mask no longer than 26 so that at least 64 IP addresses are available for proxies in the region. The recommended subnet mask length is /23.

For a complete configuration example, refer to Configuring the proxy-only subnet.

You must configure a firewall rule for your backends to accept connections from the proxy-only subnet. For a complete configuration example, including firewall rule setup, see the following:

• Set up a regional external Application Load Balancer
• Set up a regional internal Application Load Balancer
• Set up a regional internal proxy Network Load Balancer
• Set up a regional external proxy Network Load Balancer
• Set up a cross-region internal Application Load Balancer
• Set up a cross-region internal proxy Network Load Balancer

Proxy availability

Sometimes Google Cloud regions don't have enough proxy capacity for a new load balancer. If this happens, the Google Cloud console provides a proxy availability warning message when you are creating your load balancer. To resolve this issue, you can do one of the following:

• Select a different region for your load balancer. This can be a practical option if you have backends in another region.
• Select a VPC network that already has an allocated proxy-only subnet.
• Wait for the capacity issue to be resolved.

Change the size or address range of a proxy-only subnet

When the amount of traffic handled by your load balancer increases, you might need to increase the size of your proxy-only subnet to allow a larger number of Envoy proxies to be assigned to your load balancer.

You can't expand a proxy-only subnet in the same way that you would for a primary address range (with the expand-ip-range command). Instead, you must create a backup proxy-only subnet that meets your needs and then promote it to the active role. This is because only one proxy-only subnet can be active per region and per VPC network and because you can only expand a subnet's primary IP address range.

Switching a proxy-only subnet from backup to active does not interrupt new connections:

• The newly activated proxy-only subnet is used for new connections.
• The previously active (now backup) proxy-only subnet is no longer used for new connections.
• Google Cloud begins draining existing connections from proxies in the previously active (now backup) proxy-only subnet.

You can create only one active and one backup proxy-only subnet per region, per VPC network.

Note: The backup proxy-only subnet doesn't provide for failover.

Console

1. Create a backup proxy-only subnet in the same region, specifying a primary IP address range that meets your needs

   1. In the Google Cloud console, go to the VPC networks page.
      Go to the VPC networks page
   2. Click the name of the Shared VPC network that you want to add a proxy-only subnet to.
   3. Click Add subnet.
   4. Enter a Name.
   5. Select a Region.
   6. Set Purpose to one of the following:
      • For regional load balancers: Regional Managed Proxy
      • For cross-region load balancers: Cross-region Managed Proxy
      • For the Role, select Backup.
      • Enter an IP address range.
      • Click Add.

2. Create or modify ingress allow firewall rules that apply to your backend VMs or endpoints so that they include the primary IP address range of the backup proxy-only subnet.

3. Promote the backup proxy-only subnet to the active role. This action also demotes the previously active proxy-only subnet to the backup role:

   1. In the Google Cloud console, go to the VPC networks page.
      Go to the VPC networks page
   2. Click the name of the Shared VPC network that you want to modify.
   3. Under Reserved proxy-only subnets for load balancing, locate the backup subnet created in the previous step.
   4. Click Activate.
   5. Specify an optional Drain timeout.
   6. Click Activate the subnet.

4. After the connection draining timeout, or after you're confident that connections to your backend VMs or endpoints aren't coming from proxies in the previously active (now backup) proxy-only subnet, you can do the following:

   • Modify ingress allow firewall rules that apply to your backend VMs or endpoints so they don't include the primary IP address range of the previously active (now backup) proxy-only subnet.
   • Delete the previously active (now backup) proxy-only subnet to release the IP addresses that the subnet used for its primary IP address range.

gcloud

The following steps assume you already have an existing active proxy-only subnet.

1. Create a backup proxy-only subnet in the same region, specifying a primary IP address range that meets your needs, using the gcloud compute networks subnets create command with the --role=BACKUP flag.

   gcloud compute networks subnets create BACKUP_PROXY_ONLY_SUBNET_NAME \
      --purpose=SUBNET_PURPOSE \
      --role=BACKUP \
      --region=REGION \
      --network=VPC_NETWORK_NAME \
      --range=BACKUP_PROXY_ONLY_SUBNET_RANGE
   

   Replace the following:

   • BACKUP_PROXY_ONLY_SUBNET_NAME: the name of the newly created backup proxy-only subnet
   • SUBNET_PURPOSE: the purpose of the newly created backup proxy-only subnet
   • REGION: the region of the newly created backup proxy-only subnet. This should be the same region as the current active proxy-only subnet.
   • REGION: the network of the newly created backup proxy-only subnet. This should be the same network as the current active proxy-only subnet.
   • BACKUP_PROXY_ONLY_SUBNET_RANGE: the CIDR range of the newly created backup proxy-only subnet

2. Create or modify ingress allow firewall rules that apply to your backend VMs or endpoints so that they now include the primary IP address range of the backup proxy-only subnet. The firewall rule should already be accepting connections from the active subnet.

   gcloud compute firewall-rules update PROXY_ONLY_SUBNET_FIREWALL_RULE \
     --source-ranges ACTIVE_PROXY_ONLY_SUBNET_RANGE,BACKUP_PROXY_ONLY_SUBNET_RANGE
   

   Replace the following:

   • PROXY_ONLY_SUBNET_FIREWALL_RULE: the name of the firewall rule that allows traffic from the proxy-only subnet to reach your backend instances or endpoints
   • ACTIVE_PROXY_ONLY_SUBNET_RANGE: the CIDR range of the current active proxy-only subnet

3. Update the new subnet, setting it to be the ACTIVE proxy-only subnet in the region and wait for the old subnet to drain. This also demotes the previously active proxy-only subnet to the backup role.

   To drain an IP address range immediately, set the --drain-timeout to 0s. This promptly ends all connections to proxies that have assigned addresses in the subnet that is being drained.

   gcloud compute networks subnets update BACKUP_PROXY_ONLY_SUBNET_NAME \
      --region=REGION \
      --role=ACTIVE \
      --drain-timeout=CONNECTION_DRAINING_TIMEOUT
   

   Replace the following:

   • CONNECTION_DRAINING_TIMEOUT: the amount of time, in seconds, that Google Cloud uses to migrate existing connections away from proxies in the previously active proxy-only subnet

4. Monitor the status of the drain by using a list or describe command. The status of the subnet is DRAINING while it is being drained.

   gcloud compute networks subnets list
   

   Wait for draining to complete. When the old proxy-only subnet is drained, the status of the subnet is READY.

5. Update your proxy only subnet firewall rule to only allow connections from the new subnet.

   gcloud compute firewall-rules PROXY_ONLY_SUBNET_FIREWALL_RULE \
     --source-ranges BACKUP_PROXY_ONLY_SUBNET_RANGE
   

6. After you're confident that connections to your backend VMs or endpoints aren't coming from proxies in the previously active (now backup) proxy-only subnet, you can delete the old subnet.

   gcloud compute networks subnets delete ACTIVE_PROXY_ONLY_SUBNET_NAME \
     --region=REGION
   

Migrate the purpose of a proxy-only subnet

If you previously created a proxy-only subnet with --purpose=INTERNAL_HTTPS_LOAD_BALANCER, you will need to migrate the subnet's purpose to REGIONAL_MANAGED_PROXY before you can create other Envoy-based load balancers in the same region of the VPC network.

Console

If you're using the Google Cloud console to create the load balancer, you will be prompted to migrate the purpose of a previously created proxy-only subnet from --purpose=INTERNAL_HTTPS_LOAD_BALANCER to REGIONAL_MANAGED_PROXY while creating the load balancer.

gcloud

To change the purpose of an existing proxy-only subnet from --purpose=INTERNAL_HTTPS_LOAD_BALANCER to REGIONAL_MANAGED_PROXY, use the following command:

gcloud compute networks subnets update PROXY_ONLY_SUBNET \
    --purpose=REGIONAL_MANAGED_PROXY \
    --region=REGION

Migrating the purpose of a proxy-only subnet from --purpose=INTERNAL_HTTPS_LOAD_BALANCER to REGIONAL_MANAGED_PROXY does not cause any downtime. The change should take effect nearly instantly.

Delete a proxy-only subnet

Deleting a proxy-only subnet releases its primary IP address range so you can use the range for another purpose. Google Cloud enforces the following rules when it receives a request to delete a proxy-only subnet:

• An active proxy-only subnet cannot be deleted if there is at least one regional load balancer in the same region and VPC network.

• An active proxy-only subnet cannot be deleted if there is a backup proxy-only subnet in the same region and VPC network.

  If you try to delete an active proxy-only subnet before deleting the backup, the following error message appears: "Invalid resource usage: Cannot delete ACTIVE subnetwork because a BACKUP subnetwork exists."

Practically, these rules have the following effect:

• If no regional load balancer is defined in a given region and VPC network, you can delete the proxy-only subnets in that region. If a backup proxy-only subnet exists, you must first delete it before you can delete the active proxy-only subnet.

• If you have at least one regional load balancer defined in a given region and VPC network, you cannot delete the active proxy-only subnet; however, you can promote a backup proxy-only subnet to the active role, which automatically demotes the previously active proxy-only subnet to the backup role. After connections are drained, you can delete the backup (previously active) proxy-only subnet.

Refer to deleting subnets in the VPC network documentation for more information.

Limitations

The following constraints apply to proxy-only subnets:

• You can't have both an INTERNAL_HTTPS_LOAD_BALANCER and a REGIONAL_MANAGED_PROXY subnet in the same network and region, in the same way you can't have two REGIONAL_MANAGED_PROXY proxies or two INTERNAL_HTTPS_LOAD_BALANCER proxies.

• You can create only one active and one backup proxy-only subnet in each region in each VPC network.

• You cannot create a backup proxy-only subnet unless you have already created an active proxy-only subnet in that region and network.

• You can change the role of a proxy-only subnet from backup to active by updating the subnet. When you do that, Google Cloud automatically changes the previously active proxy-only subnet to backup. You cannot explicitly set the role of a proxy-only subnet to backup by updating it.

• During a proxy-only subnet's connection draining period (--drain-timeout), you cannot change the role of a proxy-only subnet from backup to active.

• Google Cloud doesn't warn you if your proxy-only subnet runs out of IP addresses. However, you can configure Monitoring to monitor the IP address usage of your proxy-only subnet. You can define alerting policies to set up an alert for the loadbalancing.googleapis.com/subnet/proxy_only/addresses metric.

• Proxy-only subnets don't support VPC Flow Logs.

What's next

• Set up a regional external Application Load Balancer with VM instance group backends
• Set up a regional external Application Load Balancer in a Shared VPC environment
• Set up a regional internal Application Load Balancer with VM instance group backends
• Set up a regional internal Application Load Balancer or cross-region internal Application Load Balancer in a Shared VPC environment
• Set up a regional internal proxy Network Load Balancer with VM instance group backends
• Set up a regional external proxy Network Load Balancer with VM instance group backends
• Set up a cross-region internal proxy Network Load Balancer with VM instance group backends