Set up an internal regional TCP proxy load balancer with hybrid connectivity

Stay organized with collections Save and categorize content based on your preferences.

The internal regional TCP proxy load balancer is a proxy-based regional Layer 4 load balancer that enables you to run and scale your TCP service traffic behind an internal IP address that is accessible only to clients in the same VPC network or clients connected to your VPC network.

This page describes how to configure an internal regional TCP proxy load balancer to load-balance traffic to backends on-premises or on other cloud environments that are connected using hybrid connectivity. Note that configuring hybrid connectivity to connect your networks to Google Cloud is not in scope for this topic.

Overview

In this example, we'll use the load balancer to distribute TCP traffic across backend VMs located on-premises or in other cloud environments.

In this example, you configure the following deployment:

Internal regional TCP proxy load balancer example configuration with hybrid NEG backends.
Internal regional TCP proxy load balancer example configuration with hybrid NEG backends

The internal regional TCP proxy load balancer is a regional load balancer. All load balancer components (backend instance groups, backend service, target proxy, and forwarding rule) must be in the same region.

Permissions

You must have the following permissions to set up hybrid load balancing:

  • On Google Cloud

    • Permission to establish hybrid connectivity between Google Cloud and your on-premises or other cloud environments the environments. For the list of permissions needed, see the relevant Network connectivity product documentation.
    • Additionally, to follow the instructions on this page, you need permissions to create a hybrid connectivity NEG and the load balancer. The Compute Load Balancer Admin role (roles/compute.loadBalancerAdmin) contains the permission required to perform the tasks described in this guide.
  • On your on-premises or other non-Google Cloud cloud environment

    • Permission to configure network endpoints that allow services on your on-premises or other cloud environments to be reachable from Google Cloud via an IP:Port combination. Contact your environment's network administrator for details.
    • Permission to create firewall rules on your on-premises or other cloud environments to allow Google's health check probes to reach the endpoints.

Set up hybrid connectivity

Your Google Cloud and on-premises or other cloud environments must be connected through hybrid connectivity, using either Cloud Interconnect VLAN attachments or Cloud VPN tunnels with Cloud Router. We recommend you use a high availability connection.

A Cloud Router enabled with Global dynamic routing learns about the specific endpoint via BGP and programs it into your Google Cloud VPC network. Regional dynamic routing is not supported. Static routes are also not supported.

The Google Cloud VPC network that you use to configure either Cloud Interconnect or Cloud VPN is the same network you use to configure the internal regional TCP proxy load balancer. Ensure that your VPC network's subnet CIDR ranges do not conflict with your remote CIDR ranges. When IP addresses overlap, subnet routes are prioritized over remote connectivity.

For instructions, see:

Set up your environment that is outside Google Cloud

Perform the following steps to set up your on-premises or other cloud environment for hybrid load balancing:

  • Configure network endpoints to expose on-premises services to Google Cloud (IP:Port).
  • Configure firewall rules on your on-premises or other cloud environment.
  • Configure Cloud Router to advertise certain required routes to your private environment.

Set up network endpoints

After you have set up hybrid connectivity, you configure one or more network endpoints within your on-premises or other cloud environments that are reachable with Cloud Interconnect or Cloud VPN using an IP:port combination. This IP:port combination will be configured as one or more endpoints for the hybrid connectivity NEG that will be created in Google Cloud later on in this process.

If there are multiple paths to the IP endpoint, routing will follow the behavior described in the Cloud Router overview.

Set up firewall rules

The following firewall rules must be created on your on-premises or other cloud environment:

  • Ingress allow firewall rules to allow traffic from Google's health-checking probes to your endpoints. The ranges to allow are: 35.191.0.0/16 and 130.211.0.0/22. For more information, see Probe IP ranges and firewall rules.
  • Ingress allow firewall rules to allow traffic that is being load-balanced to reach the endpoints.
  • Ingress allow firewall rule to allow traffic from the Google Cloud region's proxy-only subnet to reach the endpoints.

Advertise routes

Configure Cloud Router to advertise the following routes to your on-premises or other cloud environment:

Set up your Google Cloud environment

For the following steps, make sure you use the same VPC network (called NETWORK in this procedure) that was used to configure hybrid connectivity between the environments.

Additionally, make sure the region used (called REGION in this procedure) is the same as that used to create the Cloud VPN tunnel or Cloud Interconnect VLAN attachment.

Configure the proxy-only subnet

A proxy-only subnet provides a set of IP addresses that Google uses to run Envoy proxies on your behalf. The proxies terminate connections from the client and create new connections to the backends.

The proxy-only subnet is used by all Envoy-based regional load balancers in the REGION region of the NETWORK VPC network.

There can only be one active proxy-only subnet per region, per VPC network. You can skip this step if there's already a proxy-only subnet in this region.

Console

If you're using the Google Cloud console, you can wait and create the proxy-only subnet later on the Load balancing page.

If you want to create the proxy-only subnet now, use the following steps:

  1. In the Google Cloud console, go to the VPC networks page.
    Go to VPC networks
  2. Go to the network that was used to configure hybrid connectivity between the environments.
  3. Click Add subnet.
  4. Enter a Name: PROXY_ONLY_SUBNET_NAME.
  5. Select a Region: REGION.
  6. Set Purpose to Regional Managed Proxy.
  7. Enter an IP address range: PROXY_ONLY_SUBNET_RANGE.
  8. Click Add.

gcloud

Create the proxy-only subnet with the gcloud compute networks subnets create command.

gcloud compute networks subnets create PROXY_ONLY_SUBNET_NAME \
    --purpose=REGIONAL_MANAGED_PROXY \
    --role=ACTIVE \
    --region=REGION \
    --network=NETWORK \
    --range=PROXY_ONLY_SUBNET_RANGE

Reserve the load balancer's IP address

Console

You can reserve a standalone internal IP address using the Google Cloud console.

  1. In the Google Cloud console, go to the VPC networks page.

    Go to VPC networks

  2. Click the network that was used to configure hybrid connectivity between the environments. In this case, NETWORK.
  3. Click Static internal IP addresses and then click Reserve static address.
  4. Enter a Name: LB_IP_ADDRESS.
  5. For the Subnet, select LB_SUBNET.
  6. If you want to specify which IP address to reserve, under Static IP address, select Let me choose, and then fill in a Custom IP address. Otherwise, the system automatically assigns an IP address in the subnet for you.
  7. If you want to use this IP address with multiple forwarding rules, under Purpose, choose Shared.
  8. Click Reserve to finish the process.

gcloud

Reserve a regional internal IP address for the load balancer's forwarding rule.

gcloud compute addresses create LB_IP_ADDRESS \
    --region=REGION \
    --subnet=LB_SUBNET \
    --purpose=SHARED_LOADBALANCER_VIP

Create firewall rules

In this example, you create the following firewall rules:

  • fw-allow-health-check: An ingress rule, applicable to the instances being load balanced, that allows traffic from the load balancer and Google Cloud health checking systems (130.211.0.0/22 and 35.191.0.0/16). This example uses the target tag allow-health-check to identify the backend VMs to which it should apply.
  • fw-allow-ssh: An ingress rule that allows incoming SSH connectivity on TCP port 22 from any address. You can choose a more restrictive source IP range for this rule; for example, you can specify just the IP ranges of the systems from which you will initiate SSH sessions. This example uses the target tag allow-ssh to identify the VMs to which it should apply.
  • fw-allow-proxy-only-subnet: An ingress rule that allows connections from the proxy-only subnet to reach the backends.

Console

  1. In the Google Cloud console, go to the Firewalls page.
    Go to Firewalls
  2. Click Create firewall rule to create the rule to allow traffic from health check probes:
    1. Enter a Name of fw-allow-health-check.
    2. Under Network, select the network that was used to configure hybrid connectivity between the environments. In this case, NETWORK.
    3. Under Targets, select Specified target tags.
    4. Populate the Target tags field with allow-health-check.
    5. Set Source filter to IPv4 ranges.
    6. Set Source IPv4 ranges to 130.211.0.0/22 and 35.191.0.0/16.
    7. Under Protocols and ports, select Specified protocols and ports.
    8. Select the checkbox next to tcp and then enter 80 for the port number.
    9. Click Create.
  3. Click Create firewall rule again to create the rule to allow incoming SSH connections:
    1. Name: fw-allow-ssh
    2. Network: NETWORK
    3. Priority: 1000
    4. Direction of traffic: ingress
    5. Action on match: allow
    6. Targets: Specified target tags
    7. Target tags: allow-ssh
    8. Source filter: IPv4 ranges
    9. Source IPv4 ranges: 0.0.0.0/0
    10. Protocols and ports: Choose Specified protocols and ports, and then enter tcp:22.
    11. Click Create.
  4. Click Create firewall rule again to create the rule to allow incoming connections from the proxy-only subnet:
    1. Name: fw-allow-ssh
    2. Network: NETWORK
    3. Priority: 1000
    4. Direction of traffic: ingress
    5. Action on match: allow
    6. Targets: Specified target tags
    7. Target tags: allow-proxy-only-subnet
    8. Source filter: IPv4 ranges
    9. Source IPv4 ranges: PROXY_ONLY_SUBNET_RANGE
    10. Protocols and ports: Choose Specified protocols and ports, and then enter tcp:80.
    11. Click Create.

gcloud

  1. Create the fw-allow-health-check-and-proxy rule to allow the Google Cloud health checks to reach the backend instances on TCP port 80:

    gcloud compute firewall-rules create fw-allow-health-check \
        --network=NETWORK \
        --action=allow \
        --direction=ingress \
        --target-tags=allow-health-check \
        --source-ranges=130.211.0.0/22,35.191.0.0/16 \
        --rules=tcp:80
    
  2. Create the fw-allow-ssh firewall rule to allow SSH connectivity to VMs with the network tag allow-ssh.

    gcloud compute firewall-rules create fw-allow-ssh \
        --network=NETWORK \
        --action=allow \
        --direction=ingress \
        --target-tags=allow-ssh \
        --rules=tcp:22
    
  3. Create an ingress allow firewall rule for the proxy-only subnet to allow the load balancer to communicate with backend instances on TCP port 80:

    gcloud compute firewall-rules create fw-allow-proxy-only-subnet \
        --network=NETWORK \
        --action=allow \
        --direction=ingress \
        --target-tags=allow-proxy-only-subnet \
        --source-ranges=PROXY_ONLY_SUBNET_RANGE \
        --rules=tcp:80
    

Set up the hybrid connectivity NEG

When creating the NEG, use a ZONE that minimizes the geographic distance between Google Cloud and your on-premises or other cloud environment. For example, if you are hosting a service in an on-premises environment in Frankfurt, Germany, you can specify the europe-west3-a Google Cloud zone when you create the NEG.

Moreover, the ZONE used to create the NEG should be in the same region where the Cloud VPN tunnel or Cloud Interconnect VLAN attachment were configured for hybrid connectivity.

For the available regions and zones, see the Compute Engine documentation: Available regions and zones.

Console

gcloud

  1. Create a hybrid connectivity NEG using the gcloud compute network-endpoint-groups create command.

    gcloud compute network-endpoint-groups create HYBRID_NEG_NAME \
       --network-endpoint-type=NON_GCP_PRIVATE_IP_PORT \
       --zone=HYBRID_NEG_ZONE \
       --network=NETWORK
    
  2. Add the on-premises IP:Port endpoint to the hybrid NEG:

    gcloud compute network-endpoint-groups update HYBRID_NEG_NAME \
        --zone=HYBRID_NEG_ZONE \
        --add-endpoint="ip=ENDPOINT_IP_ADDRESS,port=ENDPOINT_PORT"
    

You can use this command to add the network endpoints you previously configured on-premises or in your cloud environment. Repeat --add-endpoint as many times as needed.

You can repeat these steps to create multiple hybrid NEGs if needed.

Configure the load balancer

Console

gcloud

  1. Create a regional health check for the backends.

    gcloud compute health-checks create tcp TCP_HEALTH_CHECK_NAME \
        --region=REGION \
        --use-serving-port
    
  2. Create a backend service.

    gcloud beta compute backend-services create BACKEND_SERVICE_NAME \
       --load-balancing-scheme=INTERNAL_MANAGED \
       --protocol=TCP \
       --region=REGION \
       --health-checks=TCP_HEALTH_CHECK_NAME \
       --health-checks-region=REGION
    
  3. Add the hybrid NEG backend to the backend service.

    gcloud beta compute backend-services add-backend BACKEND_SERVICE_NAME \
       --network-endpoint-group=HYBRID_NEG_NAME \
       --network-endpoint-group-zone=HYBRID_NEG_ZONE \
       --region=REGION \
       --balancing-mode=CONNECTION \
       --max-connections=MAX_CONNECTIONS
    

    For MAX_CONNECTIONS, enter the maximum concurrent connections that the backend should handle.

  4. Create the target TCP proxy.

    gcloud beta compute target-tcp-proxies create TARGET_TCP_PROXY_NAME \
       --backend-service=BACKEND_SERVICE_NAME \
       --region=REGION
    
  5. Create the forwarding rule.

    Create the forwarding rule using the gcloud beta compute forwarding-rules create command.

    Replace FWD_RULE_PORT with a single port number from 1-65535. The forwarding rule only forwards packets with a matching destination port.

    gcloud beta compute forwarding-rules create FORWARDING_RULE \
       --load-balancing-scheme=INTERNAL_MANAGED \
       --network=NETWORK \
       --subnet=LB_SUBNET \
       --address=LB_IP_ADDRESS \
       --ports=FWD_RULE_PORT \
       --region=REGION \
       --target-tcp-proxy=TARGET_TCP_PROXY_NAME \
       --target-tcp-proxy-region=REGION
    

Test the load balancer

To test the load balancer, create a client VM in the same region as the load balancer. Then send traffic from the client to the load balancer.

Create a client VM

Create a client VM (client-vm) in the same region as the load balancer.

Console

  1. In the Google Cloud console, go to the VM instances page.
    Go to VM instances
  2. Click Create instance.
  3. Set the Name to client-vm.
  4. Set the Zone to CLIENT_VM_ZONE.
  5. Click Management, security, disks, networking, sole tenancy and make the following changes:
    • Click Networking and add the allow-ssh to Network tags.
    • Click the edit button under Network interfaces and make the following changes then click Done:
      • Network: NETWORK
      • Subnet: LB_SUBNET
      • Primary internal IP: Ephemeral (automatic)
      • External IP: Ephemeral
  6. Click Create.

gcloud

The client VM must be in the same VPC network and region as the load balancer. It doesn't need to be in the same subnet or zone. The client uses the same subnet as the backend VMs.

gcloud compute instances create client-vm \
    --zone=CLIENT_VM_ZONE \
    --image-family=debian-10 \
    --image-project=debian-cloud \
    --tags=allow-ssh \
    --subnet=LB_SUBNET

Send traffic to the load balancer

Now that you have configured your load balancer, you can test sending traffic to the load balancer's IP address.

  1. Connect via SSH to the client instance.

    gcloud compute ssh client-vm \
      --zone=CLIENT_VM_ZONE
    
  2. Verify that the load balancer is serving backend hostnames as expected.

    1. Use the compute addresses describe command to view the load balancer's IP address:

      gcloud compute addresses describe LB_IP_ADDRESS \
        --region=REGION
      

      Make a note of the IP address.

    2. Send traffic to the load balancer on the IP address and port specified when creating the load balancer forwarding rule. Testing whether the hybrid NEG backends are responding to requests depends on the service running on the non-Google Cloud endpoints.

What's next