Setting up Internal TCP/UDP Load Balancing for third-party appliances

In Google Cloud, you can integrate third-party appliances in a highly available, scaled-out manner. To do this, you configure a custom static route and set its next hop to the Google Cloud internal TCP/UDP load balancer. This allows the load balancer to load balance traffic for a destination prefix to a pool of health-checked third-party VM appliances.

This guide uses an example to teach you how to configure a internal TCP/UDP load balancer to be a next hop. Before following this guide, familiarize yourself with the following:

Permissions

To follow this guide, you need to create instances and modify a network in a project. You should be either a project owner or editor, or you should have all of the following Compute Engine IAM roles:

Task Required role
Create networks, subnets, and load balancer components Network Admin
Add and remove firewall rules Security Admin
Create instances Compute Instance Admin

For more information, see the following guides:

Setting up internal TCP/UDP load balancers as next hops with common backends

This guide shows you how to use an internal TCP/UDP load balancer as the next hop for a custom static route in order to integrate scaled-out virtual appliances.

The solution discussed in this guide creates appliance VMs running Debian Linux and iptables. The example VMs do not perform any packet filtering, but you can add that functionality by modifying the iptables configuration of this example or by using different packet filtering or routing software.

The steps in this section describe how to configure the following resources:

  • Sample VPC networks and custom subnets
  • Google Cloud firewall rules that allow incoming connections to backend appliance virtual machines (VMs)
  • Custom static routes
  • Two client VMs to test connections
  • The following internal TCP/UDP load balancer components:
    • Backend VMs in a managed instance group (MIG)
    • A health check for the backend VMs
    • An internal backend service in the us-west1 region to manage connection distribution among the backend VMs
    • An internal forwarding rule and internal IP address for the frontend of the load balancer

This example shows load balancing to multiple backend NICs, as described in Load balancing to multiple NICs.

The topology looks like this:

Next-hop multi-NIC detailed example  for Internal TCP/UDP Load Balancing
Next-hop multi-NIC detailed example for Internal TCP/UDP Load Balancing (click to enlarge)

The diagram shows some of the resources that the example creates:

  • Application instances (in this case, VMs running iptables software) behind an internal TCP/UDP load balancer (fr-ilb1, in this example). The application instances only have internal IP addresses.
  • Each application instance has its can-ip-forward flag enabled. Without this flag, a Compute Engine VM can only transmit a packet if the source IP address of the packet matches the VM's internal IP address, an IP address from an alias IP range, or an IP address of a forwarding rule that resolves to the VM. The can-ip-forward flag changes this behavior so that the VM can transmit packets with any source IP address.
  • A custom static route with destination 10.50.1.0/24 and next hop set to the load balancer's forwarding rule, fr-ilb1.

The diagram also shows the traffic flow:

  • The testing VPC network has a custom static route for traffic that is destined to 10.50.1.0/24 subnet. This route directs the traffic to the load balancer.
  • The load balancer forwards traffic to one of the application instances based on the configured session affinity. (Session affinity only affects TCP traffic.)

For additional use cases, see Internal TCP/UDP load balancers as next hops.

Configuring the networks, region, and subnets

This example uses the following VPC networks, region, and subnets:

  • Networks: This example requires two networks, each with at least one subnet. Each backend third-party appliance VM must have at least two network interfaces, one in each VPC network. The networks in this example are custom mode VPC networks named testing and production. The testing network in this example contains the client and the load balancer. The production network contains the destination target VM.

  • Region: The subnets are located in the us-west1 region. The subnets must be in the same region because VM instances are zonal resources.

  • Subnets: The subnets, testing-subnet and production-subnet, use the 10.30.1.0/24 and 10.50.1.0/24 primary IP address ranges, respectively.

To create the example networks and subnets, follow these steps.

Console

Create the testing network and the testing-subnet:

  1. In the Google Cloud Console, go to the VPC networks page.

    Go to VPC networks

  2. Click Create VPC network.

  3. Enter a Name of testing.

  4. In the Subnets section:

    • Set the Subnet creation mode to Custom.
    • In the New subnet section, enter the following information:
      • Name: testing-subnet
      • Region: us-west1
      • IP address range: 10.30.1.0/24
      • Click Done.
  5. Click Create.

Create the production network and the production-subnet:

  1. In the Google Cloud Console, go to the VPC networks page.

    Go to VPC networks

  2. Click Create VPC network.

  3. Enter a Name of production.

  4. In the Subnets section:

    • Set the Subnet creation mode to Custom.
    • In the New subnet section, enter the following information:
      • Name: production-subnet
      • Region: us-west1
      • IP address range: 10.50.1.0/24
      • Click Done.
  5. Click Create.

gcloud

  1. Create the custom VPC networks:

    gcloud compute networks create testing --subnet-mode=custom
    
    gcloud compute networks create production --subnet-mode=custom
    
  2. Create subnets in the testing and production networks in the us-west1 region:

    gcloud compute networks subnets create testing-subnet \
        --network=testing \
        --range=10.30.1.0/24 \
        --region=us-west1
    
    gcloud compute networks subnets create production-subnet \
        --network=production \
        --range=10.50.1.0/24 \
        --region=us-west1
    

Configuring firewall rules

This example uses the following firewall rules:

  • fw-allow-testing-from-both: An ingress rule, applicable to all targets in the testing network. This rule allows traffic from sources in both the 10.30.1.0/24 and 10.50.1.0/24 IP address ranges. These two ranges cover the primary internal IP addresses of VMs in both networks.

  • fw-allow-production-from-both: An ingress rule, applicable to all targets in the production network. This rule allows traffic from sources in both the 10.30.1.0/24 and 10.50.1.0/24 IP address ranges. These two ranges cover the primary internal IP addresses of VMs in both networks.

  • fw-allow-testing-ssh: An ingress rule applied to the VM instances in the testing VPC network. This rule allows incoming SSH connectivity on TCP port 22 from any address. You can choose a more restrictive source IP range for this rule; for example, you can specify the IP ranges of the systems from which you plan to initiate SSH sessions. This example uses the target tag allow-ssh to identify the VMs to which the firewall rule applies.

  • fw-allow-production-ssh: An ingress rule applied to the VM instances in the production VPC network. This rule allows incoming SSH connectivity on TCP port 22 from any address. Like the fw-allow-testing-ssh rule, you can choose a more restrictive source IP range for this rule.

  • fw-allow-health-check: An ingress rule for the third-party appliance VMs that are being load balanced. This rule allows traffic from the Google Cloud health checking systems (130.211.0.0/22 and 35.191.0.0/16). This example uses the target tag allow-health-check to identify the instances to which it should apply.

  • fw-allow-production-health-check: An ingress rule for the third-party appliance VMs that are being load balanced. This rule allows traffic from the Google Cloud health checking systems (130.211.0.0/22 and 35.191.0.0/16). This example uses the target tag allow-health-check to identify the instances to which it should apply.

Without these firewall rules, the default deny ingress rule blocks incoming traffic to the backend instances. You must create a firewall rule to allow health checks from the IP ranges of Google Cloud probe systems. Refer to probe IP ranges for more information.

Console

  1. In the Google Cloud Console, go to the Firewall page.

    Go to Firewall

  2. Click Create firewall rule and enter the following information to create the rule to allow testing VMs to receive packets from the testing and the production subnets:

    • Name: fw-allow-testing-from-both
    • Network: testing
    • Priority: 1000
    • Direction of traffic: ingress
    • Action on match: allow
    • Targets: All instances in the network
    • Source filter: IP ranges
    • Source IP ranges: 10.30.1.0/24, 10.50.1.0/24
    • Protocols and ports: Allow all
  3. Click Create.

  4. Click Create firewall rule and enter the following information to create the rule to allow production VMs to receive packets from the testing and the production subnets:

    • Name: fw-allow-production-from-both
    • Network: production
    • Priority: 1000
    • Direction of traffic: ingress
    • Action on match: allow
    • Targets: All instances in the network
    • Source filter: IP ranges
    • Source IP ranges: 10.30.1.0/24, 10.50.1.0/24
    • Protocols and ports: Allow all
  5. Click Create.

  6. Click Create firewall rule to create the rule to allow incoming SSH connections in the testing environment:

    • Name: fw-allow-testing-ssh
    • Network: testing
    • Priority: 1000
    • Direction of traffic: ingress
    • Action on match: allow
    • Targets: Specified target tags
    • Target tags: allow-ssh
    • Source filter: IP ranges
    • Source IP ranges: 0.0.0.0/0
    • Protocols and ports: Choose Specified protocols and ports and type: tcp:22
  7. Click Create.

  8. Click Create firewall rule to create the rule to allow incoming SSH connections in the production environment:

    • Name: fw-allow-production-ssh
    • Network: production
    • Priority: 1000
    • Direction of traffic: ingress
    • Action on match: allow
    • Targets: Specified target tags
    • Target tags: allow-ssh
    • Source filter: IP ranges
    • Source IP ranges: 0.0.0.0/0
    • Protocols and ports: Choose Specified protocols and ports and type: tcp:22
  9. Click Create.

  10. Click Create firewall rule to create the rule to allow Google Cloud health checks in the testing environment:

    • Name: fw-allow-health-check
    • Network: testing
    • Priority: 1000
    • Direction of traffic: ingress
    • Action on match: allow
    • Targets: Specified target tags
    • Target tags: allow-health-check
    • Source filter: IP ranges
    • Source IP ranges: 130.211.0.0/22 and 35.191.0.0/16
    • Protocols and ports: tcp
  11. Click Create.

  12. Click Create firewall rule to create the rule to allow Google Cloud health checks in the production environment:

    • Name: fw-allow-production-health-check
    • Network: production
    • Priority: 1000
    • Direction of traffic: ingress
    • Action on match: allow
    • Targets: Specified target tags
    • Target tags: allow-health-check
    • Source filter: IP ranges
    • Source IP ranges: 130.211.0.0/22 and 35.191.0.0/16
    • Protocols and ports: tcp
  13. Click Create.

gcloud

  1. Create the fw-allow-testing-subnet firewall rule to allow testing VMs to receive packets from the testing and the production subnets:

    gcloud compute firewall-rules create fw-allow-testing-from-both \
        --network=testing \
        --action=allow \
        --direction=ingress \
        --source-ranges=10.30.1.0/24,10.50.1.0/24 \
        --rules=all
    
  2. Create the fw-allow-production-subnet firewall rule to allow production VMs to receive packets from the testing and the production subnets:

    gcloud compute firewall-rules create fw-allow-production-from-both \
        --network=production \
        --action=allow \
        --direction=ingress \
        --source-ranges=10.30.1.0/24,10.50.1.0/24 \
        --rules=all
    
  3. Create the fw-allow-testing-ssh firewall rule to allow SSH connectivity to VMs with the network tag allow-ssh. When you omit source-ranges, Google Cloud interprets the rule to mean any source.

    gcloud compute firewall-rules create fw-allow-testing-ssh \
        --network=testing \
        --action=allow \
        --direction=ingress \
        --target-tags=allow-ssh \
        --rules=tcp:22
    
  4. Create the fw-allow-production-ssh firewall rule to allow SSH connectivity to VMs with the network tag allow-ssh.

    gcloud compute firewall-rules create fw-allow-production-ssh \
        --network=production \
        --action=allow \
        --direction=ingress \
        --target-tags=allow-ssh \
        --rules=tcp:22
    
  5. Create the fw-allow-health-check rule to allow Google Cloud health checks to the third-party appliance VMs in the testing network.

    gcloud compute firewall-rules create fw-allow-testing-health-check \
        --network=testing \
        --action=allow \
        --direction=ingress \
        --target-tags=allow-health-check \
        --source-ranges=130.211.0.0/22,35.191.0.0/16 \
        --rules=tcp
    
  6. Create the fw-allow-production-health-check firewall rule to allow Google Cloud health checks to the third-party appliance VMs in the production network.

    gcloud compute firewall-rules create fw-allow-production-health-check \
        --network=production \
        --action=allow \
        --direction=ingress \
        --target-tags=allow-health-check \
        --source-ranges=130.211.0.0/22,35.191.0.0/16 \
        --rules=tcp
    

Creating the third-party virtual appliances

The following steps demonstrate how to create an instance template and managed regional instance group using the iptables software as a third-party virtual appliance.

Console

You must use gcloud for this step because you need to create an instance template with more than one network interface. The Cloud Console does not currently support creating instance templates with more than one network interface.

gcloud

  1. Create an instance template for your third-party virtual appliances. The instance template must include the --can-ip-forward flag so that the VM instances created from the template can forward packets from other instances in the testing and production networks.

    gcloud compute instance-templates create third-party-template-multinic \
        --region=us-west1 \
        --network-interface subnet=testing-subnet,address="" \
        --network-interface subnet=production-subnet \
        --tags=allow-ssh,allow-health-check \
        --image-family=debian-10 \
        --image-project=debian-cloud \
        --can-ip-forward \
        --metadata=startup-script='#! /bin/bash
        # Enable IP forwarding:
        echo 1 > /proc/sys/net/ipv4/ip_forward
        echo "net.ipv4.ip_forward=1" > /etc/sysctl.d/20-iptables.conf
        # Read VM network configuration:
        md_vm="http://169.254.169.254/computeMetadata/v1/instance/"
        md_net="$md_vm/network-interfaces"
        nic0_gw="$(curl $md_net/0/gateway -H "Metadata-Flavor:Google" )"
        nic0_mask="$(curl $md_net/0/subnetmask -H "Metadata-Flavor:Google")"
        nic0_addr="$(curl $md_net/0/ip -H "Metadata-Flavor:Google")"
        nic0_id="$(ip addr show | grep $nic0_addr | awk '{print $NF}')"
        nic1_gw="$(curl $md_net/1/gateway -H "Metadata-Flavor:Google")"
        nic1_mask="$(curl $md_net/1/subnetmask -H "Metadata-Flavor:Google")"
        nic1_addr="$(curl $md_net/1/ip -H "Metadata-Flavor:Google")"
        nic1_id="$(ip addr show | grep $nic1_addr | awk '{print $NF}')"
        # Source based policy routing for nic1
        echo "100 rt-nic1" >> /etc/iproute2/rt_tables
        ip rule add pri 32000 from $nic1_gw/$nic1_mask table rt-nic1
        sleep 1
        ip route add 35.191.0.0/16 via $nic1_gw dev $nic1_id table rt-nic1
        ip route add 130.211.0.0/22 via $nic1_gw dev $nic1_id table rt-nic1
        # Use a web server to pass the health check for this example.
        # You should use a more complete test in production.
        apt-get update
        apt-get install apache2 -y
        a2ensite default-ssl
        a2enmod ssl
        echo "Example web page to pass health check" | \
        tee /var/www/html/index.html
        systemctl restart apache2'
    
  2. Create a managed instance group for your third-party virtual appliances. This command creates a regional managed instance group, which can then be autoscaled, in us-west1.

    gcloud compute instance-groups managed create third-party-instance-group \
        --region=us-west1 \
        --template=third-party-template-multinic \
        --size=3
    

Creating the load balancing resources

These steps configure all of the internal TCP/UDP load balancer components starting with the health check and backend service, and then the frontend components:

  • Health check: In this example, the HTTP health check checks for an HTTP 200 (OK) response. For more information, see the health checks section of the Internal TCP/UDP Load Balancing overview.

  • Backend service: Even though this example's backend service specifies the TCP protocol, when the load balancer is the next hop for a route, Google Cloud forwards traffic for all protocols (TCP, UDP, and ICMP).

  • Forwarding rule: Even though this example forwarding rule specifies TCP port 80, when the load balancer is the next hop for a route, traffic on any TCP or UDP port is sent to the load balancer's backends.

  • Internal IP address: The example specifies an internal IP address, 10.30.1.99, for the forwarding rule.

Console

Create the first load balancer

  1. In the Google Cloud Console, go to the Load balancing page.

    Go to Load balancing

  2. Click Create load balancer.

  3. Under TCP load balancing, click Start configuration.

  4. Under Internet facing or internal only select Only between my VMs.

  5. Click Continue.

  6. Set the Name to ilb1.

  7. Click Backend configuration and make the following changes:

    1. Region: us-west1
    2. Network: testing
    3. Under Backends, in the New item section, select the third-party-instance-group instance group and click Done.
    4. Under Health check, choose Create another health check, enter the following information, and click Save and continue:
      • Name: hc-http-80
      • Protocol: HTTP
      • Port: 80
      • Proxy protocol: NONE
      • Request path: / Note that when you use the Cloud Console to create your load balancer, the health check is global. If you want to create a regional health check, use gcloud or the API.
    5. Verify that there is a blue check mark next to Backend configuration before continuing. Review this step if not.
  8. Click Frontend configuration. In the New Frontend IP and port section, make the following changes:

    1. Name: fr-ilb1
    2. Subnetwork: testing-subnet
    3. From Internal IP, choose Reserve a static internal IP address, enter the following information, and click Reserve:
      • Name: ip-ilb
      • Static IP address: Let me choose
      • Custom IP address: 10.30.1.99
    4. Ports: Choose Single, and enter 80 for the Port number. Remember that the choice of a protocol and port for the load balancer does not limit the protocols and ports that are used when the load balancer is the next hop of a route.
    5. Verify that there is a blue check mark next to Frontend configuration before continuing. Review this step if not.
  9. Click Review and finalize. Double-check your settings.

  10. Click Create.

Create the second load balancer

  1. In the Google Cloud Console, go to the Load balancing page.

    Go to Load balancing

  2. Click Create load balancer.

  3. Under TCP load balancing, click Start configuration.

  4. Under Internet facing or internal only, select Only between my VMs.

  5. Click Continue.

  6. Set the Name to ilb2.

  7. Click Backend configuration and make the following changes:

    1. Region: us-west1
    2. Network: production
    3. Under Backends, in the New item section, select the third-party-instance-group instance group and click Done.
    4. Under Health check, select hc-http-80.
    5. Verify that there is a blue check mark next to Backend configuration before continuing. Review this step if not.
  8. Click Frontend configuration. In the New Frontend IP and port section, make the following changes:

    1. Name: fr-ilb2
    2. Subnetwork: testing-subnet
    3. From Internal IP, choose Reserve a static internal IP address, enter the following information, and then click Reserve:
      • Name: ip-ilb2
      • Static IP address: Let me choose
      • Custom IP address: 10.50.1.99
    4. Ports: Choose Single, and enter 80 for the Port number. Remember that the choice of a protocol and port for the load balancer does not limit the protocols and ports that are used when the load balancer is the next hop of a route.
    5. Verify that there is a blue check mark next to Frontend configuration before continuing. Review this step if not.
  9. Click Review and finalize. Double-check your settings.

  10. Click Create.

  11. Configure the load balancer resources in the production VPC network.

gcloud

  1. Create a new HTTP health check to test TCP connectivity to the VMs on 80.

    gcloud compute health-checks create http hc-http-80 \
        --region=us-west1 \
        --port=80
    
  2. Create two internal backend services in the us-west1 region.

    gcloud compute backend-services create ilb1 \
        --load-balancing-scheme=internal \
        --health-checks-region=us-west1 \
        --health-checks=hc-http-80 \
        --region=us-west1 \
        --network=testing \
        --session-affinity=CLIENT_IP
    
    gcloud compute backend-services create ilb2 \
        --load-balancing-scheme=internal \
        --health-checks-region=us-west1 \
        --health-checks=hc-http-80 \
        --region=us-west1 \
        --network=production \
        --session-affinity=CLIENT_IP
    
  3. Add the instance groups containing the third-party virtual appliances as backends on the backend services.

    gcloud compute backend-services add-backend ilb1 \
        --instance-group=third-party-instance-group \
        --instance-group-region=us-west1 \
        --region=us-west1
    
    gcloud compute backend-services add-backend ilb2 \
        --instance-group=third-party-instance-group \
        --instance-group-region=us-west1 \
        --region=us-west1
    
  4. Create the internal forwarding rules and connect them to the backend services to complete the load balancer configuration. Remember that the protocol (TCP) and port (80) of the load balancers do not limit the ports and protocols that are forwarded to the backend instances (the third-party virtual appliances) when the load balancers are used as the next hops of routes.

    gcloud compute forwarding-rules create fr-ilb1 \
        --load-balancing-scheme=internal \
        --ports=80 \
        --network=testing \
        --subnet=testing-subnet \
        --region=us-west1 \
        --backend-service=ilb1 \
        --address=10.30.1.99
    
    gcloud compute forwarding-rules create fr-ilb2 \
        --load-balancing-scheme=internal \
        --ports=80 \
        --network=production \
        --subnet=production-subnet \
        --region=us-west1 \
        --backend-service=ilb2 \
        --address=10.50.1.99
    

Creating the static routes that define the load balancers as the next hops

Create two custom static routes shown next-hop-ilb.

Console

Create the first route

  1. In the Google Cloud Console, go to the Routes page.

    Go to Routes

  2. Click Create route.

  3. For the route Name, enter ilb-nhop-dest-10-50-1.

  4. Select the testing network.

  5. For the Destination IP range, enter 10.50.1.0/24.

  6. (Optional) Specify one or more network tags. The route can apply to specific VMs if you specify network tags on the route. If you don't specify any network tag, the route applies to all VMs in the VPC network. In this example, the route uses my-network-tag for the route's network tag.

  7. For the route's Next hop, select Specify a forwarding rule of internal TCP/UDP load balancer.

    To specify the load balancer's IP address as the next hop, use gcloud tool or the API.

  8. For the next-hop region, select us-west1.

  9. Specify the forwarding rule name or IP address. For the forwarding rule name, select fr-ilb1. For the IP address, enter 10.30.1.99. If you specify the IP address, this IP address can be learned across peers without having to export the custom route.

  10. Click Create.

Create the second route

  1. Click Create route.
  2. For the route Name, enter ilb-nhop-dest-10-30-1.
  3. Select the testing network.
  4. For the Destination IP range, enter 10.30.1.0/24.
  5. For the route's Next hop, select Specify a forwarding rule of internal TCP/UDP load balancer.

    To specify the load balancer's IP address as the next hop, use gcloud tool or the API.

  6. For the next-hop region, select us-west1.

  7. For the forwarding rule name, select fr-ilb2.

  8. Click Create.

gcloud

Create advanced routes with the next hop set to each load balancer's forwarding rule, and each destination range set accordingly.

gcloud compute routes create ilb-nhop-dest-10-50-1 \
    --network=testing \
    --destination-range=10.50.1.0/24 \
    --next-hop-ilb=10.30.1.99 \
    --next-hop-ilb-region=us-west1 \
    --tags=my-network-tag
gcloud compute routes create ilb-nhop-dest-10-30-1 \
    --network=production \
    --destination-range=10.30.1.0/24 \
    --next-hop-ilb=fr-ilb2 \
    --next-hop-ilb-region=us-west1

Creating the testing VM instance

This example creates a VM instance with the IP address 10.30.1.100 in the testing-subnet (10.30.1.0/24) in the testing VPC network.

gcloud

  1. Create the testing-vm by running the following command.

    gcloud compute instances create testing-vm \
        --zone=us-west1-a \
        --image-family=debian-10 \
        --image-project=debian-cloud \
        --tags=allow-ssh \
        --subnet=testing-subnet \
        --private-network-ip 10.30.1.100 \
        --metadata=startup-script='#! /bin/bash
        apt-get update
        apt-get install apache2 -y
        a2ensite default-ssl
        a2enmod ssl
        vm_hostname="$(curl -H "Metadata-Flavor:Google" \
        http://169.254.169.254/computeMetadata/v1/instance/name)"
        echo "Page served from: $vm_hostname" | \
        tee /var/www/html/index.html
        systemctl restart apache2'
    

Creating the production VM instance

This example creates a VM instance with the IP address 10.50.1.100 in the production-subnet (10.50.1.0/24) in the production VPC network.

gcloud

The production-vm can be in any zone in the same region as the load balancer, and it can use any subnet in that region. In this example, the production-vm is in the us-west1-a zone.

  1. Create the production-vm by running the following command.

    gcloud compute instances create production-vm \
        --zone=us-west1-a \
        --image-family=debian-10 \
        --image-project=debian-cloud \
        --tags=allow-ssh \
        --subnet=production-subnet \
        --private-network-ip 10.50.1.100 \
        --metadata=startup-script='#! /bin/bash
        apt-get update
        apt-get install apache2 -y
        a2ensite default-ssl
        a2enmod ssl
        vm_hostname="$(curl -H "Metadata-Flavor:Google" \
        http://169.254.169.254/computeMetadata/v1/instance/name)"
        echo "Page served from: $vm_hostname" | \
        tee /var/www/html/index.html
        systemctl restart apache2'
    

Testing load balancing to a multi-NIC deployment

  1. Verify the health of the load balancer backends.

    gcloud compute backend-services get-health ilb1 --region us-west1
    
    gcloud compute backend-services get-health ilb2 --region us-west1
    
  2. Test connectivity from the testing VM.

    gcloud compute ssh testing-vm --zone=us-west1-a
    
    curl http://10.50.1.100
    
    exit
    
  3. Test connectivity from the production VM.

    gcloud compute ssh production-vm --zone=us-west1-a
    
    curl http://10.30.1.100
    

Enabling symmetric hashing

When computing the hash that is mapped to the backend instance, Google Cloud ignores the direction of IP addresses and ports. The computed consistent hash value of a TCP/UDP packet is the same regardless of the direction that the packet originates from. This is called symmetric hashing.

To enable this hashing behavior on existing internal TCP/UDP load balancers, you must re-create the forwarding rule and the next-hop route.

For more information, see Symmetric hashing.

Delete and re-create the forwarding rules

Console

Delete your forwarding rule and create a new one

  1. In the Google Cloud Console, go to the Load balancing page.

    Go to Load balancing

  2. Click your be-ilb load balancer and click Edit.

  3. Click Frontend configuration.

  4. Hold the pointer over your forwarding rule and then click Delete to remove it.

  5. Click Add frontend IP and port.

  6. In the New Frontend IP and port section, make the following changes:

    1. Name: FORWARDING_RULE_NAME
    2. Subnetwork: SUBNET_NAME
    3. From Internal IP, select IP_ADDRESS
    4. Ports: PORT_NUMBER or ALL.
    5. Click Done.
    6. Verify that there is a blue check mark next to Frontend configuration before continuing. Review this step if not.
  7. Click Review and finalize. Double-check your settings.

  8. Click Create.

gcloud

  1. Delete your existing forwarding rules.

    gcloud compute forwarding-rules delete FORWARDING_RULE_NAME \
        --region=REGION
    
  2. Create replacement forwarding rules with the same name.

    gcloud compute forwarding-rules create FORWARDING_RULE_NAME \
        --load-balancing-scheme=internal \
        --ports=PORT_NUMBER or `ALL` \
        --network=NETWORK_NAME \
        --subnet=SUBNET_NAME \
        --region=REGION \
        --backend-service=BACKEND_SERVICE_NAME \
        --address=IP_ADDRESS
    

When SNAT isn't required

As demonstrated in the previous example, source network address translation (SNAT) is not required when all of the following are true:

  • The forwarding rule for the internal TCP/UDP load balancer was created on or after June 22, 2021.
  • The custom static route referencing the forwarding rule was created on or after June 22, 2021.
  • The internal TCP/UDP load balancer's backend service doesn't use the NONE session affinity setting.

You can convert an existing next hop internal TCP/UDP load balancer route to use symmetric hashing by following these steps:

  • Ensure that the internal TCP/UDP load balancer's backend service doesn't use the NONE session affinity setting

  • Create a replacement forwarding rule that references the same backend service. The replacement forwarding rule uses a different IP address.

  • Create a replacement custom static route referencing the new forwarding rule. Ensure that this replacement route has a higher priority than the existing route.

  • Delete the lower-priority existing route (referencing the previous forwarding rule) and then delete the previous forwarding rule.

Cleanup

  1. In the load balancer configuration, remove the backend from the backend services.

    gcloud compute backend-services remove-backend ilb1 \
        --instance-group=third-party-instance-group \
        --instance-group-region=us-west1 \
        --region=us-west1
    
    gcloud compute backend-services remove-backend ilb2 \
        --instance-group=third-party-instance-group \
        --instance-group-region=us-west1 \
        --region=us-west1
    
  2. Delete the routes.

    gcloud compute routes delete ilb-nhop-dest-10-50-1
    
    gcloud compute routes delete ilb-nhop-dest-10-30-1
    
  3. In the load balancer configurations, delete the forwarding rules.

    gcloud compute forwarding-rules delete fr-ilb1 \
        --region=us-west1
    
    gcloud compute forwarding-rules delete fr-ilb2 \
        --region=us-west1
    
  4. In the load balancer configurations, delete the backend services.

    gcloud compute backend-services delete ilb1 \
        --region=us-west1
    
    gcloud compute backend-services delete ilb2 \
        --region=us-west1
    
  5. In the load balancer configurations, delete the health check.

    gcloud compute health-checks delete hc-http-80 \
        --region=us-west1
    

    If you used the Cloud Console, the health check is global. Therefore, the command is as follows:

    gcloud compute health-checks delete hc-http-80 \
         --global
    
  6. Delete the managed instance group.

    gcloud compute instance-groups managed delete third-party-instance-group \
        --region=us-west1
    
  7. Delete the instance templates.

    gcloud compute instance-templates delete third-party-template
    
    gcloud compute instance-templates delete third-party-template-multinic
    
  8. Delete the testing and production instances.

    gcloud compute instances delete testing-vm \
        --zone=us-west1-a
    
    gcloud compute instances delete production-vm \
        --zone=us-west1-a
    

What's next