Setting up Internal TCP/UDP Load Balancing for third-party appliances

This guide uses an example to teach you how to configure a Google Cloud Platform internal TCP/UDP load balancer to be a next hop. Before following this guide, familiarize yourself with the following:

Permissions

To follow this guide, you need to create instances and modify a network in a project. You should be either a project owner or editor, or you should have all of the following Compute Engine IAM roles:

Task Required Role
Create networks, subnets, and load balancer components Network Admin
Add and remove firewall rules Security Admin
Create instances Instance Admin

Setup

This guide shows you how to use an internal TCP/UDP load balancer as the next hop for a custom static route in order to integrate scaled-out virtual appliances.

The solution discussed in this guide integrates virtual appliances so that you don't need to explicitly reconfigure your clients to send traffic to each virtual appliance. The example in this setup guide sends all East-West traffic through a load-balanced set of firewall virtual appliances.

The steps in this section describe how to configure the following resources:

  • Sample VPC networks and custom subnets
  • GCP firewall rules that allow incoming connections to backend VMs
  • A custom static route
  • One client VM to test connections
  • The following internal TCP/UDP load balancer components:
    • Backend VMs in a managed instance group
    • A health check for the backend VM appliances
    • An internal backend service in the us-west1 region to manage connection distribution among the backend VMs
    • An internal forwarding rule and internal IP address for the frontend of the load balancer

At a high-level, the topology looks like this:

East-West Next-Hop Example for Internal TCP/UDP Load Balancing (click to enlarge)
East-West Next-Hop Example for Internal TCP/UDP Load Balancing (click to enlarge)

The diagram shows some of the resources that the example creates:

  • Application instances (in this case, VMs running firewall appliance software) behind an internal TCP/UDP load balancer (fr-ilb, in this example). The application instances only have internal (RFC 1918) IP addresses.
  • Each application instance has its [can-ip-forward](/vpc/docs/using-routes#canipforward) flag enabled. This is because, by default, each Compute Engine VM sends only packets that the VM itself originates, meaning that by default the source IP address of the packet matches one of the VM's IP addresses. The can-ip-forward setting allows the application instances to forward packets that are originated by another node.
  • A custom static route with destination 10.50.1.0/24 and next hop set to the load balancer's forwarding rule, fr-ilb.

The diagram also shows the traffic flow:

  • Wgeb traffic is sent to the subnet using 10.50.1.0/24 as the primary IP range in the production VPC network, the traffic is routed through the load balancer.
  • The load balancer forwards traffic to one of the application instances based on the configured session affinity. (Session affinity only affects TCP traffic.)
  • The application instance performs source network address translation (SNAT) in order to deliver packets to the instance group in the production VPC network. For return traffic, it performs destination network address translation (DNAT) in order to deliver packets to the client instance in the transit VPC network.

For additional use cases, see Internal TCP/UDP load balancer as a next hop concepts.

The following diagram provides a more detailed representation of the load balancing resources and the networking topology:

East-West Next-Hop Example for Internal TCP/UDP Load Balancing (click to enlarge)
East-West Next-Hop Example for Internal TCP/UDP Load Balancing (click to enlarge)

Configuring the networks, region, and subnets

This example uses the following VPC networks, region, and subnets:

  • Networks: This example requires two two networks, each with at least one subnet. Each backend third-party appliance VM must have at least two network interfaces, one in each VPC network. The networks in this example are custom mode VPC networks named transit and production. The transit network in this example contains the client and the load balancer. The production network contains the destination target VM.

  • Region: The subnets are located in the us-west1 region. The subnets must be in the same region because VM instances are zonal resources.

  • Subnets: The subnets, transit-subnet and production-subnet, use the 10.30.1.0/24 and 10.50.1.0/24 primary IP address ranges, respectively.

To create the example networks and subnets, follow these steps.

Console

Create the transit network and the transit-subnet:

  1. Go to the VPC networks page in the Google Cloud Platform Console.
    Go to the VPC network page
  2. Click Create VPC network.
  3. Enter a Name of transit.
  4. In the Subnets section:
    • Set the Subnet creation mode to Custom.
    • In the New subnet section, enter the following information:
      • Name: transit-subnet
      • Region: us-west1
      • IP address range: 10.30.1.0/24
      • Click Done.
  5. Click Create.

Create the production network and the production-subnet:

  1. Go to the VPC networks page in the Google Cloud Platform Console.
    Go to the VPC network page
  2. Click Create VPC network.
  3. Enter a Name of production.
  4. In the Subnets section:
    • Set the Subnet creation mode to Custom.
    • In the New subnet section, enter the following information:
      • Name: production-subnet
      • Region: us-west1
      • IP address range: 10.50.1.0/24
      • Click Done.
  5. Click Create.

gcloud

  1. Create the custom VPC networks:

    gcloud compute networks create transit --subnet-mode=custom
    
    gcloud compute networks create production --subnet-mode=custom
    
  2. Create subnets in the transit and production networks in the us-west1 region:

    gcloud compute networks subnets create transit-subnet \
        --network=transit \
        --range=10.30.1.0/24 \
        --region=us-west1
    
    gcloud compute networks subnets create production-subnet \
        --network=production \
        --range=10.50.1.0/24 \
        --region=us-west1
    

Configuring firewall rules

This example uses the following firewall rules:

  • fw-allow-transit-subnet: An ingress rule, applicable to all targets in the transit network, allowing traffic from sources in the 10.30.1.0/24 ranges. This rule allows the client and third-party VM appliances in the transit-subnet to communicate.

  • fw-allow-production-subnet: An ingress rule, applicable to all targets in the production network, allowing traffic from sources in the 10.50.1.0/24 ranges. This rule allows the third-party VM appliances in the production-subnet and the destination target VM to communicate.

  • fw-allow-transit-ssh: An ingress rule applied to the instances being load balanced, allowing incoming SSH connectivity on TCP port 22 from any address. You can choose a more restrictive source IP range for this rule; for example, you can specify the IP ranges of the systems from which you plan to initiate SSH sessions. This example uses the target tag allow-ssh to identify the VMs to which the firewall rule applies.

  • fw-allow-health-check: An ingress rule, applicable to the third-party VM appliances load balanced, that allows traffic from the GCP health checking systems (130.211.0.0/22 and 35.191.0.0/16). This example uses the target tag allow-health-check to identify the instances to which it should apply.

Without these firewall rules, the default deny ingress rule blocks incoming traffic to the backend instances. You must create a firewall rule to allow health checks from the IP ranges of GCP probe systems. Refer to probe IP ranges for more information.

Console

  1. Go to the Firewall rules page in the Google Cloud Platform Console.
    Go to the Firewall rules page
  2. Click Create firewall rule and enter the following information to create the rule to allow subnet traffic:
    • Name: fw-allow-transit-subnet
    • Network: transit
    • Priority: 1000
    • Direction of traffic: ingress
    • Action on match: allow
    • Targets: All instances in the network
    • Source filter: IP ranges
    • Source IP ranges: 10.30.1.0/24
    • Protocols and ports: Allow all
  3. Click Create.
  4. Click Create firewall rule and enter the following information to create the rule to allow subnet traffic:
    • Name: fw-allow-production-subnet
    • Network: production
    • Priority: 1000
    • Direction of traffic: ingress
    • Action on match: allow
    • Targets: All instances in the network
    • Source filter: IP ranges
    • Source IP ranges: 10.50.1.0/24
    • Protocols and ports: Allow all
  5. Click Create.
  6. Click Create firewall rule again to create the rule to allow incoming SSH connections:
    • Name: fw-allow-transit-ssh
    • Network: transit
    • Priority: 1000
    • Direction of traffic: ingress
    • Action on match: allow
    • Targets: Specified target tags
    • Target tags: allow-ssh
    • Source filter: IP ranges
    • Source IP ranges: 0.0.0.0/0
    • Protocols and ports: Choose Specified protocols and ports and type: tcp:22
  7. Click Create.
  8. Click Create firewall rule a third time to create the rule to allow GCP health checks:
    • Name: fw-allow-health-check
    • Network: transit
    • Priority: 1000
    • Direction of traffic: ingress
    • Action on match: allow
    • Targets: Specified target tags
    • Target tags: allow-health-check
    • Source filter: IP ranges
    • Source IP ranges: 130.211.0.0/22 and 35.191.0.0/16
    • Protocols and ports: Allow all
  9. Click Create.

gcloud

  1. Create the fw-allow-transit-subnet firewall rule to allow communication from with the subnet:

    gcloud compute firewall-rules create fw-allow-transit-subnet \
        --network=transit \
        --action=allow \
        --direction=ingress \
        --source-ranges=10.30.1.0/24 \
        --rules=tcp,udp,icmp
    
  2. Create the fw-allow-production-subnet firewall rule to allow communication from with the subnet:

    gcloud compute firewall-rules create fw-allow-production-subnet \
        --network=production \
        --action=allow \
        --direction=ingress \
        --source-ranges=10.50.1.0/24 \
        --rules=tcp,udp,icmp
    
  3. Create the fw-allow-transit-ssh firewall rule to allow SSH connectivity to VMs with the network tag allow-ssh. When you omit source-ranges, GCP interprets the rule to mean any source.

    gcloud compute firewall-rules create fw-allow-transit-ssh \
        --network=transit \
        --action=allow \
        --direction=ingress \
        --target-tags=allow-ssh \
        --rules=tcp:22
    
  4. Create the fw-allow-health-check rule to allow GCP health checks to the third-party appliance VMs in the transit network.

    gcloud compute firewall-rules create fw-allow-transit-health-check \
        --network=transit \
        --action=allow \
        --direction=ingress \
        --target-tags=allow-health-check \
        --source-ranges=130.211.0.0/22,35.191.0.0/16 \
        --rules=tcp,udp,icmp
    

Creating the destination instance

This example creates a destination instance with the IP address 10.50.1.2 in the production-subnet (10.50.1.0/24) in the production VPC network.

Console

  1. Go to the VM instances page in the Google Cloud Platform Console.
    Go to the VM instances page
  2. For the VM Name, enter destination-instance.
  3. For the Region, choose us-west1, and choose us-west1-a for the Zone.
  4. In the Boot disk section, ensure that the selected image is Debian GNU/Linux 9 Stretch. Click Choose to change the image if necessary.
  5. Click Management, security, disks, networking, sole tenancy and make the following changes:

    • Click Networking and add the following Network tags: allow-ssh
    • Click the edit button under Network interfaces and make the following changes then click Done:
      • Network: production
      • Subnet: production-subnet
      • Primary internal IP: Ephemeral (custom)
      • Custom ephemeral IP address: 10.50.1.2
      • External IP: Ephemeral
    • Click Management. In the Startup script field, copy and paste the following script contents. The script contents are identical for all four VMs:

      #! /bin/bash
      apt-get update
      apt-get install apache2 -y
      a2ensite default-ssl
      a2enmod ssl
      vm_hostname="$(curl -H "Metadata-Flavor:Google" \
      http://169.254.169.254/computeMetadata/v1/instance/name)"
      echo "Page served from: $vm_hostname" | \
      tee /var/www/html/index.html
      systemctl restart apache2
      
  6. Click Create.

gcloud

  1. Create the VM by running the following command.

    gcloud compute instances create destination-instance \
        --zone=us-west1-a \
        --image-family=debian-9 \
        --image-project=debian-cloud \
        --tags=allow-ssh \
        --subnet=production-subnet \
        --private-network-ip 10.50.1.2 \
        --metadata=startup-script='#! /bin/bash
        apt-get update
        apt-get install apache2 -y
        a2ensite default-ssl
        a2enmod ssl
        vm_hostname="$(curl -H "Metadata-Flavor:Google" \
        http://169.254.169.254/computeMetadata/v1/instance/name)"
        echo "Page served from: $vm_hostname" | \
        tee /var/www/html/index.html
        systemctl restart apache2'
    

Creating the third-party virtual appliances

The following steps demonstrate how to create an instance template and managed regional instance group using the iptables software as a third-party virtual appliance.

Console

You must use gcloud for this step because you need to create an instance template with more than one network interface. The GCP Console does not currently support creating instance templates with more than one network interface.

gcloud

  1. Create an instance template for your third-party virtual appliances. The instance template must include the --can-ip-forward flag so that the VM instances created from the template can forward packets from other instances in the transit and production networks.

    gcloud compute instance-templates create third-party-template \
        --region=us-west1 \
        --network-interface subnet=transit-subnet,address="" \
        --network-interface subnet=production-subnet \
        --tags=allow-ssh,allow-health-check \
        --image-family=debian-9 \
        --image-project=debian-cloud \
        --can-ip-forward \
        --metadata=startup-script='#! /bin/bash
        # Enable IP forwarding:
        echo 1 > /proc/sys/net/ipv4/ip_forward
        echo "net.ipv4.ip_forward=1" > /etc/sysctl.d/20-iptables.conf
        # Read VM network configuration:
        md_vm="http://169.254.169.254/computeMetadata/v1/instance/"
        md_net="$md_vm/network-interfaces"
        nic0_gw="$(curl -H "Metadata-Flavor:Google" $md_net/0/gateway)"
        nic0_mask="$(curl -H "Metadata-Flavor:Google" $md_net/0/subnetmask)"
        nic1_gw="$(curl -H "Metadata-Flavor:Google" $md_net/1/gateway)"
        nic1_mask="$(curl -H "Metadata-Flavor:Google" $md_net/1/subnetmask)"
        nic1_addr="$(curl -H "Metadata-Flavor:Google" $md_net/1/ip)"
        # Start iptables:
        /sbin/iptables -t nat -F
        /sbin/iptables -t nat -A POSTROUTING \
        -s "$nic0_gw/$nic0_mask" \
        -d "$nic1_gw/$nic1_mask" \
        -o eth1 \
        -j SNAT \
        --to-source "$nic1_addr"
        /sbin/iptables-save
        # Use a web server to pass the health check for this example.
        # You should use a more complete test in production.
        apt-get update
        apt-get install apache2 -y
        a2ensite default-ssl
        a2enmod ssl
        echo "Example web page to pass health check" | \
        tee /var/www/html/index.html
        systemctl restart apache2'
    
  2. Create a managed instance group for your third-party virtual appliances. This command creates a regional managed instance group, which can then be autoscaled, in us-west1.

    gcloud compute instance-groups managed create third-party-instance-group \
        --region us-west1 \
        --template=third-party-template \
        --size=3
    

Creating the load balancing resources

These steps configure all of the internal TCP/UDP load balancer components starting with the health check and backend service, and then the frontend components:

  • Health check: In this example, the HTTP health check checks for an HTTP 200 (OK) response. For more information, see the health checks section of the Internal TCP/UDP Load Balancing overview.

  • Backend service: Even though this example's backend service specifies the TCP protocol, when the load balancer is the next hop for a route, both TCP and UDP traffic are sent to the load balancer's backends.

  • Forwarding rule: Even though this example forwarding rule specifies TCP port 80, when the load balancer is the next hop for a route, traffic on any TCP or UDP port is sent to the load balancer's backends.

  • Internal IP address: The example specifies an internal IP address, 10.30.1.99, for the forwarding rule.

Console

Create the load balancer and configure a backend service

  1. Go to the Load balancing page in the Google Cloud Platform Console.
    Go to the Load balancing page
  2. Click Create load balancer.
  3. Under TCP load balancing, click Start configuration.
  4. Under Internet facing or internal only select Only between my VMs.
  5. Click Continue.
  6. Set the Name to be-ilb.
  7. Click Backend configuration and make the following changes:
    1. Region: us-west1
    2. Network: transit
    3. Under Backends, in the New item section, select the third-party-instance-group instance group and click Done.
    4. Under Health check, choose Create another health check, enter the following information, and click Save and continue:
      • Name: hc-http-80
      • Protocol: HTTP
      • Port: 80
      • Proxy protocol: NONE
      • Request path: /
    5. Verify that there is a blue check mark next to Backend configuration before continuing. Review this step if not.
  8. Click Frontend configuration. In the New Frontend IP and port section, make the following changes:
    1. Name: fr-ilb
    2. Subnetwork: transit-subnet
    3. From Internal IP, choose Reserve a static internal IP address, enter the following information, and click Reserve:
      • Name: ip-ilb
      • Static IP address: Let me choose
      • Custom IP address: 10.30.1.99
    4. Ports: Choose Single, and enter 80 for the Port number. Remember that the choice of a protocol and port for the load balancer does not limit the protocols and ports that are used when the load balancer is the next hop of a route.
    5. Verify that there is a blue check mark next to Frontend configuration before continuing. Review this step if not.
  9. Click Review and finalize. Double-check your settings.
  10. Click Create.

gcloud

  1. Create a new HTTP health check to test TCP connectivity to the VMs on 80.

    gcloud compute health-checks create http hc-http-80 \
        --port=80
    
  2. Create an internal backend service in the us-west1 region.

    gcloud compute backend-services create be-ilb \
        --load-balancing-scheme=internal \
        --region=us-west1 \
        --health-checks=hc-http-80
    
  3. Add the instance group containing the third-party virtual appliances as a backend on the backend service.

    gcloud compute backend-services add-backend be-ilb \
        --instance-group=third-party-instance-group \
        --instance-group-region=us-west1 \
        --region=us-west1
    
  4. Create the internal forwarding rule and connect it to the backend service to complete the load balancer configuration. Remember that the protocol (TCP) and port (80) of the internal load balancer do not limit the ports and protocols that are forwarded to the backend instances (the third-party virtual appliances) when the load balancer is used as the next hop of a route.

    gcloud compute forwarding-rules create fr-ilb \
        --load-balancing-scheme=internal \
        --ports=80 \
        --network=transit \
        --subnet=transit-subnet \
        --region=us-west1 \
        --backend-service=be-ilb \
        --address=10.30.1.99
    

Creating the static route that defines the load balancer as the next hop

When you create a static route, you cannot use next-hop-address to point to the IP address of a load balancer's forwarding rule. This is because when you use next-hop-address, GCP passes traffic to the VM instance assigned to that IP address, and a load balancer is not a VM instance. Instead, if you want to designate a load balancer as next hop, you must use the next-hop-ilb flag, as demonstrated in this example.

Console

  1. Go to the Routes page in the Google Cloud Platform Console.
    Go to the Routes page
  2. Click Create route.
  3. For the route Name, enter `ilb-nhop-default.
  4. Select the transit network.
  5. For the Destination IP range, enter 10.50.1.0/24.
  6. Make sure tags aren't specified, as they aren't currently supported with this feature.
  7. For the route's Next hop, select Specify a forwarding rule internal TCP/UDP load balancer.
  8. For the next-hop region, select us-west1.
  9. For the forwarding rule name, select fr-ilb.
  10. Click Create.

gcloud

Create an advanced route with the next hop set to the load balancer's forwarding rule, and the destination range set to the route 10.50.1.0/24.

gcloud beta compute routes create ilb-nhop-default \
    --network=transit \
    --destination-range=10.50.1.0/24 \
    --next-hop-ilb=fr-ilb \
    --next-hop-ilb-region=us-west1

Creating a client VM

This example creates a client VM (vm-client) in the same region as the load balancer in the transit VPC network. The client is used to demonstrate how the next-hop behavior works.

Console

  1. Go to the VM instances page in the Google Cloud Platform Console.
    Go to the VM instances page
  2. Click Create instance.
  3. Set the Name to vm-client.
  4. Set the Zone to us-west1-a.
  5. Click Management, security, disks, networking, sole tenancy and make the following changes:
    • Click Networking and add the allow-ssh to Network tags.
    • Click the edit button under Network interfaces and make the following changes then click Done:
      • Network: transit
      • Subnet: transit-subnet
      • Primary internal IP: Ephemeral (automatic)
      • External IP: Ephemeral
  6. Click Create.

gcloud

The client VM can be in any zone in the same region as the load balancer, and it can use any subnet in that region. In this example, the client is in the us-west1-a zone, and it uses the same subnet used by the primary and backup VMs.

gcloud compute instances create vm-client \
    --zone=us-west1-a \
    --image-family=debian-9 \
    --image-project=debian-cloud \
    --tags=allow-ssh \
    --subnet=transit-subnet

Testing

This test contacts the example destination VM in the production VPC network from the client VM in the transit VPC network. The load balancer is used as a next hop because it routes the packet with destination 10.50.1.2 through the load balancer, rather than sending it to the IP address of the load balancer.

In this example, iptables software on the load balancer's healthy backend appliance VMs processes NAT for the packet.

  1. Connect to the client VM instance.

    gcloud compute ssh vm-client --zone=us-west1-a
    
  2. Make a web request to the destination instance's web server sofware using curl. The expected output is the content of the index page on the destination instance (Page served from: destination-instance).

    curl http://10.50.1.2
    

What's next

Var denne siden nyttig? Si fra hva du synes:

Send tilbakemelding om ...