Setting up Network Load Balancing for multiple IP protocols

Preview

This product or feature is covered by the Pre-GA Offerings Terms of the Google Cloud Platform Terms of Service. Pre-GA products and features may have limited support, and changes to pre-GA products and features may not be compatible with other pre-GA versions. For more information, see the launch stage descriptions.

This guide provides instructions for creating backend service-based network load balancers that load-balance TCP, UDP, ESP, and ICMP traffic. You can use such a configuration if you want to load-balance traffic that is using IP protocols other than TCP or UDP. Target pool-based network load balancers do not support this capability.

To configure a network load balancer for IP protocols other than TCP or UDP, you create a forwarding rule with protocol set to L3_DEFAULT. This forwarding rule points to a backend service with protocol set to UNSPECIFIED.

In this example, we use two network load balancers to distribute traffic across backend VMs in two zonal managed instance groups in the us-central1 region. Both load balancers receive traffic at the same external IP address.

One load balancer has a forwarding rule with protocol TCP and port 80, and the other load balancer has a forwarding rule with protocol L3_DEFAULT. TCP traffic arriving at the IP address on port 80 will be handled by the TCP forwarding rule. All other TCP, UDP, ESP, and ICMP traffic that does not match the TCP-specific forwarding rule will be handled by the L3_DEFAULT forwarding rule.

Network Load Balancing with zonal managed instance groups

This scenario distributes traffic across healthy instances. To support this, you create TCP health checks to ensure that traffic is sent only to healthy instances.

The network load balancer is a regional load balancer. All load balancer components must be in the same region.

Before you begin

Install the gcloud command-line tool. For a complete overview of the tool, see the gcloud Tool Guide. You can find commands related to load balancing in the API and gcloud reference guide.

If you haven't run the gcloud command-line tool previously, first run gcloud init to authenticate.

This guide assumes that you are familiar with bash.

Creating the zonal managed instance groups

For this load balancing scenario, you create two Compute Engine zonal managed instance groups and install an echo server on each instance.

Instances that participate as backend VMs for network load balancers must be running the appropriate Linux Guest Environment, Windows Guest Environment, or other processes that provide equivalent functionality.

Setting up the instance group for TCP traffic on port 80

Cloud Console

1. Go to the Instance groups page in the Cloud Console.

   Go to the Instance groups page

2. Click Create instance group.
3. Choose New managed instance group on the left.
4. For the Name, enter ig-us-tcp-80.
5. Under Location, select Single zone.
6. For the Region, select us-central1.
7. For the Zone, select us-central1-a.

8. Under Instance template, select Create a new instance template.

   1. For the Name, enter ig-us-template-tcp-80.
   2. Ensure that the Boot disk is set to a Debian image, such as Debian GNU/Linux 9 (stretch). These instructions use commands that are only available on Debian, such as apt-get.

   3. Under Management, security, disks, networking, sole tenancy, on the Management tab, insert the following script into the Startup script field.

      #! /bin/bash
      apt-get update
      apt-get install apache2 -y
      a2ensite default-ssl
      a2enmod ssl
      vm_hostname="$(curl -H "Metadata-Flavor:Google" \
      http://169.254.169.254/computeMetadata/v1/instance/name)"
      echo "Page served from: $vm_hostname" | \
      tee /var/www/html/index.html
      systemctl restart apache2
      

   4. Under Networking, populate the Tags field with the network-lb-tcp-80 tag.

   5. Click Save and continue.

9. Specify the number of instances that you want to create in the group.

   For this example, under Autoscaling mode, select:

   • Don't autoscale
   • Under Number of instances, enter 2

10. Click Create to create the new instance group.

gcloud

The gcloud instructions in this guide assume that you are using Cloud Shell or another environment with bash installed.

1. Create a VM instance template with HTTP server with the gcloud compute instance-templates create command.

   gcloud compute instance-templates create ig-us-template-tcp-80 \
   --region=us-central1 \
   --tags=network-lb-tcp-80 \
   --image-family=debian-9 \
   --image-project=debian-cloud \
   --metadata=startup-script='#! /bin/bash
   apt-get update
   apt-get install apache2 -y
   a2ensite default-ssl
   a2enmod ssl
   vm_hostname="$(curl -H "Metadata-Flavor:Google" \
   http://169.254.169.254/computeMetadata/v1/instance/name)"
   echo "Page served from: $vm_hostname" | \
   tee /var/www/html/index.html
   systemctl restart apache2'
   

2. Create a managed instance group in the zone with the gcloud compute instance-groups managed create command.

   gcloud compute instance-groups managed create ig-us-tcp-80 \
       --zone us-central1-a \
       --size 2 \
       --template ig-us-template-tcp-80
   

Setting up the instance group for TCP on port 8080, UDP, ESP, and ICMP traffic

Cloud Console

1. Go to the Instance groups page in the Cloud Console.

   Go to the Instance groups page

2. Click Create instance group.
3. Choose New managed instance group on the left.
4. For the Name, enter ig-us-l3-default.
5. Under Location, select Single zone.
6. For the Region, select us-central1.
7. For the Zone, select us-central1-c.

8. Under Instance template, select Create a new instance template.

   1. For the Name, enter ig-us-template-l3-default.
   2. Ensure that the Boot disk is set to a Debian image, such as Debian GNU/Linux 9 (stretch). These instructions use commands that are only available on Debian, such as apt-get.

   3. Under Management, security, disks, networking, sole tenancy, on the Management tab, insert the following script into the Startup script field. The startup script also configures the Apache server to listen on port 8080 instead of port 80.

      #! /bin/bash
      apt-get update
      apt-get install apache2 -y
      a2ensite default-ssl
      a2enmod ssl
      vm_hostname="$(curl -H "Metadata-Flavor:Google" \
      http://169.254.169.254/computeMetadata/v1/instance/name)"
      echo "Page served from: $vm_hostname" | \
      tee /var/www/html/index.html
      sed -ire 's/^Listen 80$/Listen 8080/g' /etc/apache2/ports.conf
      systemctl restart apache2
      

   4. Under Networking, populate the Tags field with the network-lb-l3-default tag.

   5. Click Save and continue.

9. Specify the number of instances that you want to create in the group.

   For this example, under Autoscaling mode, select:

   • Don't autoscale
   • Under Number of instances, enter 2

10. Click Create to create the new instance group.

gcloud

The gcloud instructions in this guide assume that you are using Cloud Shell or another environment with bash installed.

1. Create a VM instance template with HTTP server with the gcloud compute instance-templates create command.

   The startup script also configures the Apache server to listen on port 8080 instead of port 80.

   gcloud compute instance-templates create ig-us-template-l3-default \
   --region=us-central1 \
   --tags=network-lb-l3-default \
   --image-family=debian-9 \
   --image-project=debian-cloud \
   --metadata=startup-script='#! /bin/bash
   apt-get update
   apt-get install apache2 -y
   a2ensite default-ssl
   a2enmod ssl
   vm_hostname="$(curl -H "Metadata-Flavor:Google" \
   http://169.254.169.254/computeMetadata/v1/instance/name)"
   echo "Page served from: $vm_hostname" | \
   tee /var/www/html/index.html
   sed -ire "s/^Listen 80$/Listen 8080/g" /etc/apache2/ports.conf
   systemctl restart apache2'
   

2. Create a managed instance group in the zone with the gcloud compute instance-groups managed create command.

   gcloud compute instance-groups managed create ig-us-l3-default \
       --zone us-central1-c \
       --size 2 \
       --template ig-us-template-l3-default
   

Configuring firewall rules

Note: For Network Load Balancing, the L3_DEFAULT forwarding rule protocol can be used to load-balance TCP, UDP, ESP, and ICMP traffic.

Create the following firewall rules:

• A firewall rule that allows external TCP traffic to reach backend instances in the ig-us-tcp-80 instance group on port 80 (using target tag network-lb-tcp-80).

• A firewall rule that allows other external traffic (TCP on port 8080, UDP, ESP, and ICMP) to reach backend instances in the ig-us-l3-default instance group (using target tag network-lb-l3-default).

Console

1. Go to the Firewalls page in the Google Cloud Console.
   Go to the Firewalls page
2. Create the first firewall rule for TCP traffic.
   1. Click Create firewall rule.
   2. Enter a Name of allow-network-lb-tcp-80.
   3. Select the Network that the firewall rule applies to (Default).
   4. Under Targets, select Specified target tags.
   5. In the Target tags field, enter network-lb-tcp-80.
   6. Set the Source IP ranges to 0.0.0.0/0, which allows traffic from any source. This allows both external traffic and health check probes to reach the backend instances.
   7. Under Protocols and ports, select Specified protocols and ports. Click the checkbox next to tcp and enter 80.
   8. Click Create. It might take a moment for the Console to display the new firewall rule, or you might have to click Refresh to see the rule.
3. Create the second firewall rule for UDP, ESP, and ICMP traffic.
   1. Click Create firewall rule.
   2. Enter a Name of allow-network-lb-l3-default.
   3. Select the Network that the firewall rule applies to (Default).
   4. Under Targets, select Specified target tags.
   5. In the Target tags field, enter network-lb-l3-default.
   6. Set the Source IP ranges to 0.0.0.0/0, which allows traffic from any source. This allows both external traffic and health check probes to reach the backend instances.
   7. Under Protocols and ports, select Specified protocols and ports.
      1. Click the checkbox next to tcp and enter 8080.
      2. Click the checkbox next to udp.
      3. Click the checkbox next to Other protocols and enter esp, icmp.
   8. Click Create. It might take a moment for the Console to display the new firewall rule, or you might have to click Refresh to see the rule.

gcloud

1. Create the first firewall rule for TCP traffic.

   gcloud compute firewall-rules create allow-network-lb-tcp-80 \
       --target-tags network-lb-tcp-80 \
       --allow tcp:80
   

2. Create the second firewall rule for UDP, ESP, and ICMP traffic. This firewall also allows TCP health check probes to reach the instances on port 8080.

   gcloud compute firewall-rules create allow-network-lb-l3-default \
       --target-tags network-lb-l3-default \
       --allow tcp:8080,udp,esp,icmp
   

Configuring the load balancers

Next, set up the load balancers. One load balancer to handle TCP traffic on port 80 and another load balancer to handle TCP on port 8080, UDP, ESP, and ICMP traffic. Both load balancers will use the same external IP address with their forwarding rules.

When you configure a load balancer, your backend VM instances will receive packets that are destined for the static external IP address you configure. If you are using an image provided by Compute Engine, your instances are automatically configured to handle this IP address. If you are using any other image, you must configure this address as an alias on eth0 or as a loopback on each instance.

Note: You cannot use the Google Cloud Console to set up a network load balancer that uses the L3_DEFAULT forwarding rule protocol. Use either the gcloud command-line tool or the REST API. This example uses gcloud command-line tool to set up both load balancers.

gcloud

1. Create a static external IP address for your load balancers.

   gcloud compute addresses create network-lb-ip \
       --region us-central1
   

2. Create a TCP health check for port 80. This is used to verify the health of backends in the ig-us-tcp-80 instance group.

   gcloud compute health-checks create tcp tcp-health-check-80 \
       --region us-central1 \
       --port 80
   

3. Create a TCP health check for port 8080. This is used to verify the health of backends in the ig-us-l3-default instance group.

   gcloud compute health-checks create tcp tcp-health-check-8080 \
       --region us-central1 \
       --port 8080
   

4. Create the first load balancer for TCP traffic on port 80.

   1. Create a backend service with protocol TCP.

      gcloud alpha compute backend-services create backend-service-tcp-80 \
          --protocol TCP \
          --health-checks tcp-health-check-80 \
          --health-checks-region us-central1 \
          --region us-central1
      

   2. Add the backend instance group to the backend service.

      gcloud alpha compute backend-services add-backend backend-service-tcp-80 \
          --instance-group ig-us-tcp-80 \
          --instance-group-zone us-central1-a \
          --region us-central1
      

   3. Create a forwarding rule to route incoming TCP traffic on port 80 to the backend service. TCP is the default forwarding rule protocol and does not need to be set explicitly.

      Use the IP address reserved in step 1 as the static external IP address of the load balancer.

      gcloud alpha compute forwarding-rules create forwarding-rule-tcp-80 \
          --load-balancing-scheme external \
          --region us-central1 \
          --ports 80 \
          --address network-lb-ip \
          --backend-service backend-service-tcp-80
      

5. Create the second load balancer for TCP on port 8080, UDP, ESP, and ICMP traffic.

   1. Create a backend service with protocol UNSPECIFIED.

      gcloud alpha compute backend-services create backend-service-l3-default \
          --protocol UNSPECIFIED \
          --health-checks tcp-health-check-8080 \
          --health-checks-region us-central1 \
          --region us-central1
      

   2. Add the backend instance group to the backend service.

      gcloud alpha compute backend-services add-backend backend-service-l3-default \
          --instance-group ig-us-l3-default \
          --instance-group-zone us-central1-c \
          --region us-central1
      

   3. Create a forwarding rule with protocol set to L3_DEFAULT to handle all remaining supported IP protocol traffic (TCP on port 8080, UDP, ESP, and ICMP). All ports must be configured with L3_DEFAULT forwarding rules.

      Use the same external IP address as that used for the previous load balancer.

      gcloud alpha compute forwarding-rules create forwarding-rule-l3-default \
          --load-balancing-scheme external \
          --region us-central1 \
          --ports all \
          --ip-protocol L3_DEFAULT \
          --address network-lb-ip \
          --backend-service backend-service-l3-default
      

Testing the load balancer

Now that the load balancing service is configured, you can start sending traffic to the load balancer's external IP address and watch traffic get distributed to the backend instances.

Look up the load balancer's external IP address

Console

1. On the Advanced load balancing page, go to the Forwarding Rules tab.
   Go to the Forwarding Rules tab
2. Locate either forwarding-rule-tcp-80 or forwarding-rule-l3-default. This example uses the same IP address for both load balancers so either one will work.
3. In the IP Address column, note the external IP address listed.

gcloud

Enter the following command to view the external IP address of the forwarding rule used by the load balancer.

gcloud alpha compute forwarding-rules describe forwarding-rule-tcp-80 \
    --region us-central1

This example uses the same IP address for both forwarding rules so using forwarding-rule-l3-default will also work.

Send traffic to the load balancer

This procedure sends external traffic to the load balancer. Run the following tests to ensure that TCP traffic on port 80 is being load-balanced by the ig-us-tcp-80 instance group while all other traffic (TCP on port 8080, UDP, ESP, and ICMP) is being handled by the ig-us-l3-default instance group.

Verifying behavior with TCP requests on port 80

1. Make web requests (over TCP on port 80) to the load balancer using curl to contact its IP address.

   $ while true; do curl -m1 IP_ADDRESS; done
   

2. Note the text returned by the curl command. The name of the backend VM generating the response is displayed in that text; for example: Page served from: VM_NAME. Responses should come from instances in the ig-us-tcp-80 instance group only.

   If your response is initially unsuccessful, you might need to wait approximately 30 seconds for the configuration to be fully loaded and for your instances to be marked healthy before trying again.

Verifying behavior with TCP requests on port 8080

1. Make web requests (over TCP on port 8080) to the load balancer using curl to contact its IP address.

   $ while true; do curl -m1 IP_ADDRESS:8080; done
   

2. Note the text returned by the curl command. Responses should come from instances in the ig-us-l3-default instance group only.

   This shows that any traffic sent to the load balancer's IP address at port 8080 is being handled by backends in the ig-us-l3-default instance group only.

Verifying behavior with ICMP requests

To verify behavior with ICMP traffic, you capture output from the tcpdump command to confirm that only backend VMs in the ig-us-l3-default instance group are handling ICMP requests send to the load balancer.

1. SSH to the backend VMs.

   1. In the Cloud Console, go to the VM instances page.
      Go to the VM instances page

   2. In the list of virtual machine instances, click SSH in the row of the instance that you want to connect to.

2. Run the following command to use tcpdump to start listening for ICMP traffic.

   sudo tcpdump icmp -w ~/icmpcapture.pcap -s0 -c 10000
   tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
   

   Leave the SSH window open.

3. Repeat steps 1 and 2 for all four backend VMs.

4. Make ICMP requests to the load balancer by using ping to contact its IP address.

   ping IP_ADDRESS
   

5. Go back to each VM's open SSH window and stop the tcpdump capture command. You can use Ctrl+C to do this.

6. For each VM, check the output of the tcpdump command in the icmpcapture.pcap file.

   sudo tcpdump -r ~/icmpcapture.pcap -n
   

   For backend VMs in the ig-us-l3-default instance group, you should see file entries like:

   reading from file /home/[user-directory]/icmpcapture.pcap, link-type EN10MB (Ethernet)
   22:13:07.814486 IP 35.230.115.24 > 35.193.84.93: ICMP echo request, id 1995, seq 1, length 64
   22:13:07.814513 IP 35.193.84.93 > 35.230.115.24: ICMP echo reply, id 1995, seq 1, length 64
   22:13:08.816150 IP 35.230.115.24 > 35.193.84.93: ICMP echo request, id 1995, seq 2, length 64
   22:13:08.816175 IP 35.193.84.93 > 35.230.115.24: ICMP echo reply, id 1995, seq 2, length 64
   22:13:09.817536 IP 35.230.115.24 > 35.193.84.93: ICMP echo request, id 1995, seq 3, length 64
   22:13:09.817560 IP 35.193.84.93 > 35.230.115.24: ICMP echo reply, id 1995, seq 3, length 64
   ...
   

   For backend VMs in the ig-us-tcp-80 instance group, you should see that no packets have been received and the file should be blank:

   reading from file /home/[user-directory]/icmpcapture.pcap, link-type EN10MB (Ethernet)
   

What's next

• For information on how Network Load Balancing works with a backend service, see Network Load Balancing with backend service overview.
• To learn how to transition a network load balancer from a target pool backend to a regional backend service, see Transitioning network load balancers from target pools to backend services
• To configure a network load balancer with a target pool, see Setting up Network Load Balancing.
• Clean up the load balancer setup.