Set up an external passthrough Network Load Balancer with zonal NEGs

This document shows you how to deploy an external passthrough Network Load Balancer with zonal network endpoint group (NEG) backends. Zonal NEGs with GCE_VM_IP endpoints let you do the following:

  • Forward packets to the non-nic0 network interfaces of virtual machine (VM) instances by representing a network interface of a VM instance as the backend endpoint.
  • Create a flexible set of backend endpoints where an endpoint can be attached to multiple NEGs, and each NEG can have a different but possibly overlapping set of endpoints.

Before following this document, familiarize yourself with the following:

Permissions

To follow these steps, you need to create instances and modify a network in a project. You must either be a project owner or editor, or you must have all of the following Compute Engine IAM roles:

Task Required role
Create networks, subnets, and load balancer components Network Admin
Add and remove firewall rules Security Admin
Create instances Compute Instance Admin

For more information, see the following pages:

Setup overview

This document shows you how to configure and test an external passthrough Network Load Balancer that uses GCE_VM_IP zonal NEG backends. The steps in this section describe how to configure the following:

  1. A sample VPC network called lb-network with a custom subnet.
  2. Firewall rules that allow incoming connections to backend VMs.
  3. Four VMs:
    • VMs vm-a1 and vm-a2 in zone us-west1-a
    • VMs vm-c1 and vm-c2 in zone us-west1-c
  4. Two backend zonal NEGs, neg-a in zone us-west1-a, and neg-c in zone us-west1-c. Each NEG has the following endpoints:
    • neg-a contains these two endpoints:
      • nic1 of VM vm-a1 identified by its primary internal IP address
      • nic1 of VM vm-a2 identified by its primary internal IP address
    • neg-c contains these two endpoints:
      • nic1 of VM vm-c1 identified by its primary internal IP address
      • nic1 of VM vm-c2 identified by its primary internal IP address
  5. One client VM (vm-client) in us-west1-a to test connections
  6. The following load balancer components:
    • An external backend service in the us-west1 region to manage connection distribution to the two zonal NEGs
    • An external forwarding rule and IP address for the frontend of the load balancer

The external passthrough Network Load Balancer is a regional load balancer. All load balancer components (backend VMs, backend service, and forwarding rule) must be in the same region.

The architecture for this example looks like this:

External passthrough Network Load Balancer with zonal NEGs.
External passthrough Network Load Balancer with zonal NEGs (click to enlarge).

Before you begin

Install the Google Cloud CLI. For a complete overview of the tool, see the gcloud CLI overview. You can find commands related to load balancing in the API and gcloud references.

If you haven't run the gcloud CLI previously, first run the gcloud init command to authenticate.

This page assumes that you are familiar with bash.

Set up the network and subnets

The example on this page uses a custom mode VPC network named lb-network. You can use an auto mode VPC network if you only want to handle IPv4 traffic. However, IPv6 traffic requires a custom mode subnet.

IPv6 traffic also requires a dual-stack subnet (stack-type set to IPv4_IPv6). When you create a dual stack subnet on a custom mode VPC network, you choose an IPv6 access type for the subnet. For this example, we set the subnet's ipv6-access-type parameter to EXTERNAL. This means new VMs on this subnet can be assigned both external IPv4 addresses and external IPv6 addresses.

The backends and the load balancer components used for this example are located in this region and subnet:

  • Region: us-central1
  • Subnet: lb-subnet, with primary IPv4 address range 10.1.2.0/24. Although you choose which IPv4 address range is configured on the subnet, the IPv6 address range is assigned automatically. Google provides a fixed size (/64) IPv6 CIDR block.

To create the example network and subnet, follow these steps.

Console

To support both IPv4 and IPv6 traffic, use the following steps:

  1. In the Google Cloud console, go to the VPC networks page.

    Go to VPC networks

  2. Click Create VPC network.

  3. Enter a Name of lb-network.

  4. In the Subnets section:

    • Set the Subnet creation mode to Custom.
    • In the New subnet section, configure the following fields and click Done:
      • Name: lb-subnet
      • Region: us-central1
      • IP stack type: IPv4 and IPv6 (dual-stack)
      • IPv4 range: 10.1.2.0/24
        Although you can configure an IPv4 range of addresses for the subnet, you cannot choose the range of the IPv6 addresses for the subnet. Google provides a fixed size (/64) IPv6 CIDR block.
      • IPv6 access type: External

  5. Click Create.

To support IPv4 traffic only, use the following steps:

  1. In the Google Cloud console, go to the VPC networks page.

    Go to VPC networks

  2. Click Create VPC network.

  3. Enter a Name of lb-network.

  4. In the Subnets section:

    • Set the Subnet creation mode to Custom.
    • In the New subnet section, configure the following fields and click Done:
      • Name: lb-subnet
      • Region: us-central1
      • IP stack type: IPv4 (single-stack)
      • IPv4 range: 10.1.2.0/24

  5. Click Create.

gcloud

  1. Create the custom mode VPC network:

    gcloud compute networks create lb-network \
    --subnet-mode=custom

  2. Within the lb-network network, create a subnet for backends in the us-central1 region.

    For both IPv4 and IPv6 traffic, use the following command to create a dual-stack subnet:

    gcloud compute networks subnets create lb-subnet \
  --stack-type=IPV4_IPv6 \
  --ipv6-access-type=EXTERNAL \
  --network=lb-network \
  --range=10.1.2.0/24 \
  --region=us-central1

    For IPv4 traffic only, use the following command:

    gcloud compute networks subnets create lb-subnet \
  --network=lb-network \
  --range=10.1.2.0/24 \
  --region=us-central1

Create VMs and network endpoint groups

To demonstrate the regional nature of external passthrough Network Load Balancers, this example uses two zonal NEG backends in two different zones. Traffic is load-balanced across both NEGs, and across endpoints within each NEG.

Create VMs

For this load balancing scenario, you create four VMs and install an Apache web server on each instance. The web server listens on TCP port 80. By default, Apache is configured to bind to any IP address. External passthrough Network Load Balancers deliver packets by preserving the destination IP address.

For instructional simplicity, these backend VMs run Debian GNU Linux 10.

To handle both IPv4 and IPv6 traffic, configure the backend VMs to be dual-stack. Set the VM's stack-type to IPv4_IPv6. The VMs also inherit the ipv6-access-type setting (in this example, EXTERNAL) from the subnet. For more details about IPv6 requirements, see the External passthrough Network Load Balancer overview: Forwarding rules.

To use existing VMs as backends, update the VMs to be dual-stack by using the gcloud compute instances network-interfaces update command.

Instances that participate as backend VMs for external passthrough Network Load Balancers must run the appropriate Linux guest environment, Windows guest environment, or other processes that provide equivalent capability.

Each VM is created with two network interfaces, nic0 and nic1. This tutorial uses nic1, which is associated with the lb-network VPC network and lb-subnet subnet. Use this lb-network network and lb-subnet subnet to create the zonal NEGs later in this procedure.

Console

Create VMs

  1. In the Google Cloud console, go to the VM instances page.

    Go to VM instances

  2. Repeat the following steps to create four VMs, using the following name and zone combinations:

    • Name: vm-a1, zone: us-west1-a
    • Name: vm-a2, zone: us-west1-a
    • Name: vm-c1, zone: us-west1-c
    • Name: vm-c2, zone: us-west1-c

  3. Click Create instance.

  4. Set the Name as indicated previously.

  5. For the Region, choose us-west1, and choose a Zone as indicated previously.

  6. In the Boot disk section, ensure that the Debian operating system and the 10 (buster) version are selected for the boot disk options. If necessary, click Choose to change the image.

  7. Click Advanced options and make the following changes:

    • Click Networking and add the following Network tags: lb-tag

    • For Network interfaces, click Add network interface and make the following changes:

      For IPv4 and IPv6 backends:

      • Network: lb-network
      • Subnet: lb-subnet
      • IP stack type: IPv4 and IPv6 (dual-stack)
      • Primary internal IP: Ephemeral (automatic)
      • External IPv4 address: Ephemeral (automatic)
      • External IPv6 address: Auto-allocate

      For IPv4 only backends:

      • Network: lb-network
      • Subnet: lb-subnet
      • IP stack type: IPv4 (single-stack)
      • Primary internal IP: Ephemeral (automatic)
      • External IP: Ephemeral

      Then click Done.

    • Click Management. In the Startup script field, copy and paste the following script contents. The script contents are identical for all four VMs:

      #! /bin/bash
apt-get update
apt-get install apache2 -y
a2ensite default-ssl
a2enmod ssl
vm_hostname="$(curl -H "Metadata-Flavor:Google" \
http://metadata.google.internal/computeMetadata/v1/instance/name)"
echo "Page served from: $vm_hostname" | \
tee /var/www/html/index.html
systemctl restart apache2

  8. Click Create.

gcloud

Create the four VMs by running the following command four times, using these four combinations for [VM-NAME] and [ZONE]. The script contents are identical for all four VMs.

  • VM_NAME: vm-a1 and ZONE: us-west1-a
  • VM_NAME: vm-a2 and ZONE: us-west1-a
  • VM_NAME: vm-c1 and ZONE: us-west1-c
  • VM_NAME: vm-c2 and ZONE: us-west1-c

To handle both IPv4 and IPv6 traffic, use the following command.

gcloud compute instances create VM_NAME \
    --zone=ZONE \
    --image-family=debian-10 \
    --image-project=debian-cloud \
    --tags=lb-tag \
    --network-interface=network=default,subnet=default,stack_type=IPv4_IPv6,--ipv6-network-tier=PREMIUM \
    --network-interface=network=lb-network,subnet=lb-subnet,stack_type=IPv4_IPv6,--ipv6-network-tier=PREMIUM \
    --metadata=startup-script='#! /bin/bash
      apt-get update
      apt-get install apache2 -y
      a2ensite default-ssl
      a2enmod ssl
      vm_hostname="$(curl -H "Metadata-Flavor:Google" \
      http://metadata.google.internal/computeMetadata/v1/instance/name)"
      echo "Page served from: $vm_hostname" | \
      tee /var/www/html/index.html
      systemctl restart apache2'

To handle IPv4 traffic only traffic, use the following command:

gcloud compute instances create VM_NAME \
    --zone=ZONE \
    --image-family=debian-10 \
    --image-project=debian-cloud \
    --tags=lb-tag \
    --network-interface=network=default,subnet=default,stack_type=IPv4_ONLY \
    --network-interface=network=lb-network,subnet=lb-subnet,stack_type=IPv4_ONLY \
    --metadata=startup-script='#! /bin/bash
      apt-get update
      apt-get install apache2 -y
      a2ensite default-ssl
      a2enmod ssl
      vm_hostname="$(curl -H "Metadata-Flavor:Google" \
      http://metadata.google.internal/computeMetadata/v1/instance/name)"
      echo "Page served from: $vm_hostname" | \
      tee /var/www/html/index.html
      systemctl restart apache2'

Create zonal NEGs with GCE_VM_IP endpoints

The NEGs must be created in the same zones as the VMs created in the previous step. This example also creates the NEG in the lb-network VPC network and lb-subnet subnet that are associated with nic1 of the VMs created in the previous step. Therefore, the endpoints of the NEG are going to be on nic1 of the VMs.

Console

To create a zonal network endpoint group:

  1. Go to the Network Endpoint Groups page in the Google Cloud console.
    Go to the Network Endpoint Groups page
  2. Click Create network endpoint group.
  3. Enter a Name for the zonal NEG: neg-a.
  4. Select the Network endpoint group type: Network endpoint group (Zonal).
  5. Select the Network: lb-network
  6. Select the Subnet: lb-subnet
  7. Select the Zone: us-west1-a
  8. Click Create.
  9. Repeat these steps to create a second zonal NEG called neg-c, in the us-west1-c zone.

Add endpoints to the zonal NEG:

  1. Go to the Network Endpoint Groups page in the Google Cloud console.
    Go to the Network endpoint groups
  2. Click the Name of the first network endpoint group created in the previous step (neg-a). You see the Network endpoint group details page.

  3. In the Network endpoints in this group section, click Add network endpoint. You see the Add network endpoint page.

    1. Click VM instance and select vm-a1 to add its internal IP addresses as network endpoints.
    2. Click Create.
    3. Again click Add network endpoint and under VM instance select vm-a2.
    4. Click Create.

  4. Click the Name of the second network endpoint group created in the previous step (neg-c). You see the Network endpoint group details page.

  5. In the Network endpoints in this group section, click Add network endpoint. You see the Add network endpoint page.

    1. Click VM instance and select vm-c1 to add its internal IP addresses as network endpoints.
    2. Click Create.
    3. Again click Add network endpoint and under VM instance select vm-c2.
    4. Click Create.

gcloud

  1. Create a GCE_VM_IP zonal NEG called neg-a in us-west1-a using the gcloud compute network-endpoint-groups create command:

    gcloud compute network-endpoint-groups create neg-a \
    --network-endpoint-type=gce-vm-ip \
    --zone=us-west1-a \
    --network=lb-network \
    --subnet=lb-subnet

  2. Add endpoints to neg-a:

    gcloud compute network-endpoint-groups update neg-a \
    --zone=us-west1-a \
    --add-endpoint='instance=vm-a1' \
    --add-endpoint='instance=vm-a2'

  3. Create a GCE_VM_IP zonal NEG called neg-c in us-west1-c using the gcloud compute network-endpoint-groups create command:

    gcloud compute network-endpoint-groups create neg-c \
    --network-endpoint-type=gce-vm-ip \
    --zone=us-west1-c \
    --network=lb-network \
    --subnet=lb-subnet

  4. Add endpoints to neg-c:

    gcloud compute network-endpoint-groups update neg-c \
    --zone=us-west1-c \
    --add-endpoint='instance=vm-c1' \
    --add-endpoint='instance=vm-c2'

Configuring firewall rules

Create firewall rules that allow external traffic (which includes health check probes) to reach the backend instances.

This example creates a firewall rule that allows TCP traffic from all source ranges to reach your backend instances on port 80. If you want to create separate firewall rules specifically for the health check probes, use the source IP address ranges documented in the Health checks overview: Probe IP ranges and firewall rules.

Console

  1. In the Google Cloud console, go to the Firewall page.

    Go to Firewall

  2. To allow IPv4 traffic, perform the following steps:

    1. Click Create firewall rule.
    2. Enter a Name: allow-network-lb-ipv4.
    3. For Network, select lb-network.
    4. For Targets, select Specified target tags.
    5. For the Target tags field, enter lb-tag.
    6. For Source filter, select IPv4 ranges.
    7. Set Source IPv4 ranges to 0.0.0.0/0. This allows IPv4 traffic from any source. This also allows Google's health check probes to reach the backend instances.
    8. For Specified protocols and ports, select TCP and enter 80.
    9. Click Create.

  3. To allow IPv6 traffic, perform the following steps:

    1. Click Create firewall rule again.
    2. Enter a Name: allow-network-lb-ipv6.
    3. For Network, select lb-network.
    4. For Targets, select Specified target tags.
    5. For the Target tags field, enter lb-tag.
    6. For Source filter, select IPv6 ranges.
    7. Set Source IPv6 ranges to ::/0. This allows IPv6 traffic from any source. This also allows Google's health check probes to reach the backend instances.
    8. For Specified protocols and ports, select TCP and enter 80.
    9. Click Create.

gcloud

  1. To allow IPv4 traffic, run the following command:

    gcloud compute firewall-rules create allow-network-lb-ipv4 \
    --network=lb-network \
    --target-tags=lb-tag \
    --allow=tcp:80 \
    --source-ranges=0.0.0.0/0

  2. To allow IPv6 traffic, run the following command:

    gcloud compute firewall-rules create allow-network-lb-ipv6 \
  --network=lb-network \
  --target-tags=lb-tag \
  --allow=tcp:80 \
  --source-ranges=::/0

Configure the load balancer

Next, set up the load balancer.

When you configure the load balancer, your VMs receive packets that are destined for the static external IP address you configure. If you are using an image provided by Compute Engine, your instances are automatically configured to handle this IP address. If you are using any other image, you must configure this address as an alias on eth1 or as a loopback on each instance.

Console

Start your configuration

  1. In the Google Cloud console, go to the Load balancing page.

    Go to Load balancing

  2. Click Create load balancer.
  3. For Type of load balancer, select Network Load Balancer (TCP/UDP/SSL) and click Next.
  4. For Proxy or passthrough, select Passthrough load balancer and click Next.
  5. For Public facing or internal, select Public facing (external) and click Next.
  6. Click Configure.

Backend configuration

  1. On the Create external passthrough Network Load Balancer page, enter the name network-lb-zonal-neg for the new load balancer.
  2. For Region, select us-west1.
  3. Under Backend type, select Zonal network endpoint group.
  4. Click Backend configuration. The load balancer name that you entered previously appears, but is not modifiable.
  5. On the Backend configuration page, make the following changes:
    1. In the Network endpoint group list, select neg-a, and then click Done.
    2. Click Add backend and repeat the previous step to add neg-c.
    3. For Health check, choose Create a health check or Create another health check, and then enter the following information:
      • Name: tcp-health-check
      • Protocol: TCP
      • Port: 80
    4. Click Save.
  6. Verify that there is a blue checkmark next to Backend configuration before continuing.

Frontend configuration

  1. Click Frontend configuration.
  2. For Name, enter netlb-forwarding-rule.
  3. To handle IPv4 traffic, use the following steps:
    1. For IP version, select IPv4.
    2. For IP address, click the drop-down menu and select Create IP address.
      1. On the Reserve a new static IP address page, for Name, enter netlb-ipv4-address.
      2. Click Reserve.
    3. For Ports, choose Single. For Port number, enter 80.
    4. Click Done.

  4. To handle IPv6 traffic, use the following steps:

    1. For IP version, select IPv6.
    2. For Subnetwork, select lb-subnet.
    3. For IPv6 range, click the drop-down menu and select Create IP address.
      1. On the Reserve a new static IP address page, for Name, enter netlb-ipv6-address.
      2. Click Reserve.
    4. For Ports, choose Single. For Port number, enter 80.
    5. Click Done.

    A blue circle with a checkmark to the left of Frontend configuration indicates a successful setup.

Review the configuration

  1. Click Review and finalize.
  2. Review your load balancer configuration settings.
  3. Optional: Click Equivalent code to view the REST API request that will be used to create the load balancer.

  4. Click Create.

    On the load balancing page, under the Backend column for your new load balancer, you should see a green checkmark showing that the new load balancer is healthy.

gcloud

  1. Reserve a static external IP address.

    For IPv4 traffic: Create a static external IPv4 address for your load balancer.

    gcloud compute addresses create netlb-ipv4-address \
    --region=us-west1

    For IPv6 traffic: Create a static external IPv6 address range for your load balancer. The subnet used must be a dual-stack subnet with an external IPv6 range.

    gcloud compute addresses create netlb-ipv6-address \
    --region=us-west1 \
    --subnet=lb-subnet \
    --ip-version=IPV6 \
    --endpoint-type=NETLB

  2. Create a TCP health check. 

    gcloud compute health-checks create tcp tcp-health-check \
    --region=us-west1 \
    --port=80

  3. Create a backend service.

    gcloud compute backend-services create networklb-backend-service \
    --protocol=TCP \
    --health-checks=tcp-health-check \
    --health-checks-region=us-west1 \
    --region=us-west1

  4. Add the two zonal NEGs, neg-a and neg-c, to the backend service:

    gcloud compute backend-services add-backend networklb-backend-service \
    --region=us-west1 \
    --network-endpoint-group=neg-a \
    --network-endpoint-group-zone=us-west1-a

    gcloud compute backend-services add-backend networklb-backend-service \
    --region=us-west1 \
    --network-endpoint-group=neg-c \
    --network-endpoint-group-zone=us-west1-c

  5. Create the forwarding rules depending on whether you want to handle IPv4 traffic or IPv6 traffic. Create both forwarding rules to handle both types of traffic.

    1. For IPv4 traffic: Create a forwarding rule to route incoming TCP traffic to the backend service. Use the IPv4 address reserved in step 1 as the static external IP address of the load balancer.

      gcloud compute forwarding-rules create forwarding-rule-ipv4 \
  --load-balancing-scheme=EXTERNAL \
  --region=us-west1 \
  --ports=80 \
  --address=netlb-ipv4-address \
  --backend-service=networklb-backend-service

    2. For IPv6 traffic: Create a forwarding rule to handle IPv6 traffic. Use the IPv6 address range reserved in step 1 as the static external IP address of the load balancer. The subnet used must be a dual-stack subnet with an external IPv6 subnet range.

      gcloud compute forwarding-rules create forwarding-rule-ipv6 \
    --load-balancing-scheme=EXTERNAL \
    --region=us-west1 \
    --network-tier=PREMIUM \
    --ip-version=IPV6 \
    --subnet=lb-subnet \
    --address=netlb-ipv6-address \
    --ports=80 \
    --backend-service=networklb-backend-service

Test the load balancer

Now that the load balancing service is configured, you can start sending traffic to the load balancer's external IP address and watch traffic get distributed to the backend instances.

Look up the load balancer's external IP address

Console

  1. On the Advanced load balancing page, go to the Forwarding rules tab.

    Go to Forwarding rules

  2. Locate the forwarding rule used by the load balancer.

  3. In the Address column, note the external IP address listed.

gcloud: IPv4

Enter the following command to view the external IPv4 address of the network-lb-forwarding-rule forwarding rule used by the load balancer.

gcloud compute forwarding-rules describe forwarding-rule-ipv4 \
    --region=us-west1

gcloud: IPv6

Enter the following command to view the external IPv6 address of the network-lb-forwarding-rule forwarding rule used by the load balancer.

gcloud compute forwarding-rules describe forwarding-rule-ipv6 \
    --region=us-west1

Send traffic to the load balancer

Make web requests to the load balancer using curl to contact its IP address.

  • From clients with IPv4 connectivity, run the following command:

    $ while true; do curl -m1 IPV4_ADDRESS; done

  • From clients with IPv6 connectivity, run the following command:

    $ while true; do curl -m1 http://IPV6_ADDRESS; done

    For example, if the assigned IPv6 address is [2001:db8:1:1:1:1:1:1/96]:80, the command should look like:

    $ while true; do curl -m1 http://[2001:db8:1:1:1:1:1:1]:80; done

Note the text returned by the curl command. The name of the backend VM generating the response is displayed in that text; for example: Page served from: VM_NAME

The response from the curl command alternates randomly among the backend instances. If your response is initially unsuccessful, you might need to wait approximately 30 seconds for the configuration to be fully loaded and for your instances to be marked healthy before trying again.

Additional configuration options

To further customize your external passthrough Network Load Balancer, you can configure session affinity, traffic steering, and set up a failover policy or a connection tracking policy. These tasks are optional and you can perform them in any order. For instrutions, see Additional configuration options.

What's next