This guide uses an example to teach you how to configure failover for an external passthrough Network Load Balancer with a backend service. Before following this guide, familiarize yourself with the following:
- External passthrough Network Load Balancer with a backend service overview
- Failover concepts for external passthrough Network Load Balancers
- Firewall rules overview
- Health check concepts
Permissions
To follow this guide, you need to create instances and modify a network in a project. You should be either a project owner or editor, or you should have all of the following Compute Engine IAM roles:
Task | Required Role |
---|---|
Create load balancer components | Network Admin |
Add and remove firewall rules | Security Admin |
Create instances | Compute Instance Admin |
For more information, see the following guides:
Setup
This guide shows you how to configure and test an external passthrough Network Load Balancer that uses failover. The steps in this section describe how to configure the following:
- Backend VMs:
- One primary backend in an unmanaged instance group in zone
us-west1-a
- One failover backend in an unmanaged instance group in zone
us-west1-c
- One primary backend in an unmanaged instance group in zone
- Firewall rules that allow incoming connections to backend VMs
- One client VM to test connections and observe failover behavior
- The following external passthrough Network Load Balancer components:
- A health check for the backend service
- A backend service in the
us-west1
region to manage connection distribution among the backend VMs - A forwarding rule and IP address for the frontend of the load balancer
The architecture for this example looks like this:
Creating backend VMs and instance groups
In this step, you'll create the backend VMs and unmanaged instance groups:
- The instance group
ig-a
inus-west1-a
is a primary backend with two VMs:vm-a1
vm-a2
- The instance group
ig-c
inus-west1-c
is a failover backend with two VMs:vm-c1
vm-c2
The primary and failover backends are placed in separate zones for instructional clarity and to handle failover in case one zone goes down.
Each primary and backup VM is configured to run an Apache web server on TCP port 80. By default, Apache is configured to bind to any IP address. Network load balancers deliver packets by preserving the destination IP.
Ensure that server software running on your primary and backup VMs is listening on the IP address of the load balancer's forwarding rule. The destination IP address of a packet delivered to a backend VM by an external passthrough Network Load Balancer is the IP address of the forwarding rule.
For instructional simplicity, all primary and backup VMs run Debian GNU/Linux 9.
Console
Create backend VMs
- In the Google Cloud console, go to the VM instances page. Go to VM instances
- Repeat the following steps to create four VMs, using the following name
and zone combinations.
- Name:
vm-a1
, zone:us-west1-a
- Name:
vm-a2
, zone:us-west1-a
- Name:
vm-c1
, zone:us-west1-c
- Name:
vm-c2
, zone:us-west1-c
- Name:
- Click Create instance.
- Set the Name as indicated in step 2.
- For the Region, choose
us-west1
, and choose a Zone as indicated in step 2. - In the Boot disk section, ensure that the selected image is Debian GNU/Linux 12 (bookworm). Click Choose to change the image if necessary.
- Click Advanced options.
- Click Networking and configure the following field:
- For Network tags, enter
network-lb
.
- For Network tags, enter
Click Management. Enter the following script into the Startup script field. The script contents are identical for all four VMs:
#! /bin/bash apt-get update apt-get install apache2 -y a2ensite default-ssl a2enmod ssl vm_hostname="$(curl -H "Metadata-Flavor:Google" \ http://metadata.google.internal/computeMetadata/v1/instance/name)" echo "Page served from: $vm_hostname" | \ tee /var/www/html/index.html systemctl restart apache2
Click Create.
Create instance groups
In the Google Cloud console, go to the Instance groups page.
Repeat the following steps to create two unmanaged instance groups each with two VMs in them, using these combinations.
- Instance group:
ig-a
, zone:us-west1-a
, VMs:vm-a1
andvm-a2
- Instance group:
ig-c
, zone:us-west1-c
, VMs:vm-c1
andvm-c2
- Instance group:
Click Create instance group.
Click New unmanaged instance group.
Set Name as indicated in step 2.
In the Location section, choose
us-west1
for the Region, and then choose a Zone as indicated in step 2.For Network, enter
default
.In the VM instances section, add the VMs as indicated in step 2.
Click Create.
gcloud
Create four VMs by running the following command four times, using these four combinations for
VM-NAME
andZONE
. The script contents are identical for all four VMs.VM-NAME
ofvm-a1
andZONE
ofus-west1-a
VM-NAME
ofvm-a2
andZONE
ofus-west1-a
VM-NAME
ofvm-c1
andZONE
ofus-west1-c
VM-NAME
ofvm-c2
andZONE
ofus-west1-c
gcloud compute instances create VM-NAME \ --zone=ZONE \ --image-family=debian-12 \ --image-project=debian-cloud \ --tags=network-lb \ --metadata=startup-script='#! /bin/bash apt-get update apt-get install apache2 -y a2ensite default-ssl a2enmod ssl vm_hostname="$(curl -H "Metadata-Flavor:Google" \ http://metadata.google.internal/computeMetadata/v1/instance/name)" echo "Page served from: $vm_hostname" | \ tee /var/www/html/index.html systemctl restart apache2'
Create the two unmanaged instance groups in each zone:
gcloud compute instance-groups unmanaged create ig-a \ --zone us-west1-a gcloud compute instance-groups unmanaged create ig-c \ --zone us-west1-c
Add the VMs to the appropriate instance groups:
gcloud compute instance-groups unmanaged add-instances ig-a \ --zone us-west1-a \ --instances vm-a1,vm-a2 gcloud compute instance-groups unmanaged add-instances ig-c \ --zone us-west1-c \ --instances vm-c1,vm-c2
Configuring firewall rules
Create a firewall rule that allows external traffic to reach the backend instances.
Console
In the Google Cloud console, go to the Firewall policies page.
Click Create firewall rule to create the rule to allow traffic from any source. Use the following values:
- Name:
allow-network-lb
. - Network:
default
- Priority:
1000
- Direction of traffic: Ingress
- Action on match: Allow.
- Targets: Specified target tags
- Target tags:
network-lb
- Source filter: IPv4 ranges
- Source IPv4 ranges:
0.0.0.0/0
, which allows traffic from any source. This allows both external traffic and health check probes to reach the backend instances. - Protocols and ports: Choose Specified protocols and ports.
Select the TCP checkbox and enter
80
.
- Name:
Click Create.
Click Create firewall rule again to create the rule to allow incoming SSH connections to the instances. Use the following values:
- Name:
allow-ssh
- Network:
default
- Priority:
1000
- Direction of traffic: Ingress
- Action on match: Allow
- Targets: Specified target tags
- Target tags:
network-lb
- Source filter: IPv4 ranges
- Source IPv4 ranges:
0.0.0.0/0
- Protocols and ports: Choose Specified protocols and ports.
Select the TCP checkbox and enter
22
.
- Name:
Click Create.
gcloud
gcloud compute firewall-rules create allow-network-lb \ --target-tags network-lb \ --allow tcp:80
gcloud compute firewall-rules create allow-ssh \ --target-tags network-lb \ --allow tcp:22
Configuring load balancer components
These steps configure the following external passthrough Network Load Balancer components:
Health check: This example uses an HTTP health check that simply checks for an HTTP
200
(OK) response.Backend service: Because the example passes HTTP traffic through the load balancer, the configuration specifies TCP, not UDP. To illustrate failover, this backend service has a failover ratio of
0.75
.Forwarding rule: This example creates a single forwarding rule.
Console
Start your configuration
In the Google Cloud console, go to the Load balancing page.
- Click Create load balancer.
- For Type of load balancer, select Network Load Balancer (TCP/UDP/SSL) and click Next.
- For Proxy or passthrough, select Passthrough load balancer and click Next.
- For Public facing or internal, select Public facing (external) and click Next.
- Click Configure.
Backend configuration
- On the Create external passthrough Network Load Balancer page, enter
a Name of
tcp-network-lb
for the new load balancer. - Click Backend configuration. The load balancer Name you entered previously appears, but is not modifiable.
- Click Backend configuration and make the following changes:
- For the Region, select us-west1.
- Under Backends, use the Instance group drop-down menu to
select
ig-a
. Click Done. - Click Add backend and repeat this step to add
ig-c
. Select the Use this instance group as a failover group for backup checkbox. - Under Health check, choose Create another health check or
Create another health check, enter the following information:
- Name:
hc-http-80
- Protocol:
TCP
- Port:
80
- Name:
- Click Save and continue.
- Click Advanced configurations.
- For Failover Ratio, enter
0.75
. - Verify that there is a blue check mark next to Backend configuration before continuing.
Frontend configuration
- Click Frontend configuration.
- Enter a Name of
network-lb-forwarding-rule
. - Under IP, click the drop-down menu and select Create IP address.
- On the Reserve a new static IP address screen, assign a
Name of
network-lb-ip
. - Click Reserve.
- On the Reserve a new static IP address screen, assign a
Name of
- Choose Single, and enter
80
for the Port number. Click the Done button.
A blue circle with a checkmark to the left of Frontend configuration indicates a successful set-up.
Review the configuration
- Click the Review and finalize button to check all of your configuration settings for the load balancer.
If the settings are correct, click Create. It takes a few minutes for the load balancer to be created.
On the load balancing screen, under the Backend column for your new load balancer, you should see a green checkmark showing that the new load balancer is healthy.
gcloud
Create a static external IP address for your load balancer.
gcloud compute addresses create network-lb-ip \ --region us-west1
Create a new HTTP health check to test TCP connectivity to the VMs on 80.
gcloud compute health-checks create http hc-http-80 \ --region us-west1 \ --port 80
Create the backend service for HTTP traffic:
gcloud compute backend-services create network-lb-backend-service \ --protocol tcp \ --region us-west1 \ --health-checks hc-http-80 \ --health-checks-region us-west1 \ --failover-ratio 0.75
Add the primary backend to the backend service:
gcloud compute backend-services add-backend network-lb-backend-service \ --region us-west1 \ --instance-group ig-a \ --instance-group-zone us-west1-a
Add the failover backend to the backend service:
gcloud compute backend-services add-backend network-lb-backend-service \ --region us-west1 \ --instance-group ig-c \ --instance-group-zone us-west1-c \ --failover
Create a forwarding rule for the backend service. Use the IP address reserved in step 1 as the static external IP address of the load balancer.
gcloud compute forwarding-rules create network-lb-forwarding-rule \ --region us-west1 \ --load-balancing-scheme external \ --address network-lb-ip\ --ports 80 \ --backend-service network-lb-backend-service
Testing
These tests show how to validate your load balancer configuration and learn about its expected behavior.
Send traffic to the load balancer
This procedure sends external traffic to the load balancer. You'll use this procedure to complete the other tests.
Connect to the client VM instance.
gcloud compute forwarding-rules describe network-lb-forwarding-rule \ --region us-west1
Make a web request to the load balancer using
curl
to contact its IP address.curl http://IP_ADDRESS
Note the text returned by the
curl
command. The name of the backend VM generating the response is displayed in that text; for example:Page served from: vm-a1
Testing initial state
After you've configured the example load balancer, all four of the backend VMs should be healthy:
- the two primary VMs,
vm-a1
andvm-a2
- the two backup VMs,
vm-c1
andvm-c2
Follow the test procedure to send traffic to the load balancer.
Repeat the second step a few times. The expected behavior is for traffic to be
served by the two primary VMs, vm-a1
and vm-a2
, because both of them are
healthy. You should see each primary VM serve a response approximately half of
the time because no session affinity has been
configured for this load balancer.
Testing failover
This test simulates the failure of vm-a1
so you can observe failover behavior.
Connect to the
vm-a1
VM.gcloud compute ssh vm-a1 --zone us-west1-a
Stop the Apache web server. After ten seconds, Google Cloud considers this VM to be unhealthy. (The
hc-http-80
health check that you created in the setup uses the default check interval of five seconds and unhealthy threshold of two consecutive failed probes.)sudo apachectl stop
Follow the test procedure to send traffic to the load balancer. Repeat the second step a few times. The expected behavior is for traffic to now be served by the two backup VMs,
vm-c1
andvm-c2
. Because only one primary VM,vm-a2
, is healthy, the ratio of healthy primary VMs to total primary VMs is0.5
. This number is less than the failover threshold of0.75
, so Google Cloud reconfigured the load balancer's active pool to use the backup VMs. You should see each backup VM serve a response approximately half of the time as long as no session affinity has been configured for this load balancer.
Testing failback
This test simulates failback by restarting the Apache server on vm-a1
.
Connect to the
vm-a1
VM.gcloud compute ssh vm-a1 --zone us-west1-a
Start the Apache web server and wait 10 seconds.
sudo apachectl start
Follow the client test procedure. Repeat the second step a few times. The expected behavior is for traffic to be served by the two primary VMs,
vm-a1
andvm-a2
. With both primary VMs being healthy, the ratio of healthy primary VMs to total primary VMs is1.0
, greater than the failover threshold of0.75
, so Google Cloud configured the active pool to use the primary VMs again.
Adding more backend VMs
This section extends the example configuration by adding more primary and backup VMs to the load balancer. It does so by creating two more backend instance groups to demonstrate that you can distribute primary and backup VMs among multiple zones in the same region:
- A third instance group,
ig-d
inus-west1-c
, serves as a primary backend with two VMs:vm-d1
vm-d2
- A fourth instance group,
ig-b
inus-west1-a
, serves as a failover backend with two VMs:vm-b1
vm-b2
The modified architecture for this example looks like this:
Create additional VMs and instance groups
Follow these steps to create the additional primary and backup VMs and their corresponding unmanaged instance groups.
Console
Create backend VMs
In the Google Cloud console, go to the VM instances page.
Repeat the following steps to create four VMs, using the following name and zone combinations.
- Name:
vm-b1
, zone:us-west1-a
- Name:
vm-b2
, zone:us-west1-a
- Name:
vm-d1
, zone:us-west1-c
- Name:
vm-d2
, zone:us-west1-c
- Name:
Click Create instance.
Set the Name as indicated in step 2.
For the Region, choose
us-west1
, and choose a Zone as indicated in step 2.In the Boot disk section, ensure that the selected image is Debian GNU/Linux 9 Stretch. Click Choose to change the image if necessary.
Click Advanced options.
Click Networking and configure the following field:
- For Network tags, enter
network-lb
.
- For Network tags, enter
Click Management. Enter the following script into the Startup script field. The script contents are identical for all four VMs:
#! /bin/bash apt-get update apt-get install apache2 -y a2ensite default-ssl a2enmod ssl vm_hostname="$(curl -H "Metadata-Flavor:Google" \ http://metadata.google.internal/computeMetadata/v1/instance/name)" echo "Page served from: $vm_hostname" | \ tee /var/www/html/index.html systemctl restart apache2
Click Create.
Create instance groups
In the Google Cloud console, go to the Instance groups page.
Repeat the following steps to create two unmanaged instance groups each with two VMs in their one, using these combinations.
- Instance group:
ig-b
, zone:us-west1-a
, VMs:vm-b1
andvm-b2
- Instance group:
ig-d
, zone:us-west1-c
, VMs:vm-d1
andvm-d2
- Instance group:
Click Create instance group.
Click New unmanaged instance group.
Set Name as indicated in step 2.
In the Location section, choose
us-west1
for the Region, and then choose a Zone as indicated in step 2.For Network, enter
default
.In the VM instances section, add the VMs as indicated in step 2.
Click Create.
gcloud
Create four VMs by running the following command four times, using these four combinations for
VM-NAME
andZONE
. The script contents are identical for all four VMs.VM-NAME
ofvm-b1
andZONE
ofus-west1-a
VM-NAME
ofvm-b2
andZONE
ofus-west1-a
VM-NAME
ofvm-d1
andZONE
ofus-west1-c
VM-NAME
ofvm-d2
andZONE
ofus-west1-c
gcloud compute instances create VM-NAME \ --zone=ZONE \ --image-family=debian-12 \ --image-project=debian-cloud \ --tags=network-lb \ --metadata=startup-script='#! /bin/bash apt-get update apt-get install apache2 -y a2ensite default-ssl a2enmod ssl vm_hostname="$(curl -H "Metadata-Flavor:Google" \ http://metadata.google.internal/computeMetadata/v1/instance/name)" echo "Page served from: $vm_hostname" | \ tee /var/www/html/index.html systemctl restart apache2'
Create the two unmanaged instance groups in each zone:
gcloud compute instance-groups unmanaged create ig-b \ --zone us-west1-a gcloud compute instance-groups unmanaged create ig-d \ --zone us-west1-c
Add the VMs to the appropriate instance groups:
gcloud compute instance-groups unmanaged add-instances ig-b \ --zone us-west1-a \ --instances vm-b1,vm-b2 gcloud compute instance-groups unmanaged add-instances ig-d \ --zone us-west1-c \ --instances vm-d1,vm-d2
Adding a primary backend
You can use this procedure as a template for how to add an unmanaged instance
group to an existing external passthrough Network Load Balancer's backend service as a
primary backend. For the example configuration, this procedure shows you how to
add instance group ig-d
as a primary backend to the network-lb
load balancer.
Console
Edit the load balancer configuration to add a primary backend.
In the Google Cloud console, go to the Load balancing page.
Click on the load balancer you want to modify.
Click Edit.
Click Backend configuration and make the following changes:
- Under Backends, click Add Backend.
- From the dropdown, select the instance group to be added as a
primary backend. In this case,
ig-d
. - Click Done.
- Verify that there is a blue check mark next to Backend configuration before continuing.
Review the configuration
- Click the Review and finalize button and confirm that the new primary backend is visible under Backend.
If the settings are correct, click Create. It takes a few minutes for the load balancer to be created.
On the load balancing screen, under the Backend column for your new load balancer, you should see a green checkmark showing that the new load balancer is healthy.
gcloud
Use the following gcloud
command to add a primary backend
to an existing external passthrough Network Load Balancer's backend service.
gcloud compute backend-services add-backend BACKEND_SERVICE_NAME \ --instance-group INSTANCE_GROUP_NAME \ --instance-group-zone INSTANCE_GROUP_ZONE \ --region REGION
where:
BACKEND_SERVICE_NAME
is the name of the load balancer's backend service. For the example, usenetwork-lb-backend-service
.INSTANCE_GROUP_NAME
is the name of the instance group to add as a primary backend. For the example, useig-d
.INSTANCE_GROUP_ZONE
is the zone where the instance group is defined. For the example, useus-west1-c
.REGION
is the region of the load balancer. For the example, useus-west1
.
Adding a failover backend
You can use this procedure as a template for how to add an unmanaged instance
group to an existing external passthrough Network Load Balancer's backend service as a
failover backend. For the example configuration, this procedure shows you how to
add instance group ig-b
as a failover backend to the network-lb
load balancer.
Console
Edit the load balancer configuration to add a primary backend.
In the Google Cloud console, go to the Load balancing page.
Click on the load balancer you want to modify.
Click Edit.
Click Backend configuration and make the following changes:
- Under Backends, click Add Backend.
- From the dropdown, select the instance group to be added as a
failover backend. In this case,
ig-b
. - Select the Use this instance group as a failover group for backup checkbox.
- Click Done.
- Verify that there is a blue check mark next to Backend configuration before continuing.
Review the configuration
- Click the Review and finalize button and confirm that the new primary backend is visible under Backend.
If the settings are correct, click Create. It takes a few minutes for the load balancer to be created.
On the load balancing screen, under the Backend column for your new load balancer, you should see a green checkmark showing that the new load balancer is healthy.
gcloud
Use the following gcloud
command to add a failover backend
to an existing external passthrough Network Load Balancer's backend service.
gcloud compute backend-services add-backend BACKEND_SERVICE_NAME \ --instance-group INSTANCE_GROUP_NAME \ --instance-group-zone INSTANCE_GROUP_ZONE \ --region REGION \ --failover
where:
BACKEND_SERVICE_NAME
is the name of the load balancer's backend service. For the example, usenetwork-lb-backend-service
.INSTANCE_GROUP_NAME
is the name of the instance group to add as a failover backend. For the example, useig-b
.INSTANCE_GROUP_ZONE
is the zone where the instance group is defined. For the example, useus-west1-a
.REGION
is the region of the load balancer. For the example, useus-west1
.
Converting a primary or failover backend
You can use convert a primary backend to a failover backend, or vice versa, without having to remove the instance group from the external passthrough Network Load Balancer's backend service.
gcloud
Use the following gcloud
command to convert an existing primary backend to
a failover backend:
gcloud compute backend-services update-backend BACKEND_SERVICE_NAME \ --instance-group INSTANCE_GROUP_NAME \ --instance-group-zone INSTANCE_GROUP_ZONE \ --region REGION \ --failover
Use the following gcloud
command to convert an existing failover backend to
a primary backend:
gcloud compute backend-services update-backend BACKEND_SERVICE_NAME \ --instance-group INSTANCE_GROUP_NAME \ --instance-group-zone INSTANCE_GROUP_ZONE \ --region REGION \ --no-failover
where:
BACKEND_SERVICE_NAME
is the name of the load balancer's backend service.INSTANCE_GROUP_NAME
is the name of the instance group.INSTANCE_GROUP_ZONE
is the zone where the instance group is defined.REGION
is the region of the load balancer.
Configuring failover policies
This section describes how to manage a failover policy for an external passthrough Network Load Balancer's backend service. A failover policy consists of the:
- Failover ratio
- Dropping traffic when all backend VMs are unhealthy
- Connection draining on failover
For more information on the parameters of a failover policy, see:
- Setting a failover ratio
- Dropping traffic when there is no healthy VM
- Draining connections on failover and failback
Defining a failover policy
The following instructions describe how to define the failover policy for an existing external passthrough Network Load Balancer.
gcloud
To define a failover policy using the gcloud CLI, update the load balancer's backend service:
gcloud compute backend-services update BACKEND_SERVICE_NAME \ --region REGION \ --failover-ratio FAILOVER_RATIO \ --connection-drain-on-failover \ --connection-draining-timeout=CONNECTION_DRAINING_TIMEOUT \ --drop-traffic-if-unhealthy
where:
BACKEND_SERVICE_NAME
is the name of the load balancer's backend service. For the example, usenetwork-lb
.REGION
is the region of the load balancer. For the example, useus-west1
.FAILOVER_RATIO
is the failover ratio. Possible values are between0.0
and1.0
, inclusive. For the example, use0.75
.CONNECTION_DRAINING_TIMEOUT
allows TCP connections to persist, even on VMs that are no longer in the active pool, for up to the duration of the connection draining timeout.
Viewing a failover policy
The following instructions describe how to view the existing failover policy for an external passthrough Network Load Balancer.
gcloud
To list the failover policy settings using the gcloud CLI, use the following command. Undefined settings in a failover policy use the default failover policy values.
gcloud compute backend-services describe BACKEND_SERVICE_NAME \ --region REGION \ --format="get(failoverPolicy)"
where:
BACKEND_SERVICE_NAME
is the name of the load balancer's backend service. For the example, usenetwork-lb-backend-service
.REGION
is the region of the load balancer. For the example, useus-west1
.
What's next
- See External passthrough Network Load Balancer overview for important fundamentals.
- See Failover concepts for external passthrough Network Load Balancers for important information about failover.
- See Set up an external passthrough Network Load Balancer for an example configuration.