Migrate global external proxy Network Load Balancers to dual-stack backends

Preview

This product or feature is subject to the "Pre-GA Offerings Terms" in the General Service Terms section of the Service Specific Terms. Pre-GA products and features are available "as is" and might have limited support. For more information, see the launch stage descriptions.

This document shows you how to migrate global external proxy Network Load Balancer resources and backends from IPv4 only (single-stack) to IPv4 and IPv6 (dual-stack).

The instructions in this document apply to both SSL proxy and TCP proxy global external proxy Network Load Balancers.

You must first migrate from IPv4 only to IPv4 and IPv6 (dual-stack) backends. You can later switch the IP address selection policy of the backend service to specify the traffic type that is sent from the GFE to your backends. For more information, see configure the IP address selection policy.

Note: You cannot update the IP stack type of the subnet from IPv4 and IPv6 (dual-stack) to IPv4 only. Downgrading your subnet from dual-stack to IPv4 only addresses is not allowed.

About the migration process

The migration process requires you to update the following load balancer resources:

• Subnet. The IP stack type of the subnet can be updated to support IPv4 only (single-stack) or IPv4 and IPv6 (dual-stack). You cannot downgrade your subnet from dual-stack to single-stack addresses. To update, see Update the subnet.
• Backends. The IP stack type of the following backends can be updated to support IPv4 only (single-stack) or IPv4 and IPv6 (dual-stack):

  • All instance group backends: One or more managed, unmanaged, or a combination of managed and unmanaged instance group backends.

    To update, see Update the VM instance or templates

  • All zonal NEGs: One or more GCE_VM_IP_PORT type zonal NEGs.

    To update, see Update the zonal NEG

• Firewall rules. Create a firewall rule to allow traffic from IPv6 health check probes to reach backends. To create, see Create IPv6 health check firewall rule
• Backend service. The IP address selection policy of the backend service can be updated to specify the traffic type that is sent from the GFE to your backends. To update, see Update the backend service.
  Note: The IP address selection policy of the backend service can be set to Only IPv6, but the IP stack type of the backends can be only dual stack. IPv6 only backends are not supported.
• Forwarding rule. Create a forwarding rule for IPv6

There is no validation to check if you have updated all the required resources. After you update all the resources, the traffic should flow to the backends and you can check the logs and verify that the migration is complete.

Identify the resources to migrate

1. To list all the subnets, run the following command in Cloud Shell:

   gcloud compute networks subnets list
   

   Note the name of the subnet with IPv4 only stack to migrate to dual-stack. This name is referred to later as SUBNET. The VPC network is referred to later as NETWORK.

2. To list all the backend services, run the following command in Cloud Shell:

   gcloud beta compute backend-services list
   

   Note the name of the backend service to migrate to dual-stack. This name is referred to later as BACKEND_SERVICE.

3. To list all the VM instance and instance templates, run the following command in Cloud Shell:

   gcloud compute instances list
   

   gcloud compute instance-templates list
   

   Note the name of the instances and instance templates to migrate to dual-stack. This name is referred to later as VM_INSTANCE and INSTANCE_TEMPLATES.

4. To list all the zonal network endpoint groups (NEGs), run the following command in Cloud Shell:

   gcloud compute network-endpoint-groups list
   

   Note the name of the network endpoint groups to migrate to dual stack. This name is referred to later as ZONAL_NEG.

5. To list all the target SSL proxies, run the following command in Cloud Shell:

   gcloud compute target-ssl-proxies list
   

   Note the name of the target proxy associated with your load balancer. This name is referred to later as TARGET_PROXY.

6. To list all the target TCP proxies, run the following command in Cloud Shell:

   gcloud compute target-tcp-proxies list
   

   Note the name of the target proxy associated with your load balancer. This name is referred to later as TARGET_PROXY.

Migrate from IPv4 to IPv4 and IPv6 (dual stack)

This section describes the procedure to migrate your load balancer and backends using IPv4 only addresses to IPv4 and IPv6 (dual stack) addresses.

Prerequisites

Before you start, you must have already set up an SSL proxy or TCP proxy global external proxy Network Load Balancer with IPv4 only stack with instance group or zonal NEG backends.

Refer to the following documentation:

• Set up a global external proxy Network Load Balancer(SSL proxy) with VM instance group backends
• Set up a global external proxy Network Load Balancer(TCP proxy) with VM instance group backends

You can run the following command to list the stack type information of all VM instances in a project:

gcloud

gcloud compute instances list \
  --format= \
  "table(
name,
zone.basename(),
networkInterfaces[].stackType.notnull().list(),
networkInterfaces[].ipv6AccessConfigs[0].externalIpv6.notnull().list():label=EXTERNAL_IPV6,
networkInterfaces[].ipv6Address.notnull().list():label=INTERNAL_IPV6)"

Update the subnet

Dual-stack subnets are supported on custom mode VPC networks only. Dual-stack subnets are not supported on auto mode VPC networks or legacy networks.

To update the network to the dual-stack setting, follow these steps:

1. If you are using an auto mode VPC network, you must first convert the auto mode VPC network to custom mode.

   Note: If you are using the default network, you must convert it to custom mode VPC network.

2. To enable IPv6, see Change a subnet's stack type to dual stack.

   Note: The IPv6 access type of the subnet must be External.

Update the VM instance or templates

You cannot edit VM instances that are part of a managed or an unmanaged instance group. To update the VM instances to dual stack, follow these steps:

1. Delete specific instances from a group
2. Create a dual-stack VM.
3. Creating instances with specific names in MIGs

You can't update an existing instance template. If you need to make changes, you can create another template with similar properties. To update the VM instance templates to dual stack, follow these steps:

Console

1. In the Google Cloud console, go to the Instance templates page.

   Go to Instance templates

   1. Click the instance template that you want to copy and update.
   2. Click Create similar.
   3. Expand the Advanced options section.
   4. For Network tags, enter allow-health-check-ipv6.
   5. In the Network interfaces section, click Add a network interface.
   6. In the Network list, select the custom mode VPC network.
   7. In the Subnetwork list, select SUBNET.
   8. For IP stack type, select IPv4 and IPv6 (dual-stack).
   9. Click Create.

2. Starting a basic rolling update on the managed instance group MIG associated with the load balancer.

Update the zonal NEG

Zonal NEG endpoints cannot be edited. You must delete the IPv4 endpoints and create a new dual-stack endpoint with both IPv4 and IPv6 addresses.

To set up a zonal NEG (with GCE_VM_IP_PORT type endpoints) in the REGION_A region, first create the VMs in the GCP_NEG_ZONE zone. Then add the VM network endpoints to the zonal NEG.

Create VMs

Console

1. In the Google Cloud console, go to the VM instances page.

   Go to VM instances

2. Click Create instance.

3. Set the Name to vm-a1.

4. For the Region, choose REGION_A, and choose any value for the Zone field. This zone is referred to as GCP_NEG_ZONE in this procedure.

5. In the Boot disk section, ensure that the Debian operating system and the 10 (buster) version are selected for the boot disk options. Click Choose to change the image if necessary.

6. Expand the Advanced options section and make the following changes:

   • Expand the Networking section.
   • In the Network tags field, enter allow-health-check.
   • In the Network interfaces section, make the following changes:
     • Network: NETWORK
     • Subnet: SUBNET
     • IP stack type: IPv4 and IPv6 (dual-stack)
   • Click Done.

   • Click Management. In the Startup script field, copy and paste the following script contents.

     #! /bin/bash
     apt-get update
     apt-get install apache2 -y
     a2ensite default-ssl
     a2enmod ssl
     vm_hostname="$(curl -H "Metadata-Flavor:Google" \
     http://metadata.google.internal/computeMetadata/v1/instance/name)"
     echo "Page served from: $vm_hostname" | \
     tee /var/www/html/index.html
     systemctl restart apache2
     

7. Click Create.

8. Repeat the following steps to create a second VM, using the following name and zone combination:

   • Name: vm-a2, zone: GCP_NEG_ZONE

gcloud

Create the VMs by running the following command two times, using these combinations for the name of the VM and its zone. The script contents are identical for both VMs.

• VM_NAME of vm-a1 and any GCP_NEG_ZONE zone of your choice.

• VM_NAME of vm-a2 and the same GCP_NEG_ZONE zone.

  gcloud compute instances create VM_NAME \
      --zone=GCP_NEG_ZONE \
      --stack-type=IPV4_IPV6 \
      --image-family=debian-10 \
      --image-project=debian-cloud \
      --tags=allow-health-check \
      --subnet=SUBNET \
      --metadata=startup-script='#! /bin/bash
        apt-get update
        apt-get install apache2 -y
        a2ensite default-ssl
        a2enmod ssl
        vm_hostname="$(curl -H "Metadata-Flavor:Google" \
        http://metadata.google.internal/computeMetadata/v1/instance/name)"
        echo "Page served from: $vm_hostname" | \
        tee /var/www/html/index.html
        systemctl restart apache2'
  

Add endpoints to the zonal NEG

Console

To add endpoints to the zonal NEG:

1. In the Google Cloud console, go to the Network endpoint groups page.

   Go to Network endpoint groups

2. In the Name list, click the name of the network endpoint group (ZONAL_NEG). You see the Network endpoint group details page.

3. In the Network endpoints in this group section, select the previously created NEG endpoint. Click Remove endpoint.

4. In the Network endpoints in this group section, click Add network endpoint.

5. Select the VM instance.

6. In the Network interface section, the name, zone, and subnet of the VM is displayed.

7. In the IPv4 address field, enter the IPv4 address of the new network endpoint.

8. In the IPv6 address field, enter the IPv6 address of the new network endpoint.

   Note: A backend service with multiple endpoints must have unique IPv6 addresses. The endpoints can be in different subnets, but the same IPv6 address cannot be used for multiple endpoints.

9. Select the Port type.

   1. If you select Default, the endpoint uses the default port 80 for all endpoints in the network endpoint group. This is sufficient for our example because the Apache server is serving requests at port 80.
   2. If you select Custom, enter the Port number for the endpoint to use.

10. To add more endpoints, click Add network endpoint and repeat the previous steps.

11. After you add all the endpoints, click Create.

gcloud

1. Add endpoints (GCE_VM_IP_PORT endpoints) to ZONAL_NEG.

   Note: A backend service with multiple endpoints must have unique IPv6 addresses. The endpoints can be in different subnets, but the same IPv6 address cannot be used for multiple endpoints.

   gcloud compute network-endpoint-groups update ZONAL_NEG \
       --zone=GCP_NEG_ZONE \
       --add-endpoint='instance=vm-a1,ip=IPv4_ADDRESS, \
         ipv6=IPv6_ADDRESS,port=80' \
       --add-endpoint='instance=vm-a2,ip=IPv4_ADDRESS, \
         ipv6=IPv6_ADDRESS,port=80'
   

Replace the following:

IPv4_ADDRESS: IPv4 address of the network endpoint. The IPv4 must belong to a VM in Compute Engine (either the primary IP or as part of an aliased IP range). If the IP address is not specified, then the primary IP address for the VM instance in the network that the network endpoint group belongs to is used.

IPv6_ADDRESS: IPv6 address of the network endpoint. The IPv6 address must belong to a VM instance in the network that the network endpoint group belongs (external IPv6 address).

Create an IPv6 health check firewall rule

Ensure that you have an ingress rule that is applicable to the instances being load balanced and that allows traffic from the Google Cloud health checking systems (2600:2d00:1:b029::/64,2600:2d00:1:1::/64). This example uses the target tag allow-health-check-ipv6 to identify the VM instances to which it applies.

Without this firewall rule, the default deny ingress rule blocks incoming IPv6 traffic to the backend instances.

Note: You must create a firewall rule to allow health checks from the IP ranges of Google Cloud probe systems. For more information, see probe IP ranges.

Console

1. In the Google Cloud console, go to the Firewall policies page.

   Go to Firewall policies

2. To allow IPv6 subnet traffic, click Create firewall rule again and enter the following information:

   • Name: fw-allow-lb-access-ipv6
   • Network: NETWORK
   • Priority: 1000
   • Direction of traffic: ingress
   • Targets: Specified target tags
   • Target tags: allow-health-check-ipv6
   • Source filter: IPv6 ranges
   • Source IPv6 ranges: 2600:2d00:1:b029::/64,2600:2d00:1:1::/64
   • Protocols and ports: Allow all

3. Click Create.

gcloud

1. Create the fw-allow-lb-access-ipv6 firewall rule to allow communication with the subnet:

   gcloud compute firewall-rules create fw-allow-lb-access-ipv6 \
       --network=NETWORK \
       --action=allow \
       --direction=ingress \
       --target-tags=allow-health-check-ipv6 \
       --source-ranges=2600:2d00:1:b029::/64,2600:2d00:1:1::/64 \
       --rules=all
   

Update the backend service and create a forwarding rule for IPv6

You can update the existing BACKEND_SERVICE to support dual stack, this section provides instructions to update the backend service with the IP address selection policy as Prefer IPv6 and add a zonal NEG backend.

If you have an instance group associated with your backend service, you have to update the backend service.

Console

1. In the Google Cloud console, go to the Load balancing page.

   Go to Load balancing

2. Click the name of the load balancer.

3. Click Edit.

Configure the backend service for IPv6

1. Click Backend configuration.
2. For Backend type, select Zonal network endpoint group.
3. In the IP address selection policy list, select Prefer IPv6.
4. In the Protocol field:
   • For TCP proxy, select TCP.
   • For SSL proxy, select SSL.
5. In the Backends section, click Add a backend.
6. In the New Backend panel, do the following:
   • In the network endpoint group list, select ZONAL_NEG.
   • In the Maximum connections field, enter 10.
7. Click Done.
8. In the Health check list, select an HTTP health check.

Configure the IPv6 forwarding rule

1. Click Frontend configuration.
2. Click Add frontend IP and port.
3. In the Name field, enter a name for the forwarding rule.
4. In the Protocol field:
   • For TCP proxy, select TCP.
   • For SSL proxy, select SSL.
5. Set IP version to IPv6.
6. For SSL proxy, in the Certificates list, select a certificate.
7. Click Done.
8. Click Update.

gcloud

1. Create the backend service for HTTP traffic:

   gcloud beta compute backend-services update BACKEND_SERVICE \
      --protocol=[SSL|TCP] \
      --port-name=http \
      --ip-address-selection-policy=PREFER_IPV6 \
      --health-checks=HEALTH_CHECK \
      --global
   

2. Add zonal NEGs as the backend to the backend service.

   gcloud beta compute backend-services add-backend BACKEND_SERVICE \
      --network-endpoint-group=ZONAL_NEG \
      --global
   

3. Reserve an external IPv6 address that your customers use to reach your load balancer.

   gcloud compute addresses create lb-ipv6-1 \
      --ip-version=IPV6 \
      --network-tier=PREMIUM \
      --global
   

4. For SSL proxy, create a forwarding rule for the backend service.

   gcloud beta compute target-ssl-proxies create TARGET_PROXY \
      --backend-service=BACKEND_SERVICE \
      --ssl-certificates=[SSL_CERT_1][,[SSL_CERT_2],...] \
      --ssl-policy=my-ssl-policy \
      --proxy-header=NONE
   

   gcloud beta compute forwarding-rules create  \
      --load-balancing-scheme=EXTERNAL_MANAGED \
      --network-tier=PREMIUM \
      --address=lb-ipv6-1 \
      --global \
      --target-ssl-proxy=TARGET_PROXY \
      --ports=443
   

5. For TCP proxy, create a forwarding rule for the backend service.

   gcloud beta compute target-tcp-proxies create TARGET_PROXY \
      --backend-service=BACKEND_SERVICE \
      --proxy-header=NONE
   

   gcloud beta compute forwarding-rules create  \
      --load-balancing-scheme=EXTERNAL_MANAGED \
      --network-tier=PREMIUM \
      --address=lb-ipv6-1 \
      --global \
      --target-tcp-proxy=TARGET_PROXY \
      --ports=443
   

Update the backend service for an instance group

If you have an instance group associated with your backend service, you have to update the backend service as follows:

Console

1. In the Google Cloud console, go to the Load balancing page.

   Go to Load balancing

2. Click the name of the load balancer.

3. Click Edit.

Configure the backend service

1. Click Backend configuration.
2. For Backend type, select Instance group.
3. In the IP address selection policy list, select Prefer IPv6.
4. In the Protocol field:
   • For TCP proxy, select TCP.
   • For SSL proxy, select SSL.
5. If you have updated the VM instance or templates to dual stack, the Backends section, need not be updated again.
6. Select a Health check.
7. Click Update.

gcloud

1. Create the backend service for HTTP traffic:

   gcloud beta compute backend-services update BACKEND_SERVICE \
      --protocol=[SSL|TCP] \
      --port-name=http \
      --ip-address-selection-policy=PREFER_IPV6 \
      --health-checks=HEALTH_CHECK \
      --global
   

2. Add instance group as the backend to the backend service. If you have updated the VM instance or templates to dual stack, you can skip this step.

   gcloud beta compute backend-services add-backend BACKEND_SERVICE \
      --instance-group=INSTANCE_GROUP \
      --global
   

Configure the IP address selection policy

This step is optional and provides instructions to configure the IP address selection policy of the backend service. After you have migrated your backends to dual stack, you can use the IP address selection policy to specify the traffic type that is sent from the GFE to your backends.

Replace IP_ADDRESS_SELECTION_POLICY with any of the following values:

IP address selection policy	Description
Only IPv4	Only send IPv4 traffic to the backends of the backend service, regardless of traffic from the client to the GFE. Only IPv4 health checks are used to check the health of the backends.
Prefer IPv6	Prioritize the backend's IPv6 connection over the IPv4 connection (provided there is a healthy backend with IPv6 addresses). The health checks periodically monitor the backends' IPv6 and IPv4 connections. The GFE first attempts the IPv6 connection; if the IPv6 connection is broken or slow, the GFE uses happy eyeballs to fall back and connect to IPv4. Even if one of the IPv6 or IPv4 connections is unhealthy, the backend is still treated as healthy, and both connections can be tried by the GFE, with happy eyeballs ultimately selecting which one to use.
Only IPv6	Only send IPv6 traffic to the backends of the backend service, regardless of traffic from the client to the proxy. Only IPv6 health checks are used to check the health of the backends. There is no validation to check if the backend traffic type matches the IP address selection policy. For example, if you have IPV4 backends and select Only IPv6 as the IP address selection policy, you won't observe configuration errors but the traffic won't flow to your backends.

Console

1. In the Google Cloud console, go to the Load balancing page.

   Go to Load balancing

2. Click the name of the load balancer.

3. Click Edit.

4. Click Backend configuration.

5. In the Backend service field, select BACKEND_SERVICE.

6. The Backend type must be Zonal network endpoint group or Instance group.

7. In the IP address selection policy list, select IP_ADDRESS_SELECTION_POLICY.

8. Click Done.

gcloud

1. Update the backend service:

   gcloud beta compute backend-services update BACKEND_SERVICE \
      --load-balancing-scheme=EXTERNAL_MANAGED \
      --protocol=HTTP \
      --ip-address-selection-policy=IP_ADDRESS_SELECTION_POLICY \
      --global
   

Test your load balancer

Test the load balancer to confirm that the migration is successful and the incoming traffic is reaching the backends as expected.

Look up the external IP addresses

Console

1. In the Google Cloud console, go to the Load balancing page.

   Go to Load balancing

2. Click the name of the load balancer.

3. In the Frontend section, two load balancer IP addresses are displayed. In this procedure, the IPv4 address is referred to as IP_ADDRESS_IPV4 and the IPv6 address is referred as IP_ADDRESS_IPV6.

4. In the Backends section, when the IP address selection policy is Prefer IPv6 two health check status are displayed for the backends.

Test traffic sent to your instances

In this example, requests from the curl command are distributed randomly to the backends.

1. Repeat the following commands a few times until you see all the backend VMs responding:

   curl -m1 IP_ADDRESS_IPV4:PORT
   

   curl -m1 IP_ADDRESS_IPV6:PORT
   

   For example, if the IPv6 address is [fd20:1db0:b882:802:0:46:0:0]:80, the command looks similar to this::

   curl -m1 [fd20:1db0:b882:802:0:46:0:0]:80
   

Check the logs

Every log entry captures the destination IPv4 and IPv6 address for the backend. Because we support dual stack, it is important to observe the IP address used by the backend.

You can check that traffic is going to IPv6 or failing back to IPv4 by viewing the logs.

The logs contain the backend_ip address associated with the backend. By examining the logs and comparing the destination IPv4 or IPv6 address of the backend_ip, you can confirm which IP address is used.

Limitations

When the IP address selection policy is configured as IPV6_ONLY, you can still configure IPv4 only backends. Such a configuration results in no healthy backends, clients get response code 503, and there is no upstream traffic. By examining the logs, you can see the statusDetails message with failed_to_pick_backend.