Set up a cross-region internal Application Load Balancer with VM instance group backends

This document provides instructions for configuring a cross-region internal Application Load Balancer for your services that run on Compute Engine virtual machine (VM) instances.

Before you begin

Before following this guide, familiarize yourself with the following:

Set up an SSL certificate resource

Create a Certificate Manager SSL certificate resource as described in the following:

We recommend using a Google-managed certificate.

Permissions

To follow this guide, you must be able to create instances and modify a network in a project. You must be either a project owner or editor, or you must have all of the following Compute Engine IAM roles.

Task Required role
Create networks, subnets, and load balancer components Compute Network Admin
Add and remove firewall rules Compute Security Admin
Create instances Compute Instance Admin

For more information, see the following guides:

Setup overview

You can configure load balancer as shown in the following diagram:

Cross-region internal Application Load Balancer high availability deployment.
Cross-region internal Application Load Balancer high availability deployment (click to enlarge).

As shown in the diagram, this example creates a cross-region internal Application Load Balancer in a VPC network, with one backend service and two backend managed instance groups in the REGION_A and REGION_B regions.

The diagram shows the following:

  1. A VPC network with the following subnets:

    • Subnet SUBNET_A and a proxy-only subnet in REGION_A.
    • Subnet SUBNET_B and a proxy-only subnet in REGION_B.

    You must create proxy-only subnets in each region of a VPC network where you use cross-region internal Application Load Balancers. The region's proxy-only subnet is shared among all cross-region internal Application Load Balancers in the region. Source addresses of packets sent from the load balancer to your service's backends are allocated from the proxy-only subnet. In this example, the proxy-only subnet for the region REGION_A has a primary IP address range of 10.129.0.0/23 and for REGION_B has a primary IP address range of 10.130.0.0/23 which is the recommended subnet size.

  2. High availability setup has managed instance group backends for Compute Engine VM deployments in REGION_A and REGION_B regions. If backends in one region happen to be down, traffic fails over to the other region.

  3. A global backend service that monitors the usage and health of backends.

  4. A global URL map that parses the URL of a request and forwards requests to specific backend services based on the host and path of the request URL.

  5. A global target HTTP or HTTPS proxy, receives a request from the user and forwards it to the URL map. For HTTPS, configure a global SSL certificate resource. The target proxy uses the SSL certificate to decrypt SSL traffic if you configure HTTPS load balancing. The target proxy can forward traffic to your instances by using HTTP or HTTPS.

  6. Global forwarding rules, has the regional internal IP address of your load balancer, to forward each incoming request to the target proxy.

    The internal IP address associated with the forwarding rule can come from a subnet in the same network and region as the backends. Note the following conditions:

    • The IP address can (but does not need to) come from the same subnet as the backend instance groups.
    • The IP address must not come from a reserved proxy-only subnet that has its --purpose flag set to GLOBAL_MANAGED_PROXY.
    • If you want to use the same internal IP address with multiple forwarding rules, set the IP address --purpose flag to SHARED_LOADBALANCER_VIP.

  7. Optional: Configure DNS routing policies of type GEO to route client traffic to the load balancer VIP in the region closest to the client.

Configure the network and subnets

Within the VPC network, configure a subnet in each region where your backends are configured. In addition, configure a proxy-only-subnet in each region that you want to configure the load balancer.

This example uses the following VPC network, region, and subnets:

  • Network. The network is a custom mode VPC network named NETWORK.

  • Subnets for backends.

    • A subnet named SUBNET_A in the REGION_A region uses 10.1.2.0/24 for its primary IP range.
    • A subnet named SUBNET_B in the REGION_B region uses 10.1.3.0/24 for its primary IP range.

  • Subnets for proxies.

    • A subnet named PROXY_SN_A in the REGION_A region uses 10.129.0.0/23 for its primary IP range.
    • A subnet named PROXY_SN_B in the REGION_B region uses 10.130.0.0/23 for its primary IP range.

Cross-region internal Application Load Balancers can be accessed from any region within the VPC. So clients from any region can globally access your load balancer backends.

Configure the backend subnets

Console

  1. In the Google Cloud console, go to the VPC networks page.

    Go to VPC networks

  2. Click Create VPC network.

  3. Provide a Name for the network.

  4. In the Subnets section, set the Subnet creation mode to Custom.

  5. Create a subnet for the load balancer's backends. In the New subnet section, enter the following information:

    • Provide a Name for the subnet.
    • Select a Region: REGION_A
    • Enter an IP address range: 10.1.2.0/24

  6. Click Done.

  7. Click Add subnet.

  8. Create a subnet for the load balancer's backends. In the New subnet section, enter the following information:

    • Provide a Name for the subnet.
    • Select a Region: REGION_B
    • Enter an IP address range: 10.1.3.0/24

  9. Click Done.

  10. Click Create.

gcloud

  1. Create the custom VPC network with the gcloud compute networks create command:

    gcloud compute networks create NETWORK \
    --subnet-mode=custom

  2. Create a subnet in the NETWORK network in the REGION_A region with the gcloud compute networks subnets create command:

    gcloud compute networks subnets create SUBNET_A \
    --network=NETWORK \
    --range=10.1.2.0/24 \
    --region=REGION_A

  3. Create a subnet in the NETWORK network in the REGION_B region with the gcloud compute networks subnets create command:

    gcloud compute networks subnets create SUBNET_B \
    --network=NETWORK \
    --range=10.1.3.0/24 \
    --region=REGION_B

Terraform

To create the VPC network, use the google_compute_network resource.

resource "google_compute_network" "default" {
  auto_create_subnetworks = false
  name                    = "lb-network-crs-reg"
  provider                = google-beta
}

To create the VPC subnets in the lb-network-crs-reg network, use the google_compute_subnetwork resource.

resource "google_compute_subnetwork" "subnet_a" {
  provider      = google-beta
  ip_cidr_range = "10.1.2.0/24"
  name          = "lbsubnet-uswest1"
  network       = google_compute_network.default.id
  region        = "us-west1"
}
resource "google_compute_subnetwork" "subnet_b" {
  provider      = google-beta
  ip_cidr_range = "10.1.3.0/24"
  name          = "lbsubnet-useast1"
  network       = google_compute_network.default.id
  region        = "us-east1"
}

API

Make a POST request to the networks.insert method. Replace PROJECT_ID with your project ID.

POST https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/global/networks

{
 "routingConfig": {
   "routingMode": "regional"
 },
 "name": "NETWORK",
 "autoCreateSubnetworks": false
}

Make a POST request to the subnetworks.insert method. Replace PROJECT_ID with your project ID.

POST https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/regions/REGION_A/subnetworks

{
 "name": "SUBNET_A",
 "network": "projects/PROJECT_ID/global/networks/lb-network-crs-reg",
 "ipCidrRange": "10.1.2.0/24",
 "region": "projects/PROJECT_ID/regions/REGION_A",
}

Make a POST request to the subnetworks.insert method. Replace PROJECT_ID with your project ID.

POST https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/regions/REGION_B/subnetworks

{
 "name": "SUBNET_B",
 "network": "projects/PROJECT_ID/global/networks/NETWORK",
 "ipCidrRange": "10.1.3.0/24",
 "region": "projects/PROJECT_ID/regions/REGION_B",
}

Configure the proxy-only subnet

A proxy-only subnet provides a set of IP addresses that Google Cloud uses to run Envoy proxies on your behalf. The proxies terminate connections from the client and create new connections to the backends.

This proxy-only subnet is used by all Envoy-based regional load balancers in the same region as the VPC network. There can only be one active proxy-only subnet for a given purpose, per region, per network.

Console

If you're using the Google Cloud console, you can wait and create the proxy-only subnet later on the Load balancing page.

If you want to create the proxy-only subnet now, use the following steps:

  1. In the Google Cloud console, go to the VPC networks page.

    Go to VPC networks

  2. Click the name of the VPC network.
  3. On the Subnet tab, click Add subnet.
  4. Provide a Name for the proxy-only subnet.
  5. Select a Region: REGION_A
  6. In the Purpose list, select Cross-region Managed Proxy.
  7. In the IP address range field, enter 10.129.0.0/23.
  8. Click Add.

Create the proxy-only subnet in REGION_B

  1. On the Subnet tab, click Add subnet.
  2. Provide a Name for the proxy-only subnet.
  3. Select a Region: REGION_B
  4. In the Purpose list, select Cross-region Managed Proxy.
  5. In the IP address range field, enter 10.130.0.0/23.
  6. Click Add.

gcloud

Create the proxy-only subnets with the gcloud compute networks subnets create command.

    gcloud compute networks subnets create PROXY_SN_A \
        --purpose=GLOBAL_MANAGED_PROXY \
        --role=ACTIVE \
        --region=REGION_A \
        --network=NETWORK \
        --range=10.129.0.0/23
    
    gcloud compute networks subnets create PROXY_SN_B \
        --purpose=GLOBAL_MANAGED_PROXY \
        --role=ACTIVE \
        --region=REGION_B \
        --network=NETWORK \
        --range=10.130.0.0/23

Terraform

To create the VPC proxy-only subnet in the lb-network-crs-reg network, use the google_compute_subnetwork resource.

resource "google_compute_subnetwork" "proxy_subnet_a" {
  provider      = google-beta
  ip_cidr_range = "10.129.0.0/23"
  name          = "proxy-only-subnet1"
  network       = google_compute_network.default.id
  purpose       = "GLOBAL_MANAGED_PROXY"
  region        = "us-west1"
  role          = "ACTIVE"
  lifecycle {
    ignore_changes = [ipv6_access_type]
  }
}
resource "google_compute_subnetwork" "proxy_subnet_b" {
  provider      = google-beta
  ip_cidr_range = "10.130.0.0/23"
  name          = "proxy-only-subnet2"
  network       = google_compute_network.default.id
  purpose       = "GLOBAL_MANAGED_PROXY"
  region        = "us-east1"
  role          = "ACTIVE"
  lifecycle {
    ignore_changes = [ipv6_access_type]
  }
}

API

Create the proxy-only subnets with the subnetworks.insert method, replacing PROJECT_ID with your project ID.

    POST https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/regions/REGION_A/subnetworks

    {
      "name": " PROXY_SN_A",
      "ipCidrRange": "10.129.0.0/23",
      "network": "projects/PROJECT_ID/global/networks/NETWORK",
      "region": "projects/PROJECT_ID/regions/REGION_A",
      "purpose": "GLOBAL_MANAGED_PROXY",
      "role": "ACTIVE"
    }
   
    POST https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/regions/REGION_B/subnetworks

    {
      "name": "PROXY_SN_B",
      "ipCidrRange": "10.130.0.0/23",
      "network": "projects/PROJECT_ID/global/networks/NETWORK",
      "region": "projects/PROJECT_ID/regions/REGION_B",
      "purpose": "GLOBAL_MANAGED_PROXY",
      "role": "ACTIVE"
    }

Configure firewall rules

This example uses the following firewall rules:

  • fw-ilb-to-backends. An ingress rule, applicable to the instances being load balanced, that allows incoming SSH connectivity on TCP port 22 from any address. You can choose a more restrictive source IP address range for this rule; for example, you can specify just the IP address ranges of the system from which you initiate SSH sessions. This example uses the target tag allow-ssh to identify the VMs that the firewall rule applies to.

  • fw-healthcheck. An ingress rule, applicable to the instances being load balanced, that allows all TCP traffic from the Google Cloud health checking systems (in 130.211.0.0/22 and 35.191.0.0/16). This example uses the target tag load-balanced-backend to identify the VMs that the firewall rule applies to.

  • fw-backends. An ingress rule, applicable to the instances being load balanced, that allows TCP traffic on ports 80, 443, and 8080 from the internal Application Load Balancer's managed proxies. This example uses the target tag load-balanced-backend to identify the VMs that the firewall rule applies to.

Without these firewall rules, the default deny ingress rule blocks incoming traffic to the backend instances.

The target tags define the backend instances. Without the target tags, the firewall rules apply to all of your backend instances in the VPC network. When you create the backend VMs, make sure to include the specified target tags, as shown in Creating a managed instance group.

Console

  1. In the Google Cloud console, go to the Firewall policies page.

    Go to Firewall policies

  2. Click Create firewall rule to create the rule to allow incoming SSH connections:

    • Name: fw-ilb-to-backends
    • Network: NETWORK
    • Direction of traffic: Ingress
    • Action on match: Allow
    • Targets: Specified target tags
    • Target tags: allow-ssh
    • Source filter: IPv4 ranges
    • Source IPv4 ranges: 0.0.0.0/0
    • Protocols and ports:
      • Choose Specified protocols and ports.
      • Select the TCP checkbox, and then enter 22 for the port number.

  3. Click Create.

  4. Click Create firewall rule a second time to create the rule to allow Google Cloud health checks:

    • Name: fw-healthcheck
    • Network: NETWORK
    • Direction of traffic: Ingress
    • Action on match: Allow
    • Targets: Specified target tags
    • Target tags: load-balanced-backend
    • Source filter: IPv4 ranges
    • Source IPv4 ranges: 130.211.0.0/22 and 35.191.0.0/16

    • Protocols and ports:

      • Choose Specified protocols and ports.
      • Select the TCP checkbox, and then enter 80 for the port number.

      As a best practice, limit this rule to just the protocols and ports that match those used by your health check. If you use tcp:80 for the protocol and port, Google Cloud can use HTTP on port 80 to contact your VMs, but it cannot use HTTPS on port 443 to contact them.

  5. Click Create.

  6. Click Create firewall rule a third time to create the rule to allow the load balancer's proxy servers to connect the backends:

    • Name: fw-backends
    • Network: NETWORK
    • Direction of traffic: Ingress
    • Action on match: Allow
    • Targets: Specified target tags
    • Target tags: load-balanced-backend
    • Source filter: IPv4 ranges
    • Source IPv4 ranges: 10.129.0.0/23 and 10.130.0.0/23
    • Protocols and ports:
      • Choose Specified protocols and ports.
      • Select the TCP checkbox, and then enter 80, 443, 8080 for the port numbers.

  7. Click Create.

gcloud

  1. Create the fw-ilb-to-backends firewall rule to allow SSH connectivity to VMs with the network tag allow-ssh. When you omit source-ranges, Google Cloud interprets the rule to mean any source.

    gcloud compute firewall-rules create fw-ilb-to-backends \
    --network=NETWORK \
    --action=allow \
    --direction=ingress \
    --target-tags=allow-ssh \
    --rules=tcp:22

  2. Create the fw-healthcheck rule to allow Google Cloud health checks. This example allows all TCP traffic from health check probers; however, you can configure a narrower set of ports to meet your needs.

    gcloud compute firewall-rules create fw-healthcheck \
    --network=NETWORK \
    --action=allow \
    --direction=ingress \
    --source-ranges=130.211.0.0/22,35.191.0.0/16 \
    --target-tags=load-balanced-backend \
    --rules=tcp

  3. Create the fw-backends rule to allow the internal Application Load Balancer's proxies to connect to your backends. Set source-ranges to the allocated ranges of your proxy-only subnet, for example, 10.129.0.0/23 and 10.130.0.0/23.

    gcloud compute firewall-rules create fw-backends \
    --network=NETWORK \
    --action=allow \
    --direction=ingress \
    --source-ranges=source-range \
    --target-tags=load-balanced-backend \
    --rules=tcp:80,tcp:443,tcp:8080

Terraform

To create the firewall rules, use the google_compute_firewall resource.

resource "google_compute_firewall" "fw_healthcheck" {
  name          = "gl7-ilb-fw-allow-hc"
  provider      = google-beta
  direction     = "INGRESS"
  network       = google_compute_network.default.id
  source_ranges = ["130.211.0.0/22", "35.191.0.0/16", "35.235.240.0/20"]
  allow {
    protocol = "tcp"
  }
}
resource "google_compute_firewall" "fw_ilb_to_backends" {
  name          = "fw-ilb-to-fw"
  provider      = google-beta
  network       = google_compute_network.default.id
  source_ranges = ["0.0.0.0/0"]
  allow {
    protocol = "tcp"
    ports    = ["22", "80", "443", "8080"]
  }
}
resource "google_compute_firewall" "fw_backends" {
  name          = "gl7-ilb-fw-allow-ilb-to-backends"
  direction     = "INGRESS"
  network       = google_compute_network.default.id
  source_ranges = ["10.129.0.0/23", "10.130.0.0/23"]
  target_tags   = ["http-server"]
  allow {
    protocol = "tcp"
    ports    = ["80", "443", "8080"]
  }
}

API

Create the fw-ilb-to-backends firewall rule by making a POST request to the firewalls.insert method, replacing PROJECT_ID with your project ID.

POST https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/global/firewalls

{
 "name": "fw-ilb-to-backends",
 "network": "projects/PROJECT_ID/global/networks/NETWORK",
 "sourceRanges": [
   "0.0.0.0/0"
 ],
 "targetTags": [
   "allow-ssh"
 ],
 "allowed": [
  {
    "IPProtocol": "tcp",
    "ports": [
      "22"
    ]
  }
 ],
"direction": "INGRESS"
}

Create the fw-healthcheck firewall rule by making a POST request to the firewalls.insert method, replacing PROJECT_ID with your project ID.

POST https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/global/firewalls

{
 "name": "fw-healthcheck",
 "network": "projects/PROJECT_ID/global/networks/NETWORK",
 "sourceRanges": [
   "130.211.0.0/22",
   "35.191.0.0/16"
 ],
 "targetTags": [
   "load-balanced-backend"
 ],
 "allowed": [
   {
     "IPProtocol": "tcp"
   }
 ],
 "direction": "INGRESS"
}

Create the fw-backends firewall rule to allow TCP traffic within the proxy subnet for the firewalls.insert method, replacing PROJECT_ID with your project ID.

POST https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/global/firewalls

{
 "name": "fw-backends",
 "network": "projects/PROJECT_ID/global/networks/NETWORK",
 "sourceRanges": [
   "10.129.0.0/23",
   "10.130.0.0/23"
 ],
 "targetTags": [
   "load-balanced-backend"
 ],
 "allowed": [
   {
     "IPProtocol": "tcp",
     "ports": [
       "80"
     ]
   },
 {
     "IPProtocol": "tcp",
     "ports": [
       "443"
     ]
   },
   {
     "IPProtocol": "tcp",
     "ports": [
       "8080"
     ]
   }
 ],
 "direction": "INGRESS"
}

Create a managed instance group

This section shows how to create a template and a managed instance group. The managed instance group provides VM instances running the backend servers of an example cross-region internal Application Load Balancer. For your instance group, you can define an HTTP service and map a port name to the relevant port. The backend service of the load balancer forwards traffic to the named ports. Traffic from clients is load balanced to backend servers. For demonstration purposes, backends serve their own hostnames.

Console

  1. In the Google Cloud console, go to the Instance templates page.

    Go to Instance templates

    1. Click Create instance template.
    2. For Name, enter gil7-backendeast1-template.
    3. Ensure that the Boot disk is set to a Debian image, such as Debian GNU/Linux 10 (buster). These instructions use commands that are only available on Debian, such as apt-get.
    4. Click Advanced options.
    5. Click Networking and configure the following fields:
      1. For Network tags, enter allow-ssh and load-balanced-backend.
      2. For Network interfaces, select the following:
        • Network: NETWORK
        • Subnet: SUBNET_B

    6. Click Management. Enter the following script into the Startup script field.

      #! /bin/bash
apt-get update
apt-get install apache2 -y
a2ensite default-ssl
a2enmod ssl
vm_hostname="$(curl -H "Metadata-Flavor:Google" \
http://169.254.169.254/computeMetadata/v1/instance/name)"
echo "Page served from: $vm_hostname" | \
tee /var/www/html/index.html
systemctl restart apache2

    7. Click Create.

    8. Click Create instance template.

    9. For Name, enter gil7-backendwest1-template.

    10. Ensure that the Boot disk is set to a Debian image, such as Debian GNU/Linux 10 (buster). These instructions use commands that are only available on Debian, such as apt-get.

    11. Click Advanced options.

    12. Click Networking and configure the following fields:

      1. For Network tags, enter allow-ssh and load-balanced-backend.
      2. For Network interfaces, select the following:
        • Network: NETWORK
        • Subnet: SUBNET_A

    13. Click Management. Enter the following script into the Startup script field.

      #! /bin/bash
apt-get update
apt-get install apache2 -y
a2ensite default-ssl
a2enmod ssl
vm_hostname="$(curl -H "Metadata-Flavor:Google" \
http://169.254.169.254/computeMetadata/v1/instance/name)"
echo "Page served from: $vm_hostname" | \
tee /var/www/html/index.html
systemctl restart apache2

    14. Click Create.

  2. In the Google Cloud console, go to the Instance groups page.

    Go to Instance groups

    1. Click Create instance group.
    2. Select New managed instance group (stateless). For more information, see Stateless or stateful MIGs.
    3. For Name, enter gl7-ilb-mig-a.
    4. For Location, select Single zone.
    5. For Region, select REGION_A.
    6. For Zone, select ZONE_A.
    7. For Instance template, select gil7-backendwest1-template.

    8. Specify the number of instances that you want to create in the group.

      For this example, specify the following options under Autoscaling:

      • For Autoscaling mode, select Off:do not autoscale.
      • For Maximum number of instances, enter 2.

      Optionally, in the Autoscaling section of the UI, you can configure the instance group to automatically add or remove instances based on instance CPU use.

    9. Click Create.

    10. Click Create instance group.

    11. Select New managed instance group (stateless). For more information, see Stateless or stateful MIGs.

    12. For Name, enter gl7-ilb-mig-b.

    13. For Location, select Single zone.

    14. For Region, select REGION_B.

    15. For Zone, select ZONE_B.

    16. For Instance template, select gil7-backendeast1-template.

    17. Specify the number of instances that you want to create in the group.

      For this example, specify the following options under Autoscaling:

      • For Autoscaling mode, select Off:do not autoscale.
      • For Maximum number of instances, enter 2.

      Optionally, in the Autoscaling section of the UI, you can configure the instance group to automatically add or remove instances based on instance CPU usage.

    18. Click Create.

gcloud

The gcloud CLI instructions in this guide assume that you are using Cloud Shell or another environment with bash installed.

  1. Create a VM instance template with HTTP server with the gcloud compute instance-templates create command.

    gcloud compute instance-templates create gil7-backendwest1-template \
  --region=REGION_A \
  --network=NETWORK \
  --subnet=SUBNET_A \
  --tags=allow-ssh,load-balanced-backend \
  --image-family=debian-10 \
  --image-project=debian-cloud \
  --metadata=startup-script='#! /bin/bash
    apt-get update
    apt-get install apache2 -y
    a2ensite default-ssl
    a2enmod ssl
    vm_hostname="$(curl -H "Metadata-Flavor:Google" \
    http://169.254.169.254/computeMetadata/v1/instance/name)"
    echo "Page served from: $vm_hostname" | \
    tee /var/www/html/index.html
    systemctl restart apache2'

    gcloud compute instance-templates create gil7-backendeast1-template \
    --region=REGION_B \
    --network=NETWORK \
    --subnet=SUBNET_B \
    --tags=allow-ssh,load-balanced-backend \
    --image-family=debian-10 \
    --image-project=debian-cloud \
    --metadata=startup-script='#! /bin/bash
      apt-get update
      apt-get install apache2 -y
      a2ensite default-ssl
      a2enmod ssl
      vm_hostname="$(curl -H "Metadata-Flavor:Google" \
      http://169.254.169.254/computeMetadata/v1/instance/name)"
      echo "Page served from: $vm_hostname" | \
      tee /var/www/html/index.html
      systemctl restart apache2'

  2. Create a managed instance group in the zone with the gcloud compute instance-groups managed create command.

    gcloud compute instance-groups managed create gl7-ilb-mig-a \
    --zone=ZONE_A \
    --size=2 \
    --template=gil7-backendwest1-template

    gcloud compute instance-groups managed create gl7-ilb-mig-b \
    --zone=ZONE_B \
    --size=2 \
    --template=gil7-backendeast1-template

Terraform

To create the instance template, use the google_compute_instance_template resource.

resource "google_compute_instance_template" "instance_template_a" {
  name         = "gil7-backendwest1-template"
  provider     = google-beta
  machine_type = "e2-small"
  region       = "us-west1"
  tags         = ["http-server"]

  network_interface {
    network    = google_compute_network.default.id
    subnetwork = google_compute_subnetwork.subnet_a.id
    access_config {
      # add external ip to fetch packages
    }
  }
  disk {
    source_image = "debian-cloud/debian-11"
    auto_delete  = true
    boot         = true
  }

  # install nginx and serve a simple web page
  metadata = {
    startup-script = <<-EOF1
      #! /bin/bash
      set -euo pipefail

      export DEBIAN_FRONTEND=noninteractive
      apt-get update
      apt-get install -y nginx-light jq

      NAME=$(curl -H "Metadata-Flavor: Google" "http://metadata.google.internal/computeMetadata/v1/instance/hostname")
      IP=$(curl -H "Metadata-Flavor: Google" "http://metadata.google.internal/computeMetadata/v1/instance/network-interfaces/0/ip")
      METADATA=$(curl -f -H "Metadata-Flavor: Google" "http://metadata.google.internal/computeMetadata/v1/instance/attributes/?recursive=True" | jq 'del(.["startup-script"])')

      cat <<EOF > /var/www/html/index.html
      <pre>
      Name: $NAME
      IP: $IP
      Metadata: $METADATA
      </pre>
      EOF
    EOF1
  }
  lifecycle {
    create_before_destroy = true
  }
}
resource "google_compute_instance_template" "instance_template_b" {
  name         = "gil7-backendeast1-template"
  provider     = google-beta
  machine_type = "e2-small"
  region       = "us-east1"
  tags         = ["http-server"]

  network_interface {
    network    = google_compute_network.default.id
    subnetwork = google_compute_subnetwork.subnet_b.id
    access_config {
      # add external ip to fetch packages
    }
  }
  disk {
    source_image = "debian-cloud/debian-11"
    auto_delete  = true
    boot         = true
  }

  # install nginx and serve a simple web page
  metadata = {
    startup-script = <<-EOF1
      #! /bin/bash
      set -euo pipefail

      export DEBIAN_FRONTEND=noninteractive
      apt-get update
      apt-get install -y nginx-light jq

      NAME=$(curl -H "Metadata-Flavor: Google" "http://metadata.google.internal/computeMetadata/v1/instance/hostname")
      IP=$(curl -H "Metadata-Flavor: Google" "http://metadata.google.internal/computeMetadata/v1/instance/network-interfaces/0/ip")
      METADATA=$(curl -f -H "Metadata-Flavor: Google" "http://metadata.google.internal/computeMetadata/v1/instance/attributes/?recursive=True" | jq 'del(.["startup-script"])')

      cat <<EOF > /var/www/html/index.html
      <pre>
      Name: $NAME
      IP: $IP
      Metadata: $METADATA
      </pre>
      EOF
    EOF1
  }
  lifecycle {
    create_before_destroy = true
  }
}

To create the managed instance group, use the google_compute_instance_group_manager resource.

resource "google_compute_region_instance_group_manager" "mig_a" {
  name     = "gl7-ilb-miga"
  provider = google-beta
  region   = "us-west1"
  version {
    instance_template = google_compute_instance_template.instance_template_a.id
    name              = "primary"
  }
  base_instance_name = "vm"
  target_size        = 2
}
resource "google_compute_region_instance_group_manager" "mig_b" {
  name     = "gl7-ilb-migb"
  provider = google-beta
  region   = "us-east1"
  version {
    instance_template = google_compute_instance_template.instance_template_b.id
    name              = "primary"
  }
  base_instance_name = "vm"
  target_size        = 2
}

API

Create the instance template with the instanceTemplates.insert method, replacing PROJECT_ID with your project ID.

POST https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/global/instanceTemplates

{
  "name":"gil7-backendwest1-template",
  "properties":{
     "machineType":"e2-standard-2",
     "tags":{
       "items":[
         "allow-ssh",
         "load-balanced-backend"
       ]
     },
     "metadata":{
        "kind":"compute#metadata",
        "items":[
          {
            "key":"startup-script",
            "value":"#! /bin/bash\napt-get update\napt-get install
            apache2 -y\na2ensite default-ssl\na2enmod ssl\n
            vm_hostname=\"$(curl -H \"Metadata-Flavor:Google\"           
            \\\nhttp://169.254.169.254/computeMetadata/v1/instance/name)\"\n
            echo \"Page served from: $vm_hostname\" | \\\ntee
            /var/www/html/index.html\nsystemctl restart apache2"
          }
        ]
     },
     "networkInterfaces":[
       {
         "network":"projects/PROJECT_ID/global/networks/NETWORK",
         "subnetwork":"regions/REGION_A/subnetworks/SUBNET_A",
         "accessConfigs":[
           {
             "type":"ONE_TO_ONE_NAT"
           }
         ]
       }
     ],
     "disks":[
       {
         "index":0,
         "boot":true,
         "initializeParams":{
           "sourceImage":"projects/debian-cloud/global/images/family/debian-10"
         },
         "autoDelete":true
       }
     ]
  }
}

Create a managed instance group in each zone with the instanceGroupManagers.insert method, replacing PROJECT_ID with your project ID.

POST https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/zones/{zone}/instanceGroupManagers

{
  "name": "gl7-ilb-mig-a",
  "zone": "projects/PROJECT_ID/zones/ZONE_A",
  "instanceTemplate": "projects/PROJECT_ID/global/instanceTemplates/gil7-backendwest1-template",
  "baseInstanceName": "gl7-ilb-mig-b",
  "targetSize": 2
}

POST https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/global/instanceTemplates

{
  "name":"gil7-backendeast1-template",
  "properties":{
     "machineType":"e2-standard-2",
     "tags":{
       "items":[
         "allow-ssh",
         "load-balanced-backend"
       ]
     },
     "metadata":{
        "kind":"compute#metadata",
        "items":[
          {
            "key":"startup-script",
            "value":"#! /bin/bash\napt-get update\napt-get install
            apache2 -y\na2ensite default-ssl\na2enmod ssl\n
            vm_hostname=\"$(curl -H \"Metadata-Flavor:Google\"           
            \\\nhttp://169.254.169.254/computeMetadata/v1/instance/name)\"\n
            echo \"Page served from: $vm_hostname\" | \\\ntee
            /var/www/html/index.html\nsystemctl restart apache2"
          }
        ]
     },
     "networkInterfaces":[
       {
         "network":"projects/PROJECT_ID/global/networks/NETWORK",
         "subnetwork":"regions/REGION_B/subnetworks/SUBNET_B",
         "accessConfigs":[
           {
             "type":"ONE_TO_ONE_NAT"
           }
         ]
       }
     ],
     "disks":[
       {
         "index":0,
         "boot":true,
         "initializeParams":{
           "sourceImage":"projects/debian-cloud/global/images/family/debian-10"
         },
         "autoDelete":true
       }
     ]
  }
}

Create a managed instance group in each zone with the instanceGroupManagers.insert method, replacing PROJECT_ID with your project ID.

POST https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/zones/{zone}/instanceGroupManagers

{
  "name": "gl7-ilb-mig-b",
  "zone": "projects/PROJECT_ID/zones/ZONE_B",
  "instanceTemplate": "projects/PROJECT_ID/global/instanceTemplates/gil7-backendwest1-template",
  "baseInstanceName": "gl7-ilb-mig-b",
  "targetSize": 2
}

Configure the load balancer

This example shows you how to create the following cross-region internal Application Load Balancer resources:

  • A global HTTP health check.
  • A global backend service with the managed instance groups as the backend.
  • An URL map. Make sure to refer to a global URL map for the target HTTP(S) proxy. A global URL map routes requests to a global backend service based on rules that you define for the host and path of an incoming URL. A global URL map can be referenced by a global target proxy rule.
  • A global SSL certificate (for HTTPS).
  • A global target proxy.

  • Two global forwarding rules with regional IP addresses. For the forwarding rule's IP address, use the SUBNET_A or SUBNET_B IP address range. If you try to use the proxy-only subnet, forwarding rule creation fails.

Console

Start your configuration

  1. In the Google Cloud console, go to the Load balancing page.

    Go to Load balancing

  2. Click Create load balancer.
  3. For Type of load balancer, select Application Load Balancer (HTTP/HTTPS) and click Next.
  4. For Public facing or internal, select Internal and click Next.
  5. For Cross-region or single region deployment, select Best for cross-region workloads and click Next.
  6. Click Configure.

Basic configuration

  1. Provide a Name for the load balancer.
  2. For Network, select NETWORK.

Configure the frontend with two forwarding rules

For HTTP:

  1. Click Frontend configuration.
    1. Provide a Name for the forwarding rule.
    2. In the Subnetwork region list, select REGION_A.

      Reserve a proxy-only subnet

    3. In the Subnetwork list, select SUBNET_A.
    4. In the IP address list, click Create IP address. The Reserve a static internal IP address page opens.
      • Provide a Name for the static IP address.
      • In the Static IP address list, select Let me choose.
      • In the Custom IP address field, enter 10.1.2.99.
      • Select Reserve.
  2. Click Done.
  3. To add the second forwarding rule, click Add frontend IP and port.
    1. Provide a Name for the forwarding rule.
    2. In the Subnetwork region list, select REGION_B.

      Reserve a proxy-only subnet

    3. In the Subnetwork list, select SUBNET_B.
    4. In the IP address list, click Create IP address. The Reserve a static internal IP address page opens.
      • Provide a Name for the static IP address.
      • In the Static IP address list, select Let me choose.
      • In the Custom IP address field, enter 10.1.3.99.
      • Select Reserve.
  4. Click Done.

For HTTPS:

If you are using HTTPS between the client and the load balancer, you need one or more SSL certificate resources to configure the proxy. To create an all-regions Google-managed certificate, see the following documentation:

After you create the Google-managed certificate, attach the certificate directly to the target proxy. Certificate maps are not supported by cross-region internal Application Load Balancers.

To create an all-regions self-managed certificate, see the following documentation: Deploy a regional self-managed certificate.

  1. Click Frontend configuration.
    1. Provide a Name for the forwarding rule.
    2. In the Protocol field, select HTTPS (includes HTTP/2).
    3. Ensure that the Port is set to 443.
    4. In the Subnetwork region list, select REGION_A.

      Reserve a proxy-only subnet

    5. In the Subnetwork list, select SUBNET_A.
    6. In the IP address list, click Create IP address. The Reserve a static internal IP address page opens.
      • Provide a Name for the static IP address.
      • In the Static IP address list, select Let me choose.
      • In the Custom IP address field, enter 10.1.3.99.
      • Select Reserve.
    7. In the Add certificate section, select the certificate.
    8. Optional: To add certificates in addition to the primary SSL certificate:
      1. Click Add certificate.
      2. Select the certificate from the list.
    9. Select an SSL policy from the SSL policy list. If you have not created any SSL policies, a default Google Cloud SSL policy is applied.
    10. Click Done.

    Add the second frontend configuration:

    1. Provide a Name for the frontend configuration.
    2. In the Protocol field, select HTTPS (includes HTTP/2).
    3. Ensure that the Port is set to 443.
    4. In the Subnetwork region list, select REGION_B.

      Reserve a proxy-only subnet

    5. In the Subnetwork list, select SUBNET_B.
    6. In the IP address list, click Create IP address. The Reserve a static internal IP address page opens.
      • Provide a Name for the static IP address.
      • In the Static IP address list, select Let me choose.
      • In the Custom IP address field, enter 10.1.3.99.
      • Select Reserve.
    7. In the Add certificate section, select the certificate.
    8. Optional: To add certificates in addition to the primary SSL certificate:
      1. Click Add certificate.
      2. Select the certificate from the list.
    9. Select an SSL policy from the SSL policy list. If you have not created any SSL policies, a default Google Cloud SSL policy is applied.
    10. Click Done.
    Configure the backend service
    1. Click Backend configuration.
    2. In the Create or select backend services list, click Create a backend service.
    3. Provide a Name for the backend service.
    4. For Protocol, select HTTP.
    5. For Named Port, enter http.
    6. In the Backend type list, select Instance group.
    7. In the New backend section:
      • In the Instance group list, select gl7-ilb-mig-a in REGION_A.
      • Set Port numbers to 80.
      • Click Done.
      • To add another backend, click Add backend.
      • In the Instance group list, select gl7-ilb-mig-b in REGION_B.
      • Set Port numbers to 80.
      • Click Done.
    8. In the Health check list, click Create a health check.
      • In the Name field, enter global-http-health-check.
      • Set Protocol to HTTP.
      • Set Port to 80.
      • Click Save.

    Configure the routing rules

    1. Click Routing rules.
    2. For Mode, select Simple host and path rule.
    3. Ensure that there is only one backend service for any unmatched host and any unmatched path.

    Review the configuration

    1. Click Review and finalize.
    2. Review your load balancer configuration settings.
    3. Click Create.

gcloud

  1. Define the HTTP health check with the gcloud compute health-checks create http command.

    gcloud compute health-checks create http global-http-health-check \
   --use-serving-port \
   --global

  2. Define the backend service with the gcloud compute backend-services create command.

    gcloud compute backend-services create BACKEND_SERVICE_NAME \
  --load-balancing-scheme=INTERNAL_MANAGED \
  --protocol=HTTP \
  --enable-logging \
  --logging-sample-rate=1.0 \
  --health-checks=global-http-health-check \
  --global-health-checks \
  --global

  3. Add backends to the backend service with the gcloud compute backend-services add-backend command.

    gcloud compute backend-services add-backend BACKEND_SERVICE_NAME \
  --balancing-mode=UTILIZATION \
  --instance-group=gl7-ilb-mig-a \
  --instance-group-zone=ZONE_A \
  --global

    gcloud compute backend-services add-backend BACKEND_SERVICE_NAME \
  --balancing-mode=UTILIZATION \
  --instance-group=gl7-ilb-mig-b \
  --instance-group-zone=ZONE_B \
  --global

  4. Create the URL map with the gcloud compute url-maps create command.

    gcloud compute url-maps create gl7-gilb-url-map \
  --default-service=BACKEND_SERVICE_NAME \
  --global

  5. Create the target proxy.

    For HTTP:

    Create the target proxy with the gcloud compute target-http-proxies create command.

    gcloud compute target-http-proxies create gil7-http-proxy \
  --url-map=gl7-gilb-url-map \
  --global

    For HTTPS:

    To create a Google-managed certificate, see the following documentation:

    After you create the Google-managed certificate, attach the certificate directly to the target proxy. Certificate maps are not supported by cross-region internal Application Load Balancers.

    To create a self-managed certificate, see the following documentation:

    Assign your file paths to variable names.

    export LB_CERT=PATH_TO_PEM_FORMATTED_FILE

    export LB_PRIVATE_KEY=PATH_TO_LB_PRIVATE_KEY_FILE

    Create an all region SSL certificate using the gcloud beta certificate-manager certificates create command.

    gcloud certificate-manager certificates create gilb-certificate \
  --private-key-file=$LB_PRIVATE_KEY \
  --certificate-file=$LB_CERT \
  --scope=all-regions

    Use the SSL certificate to create a target proxy with the gcloud compute target-https-proxies create command

    gcloud compute target-https-proxies create gil7-https-proxy \
  --url-map=gl7-gilb-url-map \
  --certificate-manager-certificates=gilb-certificate \
  --global

  6. Create two forwarding rules, one with a VIP (10.1.2.99) in the REGION_B region and another one with a VIP (10.1.3.99) in the REGION_A region. For more information, see Reserve a static internal IPv4 address.

    For custom networks, you must reference the subnet in the forwarding rule. Note that this is the VM subnet, not the proxy subnet.

    For HTTP:

    Use the gcloud compute forwarding-rules create command with the correct flags.

    gcloud compute forwarding-rules create FWRULE_A \
  --load-balancing-scheme=INTERNAL_MANAGED \
  --network=NETWORK \
  --subnet=SUBNET_A \
  --subnet-region=REGION_A \
  --address=10.1.2.99 \
  --ports=80 \
  --target-http-proxy=gil7-http-proxy \
  --global

    gcloud compute forwarding-rules create FWRULE_B \
  --load-balancing-scheme=INTERNAL_MANAGED \
  --network=NETWORK \
  --subnet=SUBNET_B \
  --subnet-region=REGION_B \
  --address=10.1.3.99 \
  --ports=80 \
  --target-http-proxy=gil7-http-proxy \
  --global

    For HTTPS:

    Use the gcloud compute forwarding-rules create command with the correct flags.

    gcloud compute forwarding-rules create FWRULE_A \
  --load-balancing-scheme=INTERNAL_MANAGED \
  --network=NETWORK \
  --subnet=SUBNET_A \
  --subnet-region=REGION_A \
  --address=10.1.2.99 \
  --ports=443 \
  --target-https-proxy=gil7-https-proxy \
  --global

    gcloud compute forwarding-rules create FWRULE_B \
  --load-balancing-scheme=INTERNAL_MANAGED \
  --network=NETWORK \
  --subnet=SUBNET_B \
  --subnet-region=REGION_B \
  --address=10.1.3.99 \
  --ports=443 \
  --target-https-proxy=gil7-https-proxy \
  --global

Terraform

To create the health check, use the google_compute_health_check resource.

resource "google_compute_health_check" "default" {
  provider = google-beta
  name     = "global-http-health-check"
  http_health_check {
    port_specification = "USE_SERVING_PORT"
  }
}

To create the backend service, use the google_compute_backend_service resource.

resource "google_compute_backend_service" "default" {
  name                  = "gl7-gilb-backend-service"
  provider              = google-beta
  protocol              = "HTTP"
  load_balancing_scheme = "INTERNAL_MANAGED"
  timeout_sec           = 10
  health_checks         = [google_compute_health_check.default.id]
  backend {
    group           = google_compute_region_instance_group_manager.mig_a.instance_group
    balancing_mode  = "UTILIZATION"
    capacity_scaler = 1.0
  }
  backend {
    group           = google_compute_region_instance_group_manager.mig_b.instance_group
    balancing_mode  = "UTILIZATION"
    capacity_scaler = 1.0
  }
}

To create the URL map, use the google_compute_url_map resource.

resource "google_compute_url_map" "default" {
  name            = "gl7-gilb-url-map"
  provider        = google-beta
  default_service = google_compute_backend_service.default.id
}

To create the target HTTP proxy, use the google_compute_target_http_proxy resource.

resource "google_compute_target_http_proxy" "default" {
  name     = "gil7target-http-proxy"
  provider = google-beta
  url_map  = google_compute_url_map.default.id
}

To create the forwarding rules, use the google_compute_forwarding_rule resource.

resource "google_compute_global_forwarding_rule" "fwd_rule_a" {
  provider              = google-beta
  depends_on            = [google_compute_subnetwork.proxy_subnet_a]
  ip_address            = "10.1.2.99"
  ip_protocol           = "TCP"
  load_balancing_scheme = "INTERNAL_MANAGED"
  name                  = "gil7forwarding-rule-a"
  network               = google_compute_network.default.id
  port_range            = "80"
  target                = google_compute_target_http_proxy.default.id
  subnetwork            = google_compute_subnetwork.subnet_a.id
}
resource "google_compute_global_forwarding_rule" "fwd_rule_b" {
  provider              = google-beta
  depends_on            = [google_compute_subnetwork.proxy_subnet_b]
  ip_address            = "10.1.3.99"
  ip_protocol           = "TCP"
  load_balancing_scheme = "INTERNAL_MANAGED"
  name                  = "gil7forwarding-rule-b"
  network               = google_compute_network.default.id
  port_range            = "80"
  target                = google_compute_target_http_proxy.default.id
  subnetwork            = google_compute_subnetwork.subnet_b.id
}

To learn how to apply or remove a Terraform configuration, see Basic Terraform commands.

API

Create the health check by making a POST request to the healthChecks.insert method, replacing PROJECT_ID with your project ID.

POST https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/global/healthChecks

{
"name": "global-http-health-check",
"type": "HTTP",
"httpHealthCheck": {
  "portSpecification": "USE_SERVING_PORT"
}
}

Create the global backend service by making a POST request to the backendServices.insert method, replacing PROJECT_ID with your project ID.

POST https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/global/backendServices

{
"name": "BACKEND_SERVICE_NAME",
"backends": [
  {
    "group": "projects/PROJECT_ID/zones/ZONE_A/instanceGroups/gl7-ilb-mig-a",
    "balancingMode": "UTILIZATION"
  },
  {
    "group": "projects/PROJECT_ID/zones/ZONE_B/instanceGroups/gl7-ilb-mig-b",
    "balancingMode": "UTILIZATION"
  }
],
"healthChecks": [
  "projects/PROJECT_ID/regions/global/healthChecks/global-http-health-check"
],
"loadBalancingScheme": "INTERNAL_MANAGED"
}

Create the URL map by making a POST request to the urlMaps.insert method, replacing PROJECT_ID with your project ID.

POST https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/global/urlMaps

{
"name": "l7-ilb-map",
"defaultService": "projects/PROJECT_ID/global/backendServices/BACKEND_SERVICE_NAME"
}

For HTTP:

Create the target HTTP proxy by making a POST request to the targetHttpProxies.insert method, replacing PROJECT_ID with your project ID.

POST https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/global/targetHttpProxy

{
"name": "l7-ilb-proxy",
"urlMap": "projects/PROJECT_ID/global/urlMaps/l7-ilb-map"
}

Create the forwarding rule by making a POST request to the forwardingRules.insert method, replacing PROJECT_ID with your project ID.

POST https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/global/forwardingRules

{
"name": "FWRULE_A",
"IPAddress": "10.1.2.99",
"IPProtocol": "TCP",
"portRange": "80-80",
"target": "projects/PROJECT_ID/global/targetHttpProxies/l7-ilb-proxy",
"loadBalancingScheme": "INTERNAL_MANAGED",
"subnetwork": "projects/PROJECT_ID/regions/REGION_A/subnetworks/SUBNET_A",
"network": "projects/PROJECT_ID/global/networks/NETWORK",
"networkTier": "PREMIUM"
}

POST https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/global/forwardingRules

{
"name": "gil7forwarding-rule-b",
"IPAddress": "10.1.3.99",
"IPProtocol": "TCP",
"portRange": "80-80",
"target": "projects/PROJECT_ID/global/targetHttpProxies/l7-ilb-proxy",
"loadBalancingScheme": "INTERNAL_MANAGED",
"subnetwork": "projects/PROJECT_ID/regions/REGION_B/subnetworks/SUBNET_B",
"network": "projects/PROJECT_ID/global/networks/NETWORK",
"networkTier": "PREMIUM"
}

For HTTPS:

Read the certificate and private key files, and then create the SSL certificate. The following example shows how to do this with Python.

from pathlib import Path
from pprint import pprint
from typing import Union

from googleapiclient import discovery


def create_regional_certificate(
    project_id: str,
    region: str,
    certificate_file: Union[str, Path],
    private_key_file: Union[str, Path],
    certificate_name: str,
    description: str = "Certificate created from a code sample.",
) -> dict:
    """
    Create a regional SSL self-signed certificate within your Google Cloud project.

    Args:
        project_id: project ID or project number of the Cloud project you want to use.
        region: name of the region you want to use.
        certificate_file: path to the file with the certificate you want to create in your project.
        private_key_file: path to the private key you used to sign the certificate with.
        certificate_name: name for the certificate once it's created in your project.
        description: description of the certificate.

        Returns:
        Dictionary with information about the new regional SSL self-signed certificate.
    """
    service = discovery.build("compute", "v1")

    # Read the cert into memory
    with open(certificate_file) as f:
        _temp_cert = f.read()

    # Read the private_key into memory
    with open(private_key_file) as f:
        _temp_key = f.read()

    # Now that the certificate and private key are in memory, you can create the
    # certificate resource
    ssl_certificate_body = {
        "name": certificate_name,
        "description": description,
        "certificate": _temp_cert,
        "privateKey": _temp_key,
    }
    request = service.regionSslCertificates().insert(
        project=project_id, region=region, body=ssl_certificate_body
    )
    response = request.execute()
    pprint(response)

    return response

Create the target HTTPS proxy by making a POST request to the targetHttpsProxies.insert method, replacing PROJECT_ID with your project ID.

POST https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/global/targetHttpsProxy

{
"name": "l7-ilb-proxy",
"urlMap": "projects/PROJECT_ID/global/urlMaps/l7-ilb-map",
"sslCertificates": /projects/PROJECT_ID/global/sslCertificates/SSL_CERT_NAME
}

Create the forwarding rule by making a POST request to the globalForwardingRules.insert method, replacing PROJECT_ID with your project ID.

POST https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/global/forwardingRules

{
"name": "FWRULE_A",
"IPAddress": "10.1.2.99",
"IPProtocol": "TCP",
"portRange": "80-80",
"target": "projects/PROJECT_ID/global/targetHttpsProxies/l7-ilb-proxy",
"loadBalancingScheme": "INTERNAL_MANAGED",
"subnetwork": "projects/PROJECT_ID/regions/REGION_A/subnetworks/SUBNET_A",
"network": "projects/PROJECT_ID/global/networks/NETWORK",
"networkTier": "PREMIUM"
}

POST https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/global/forwardingRules

{
"name": "FWRULE_B",
"IPAddress": "10.1.3.99",
"IPProtocol": "TCP",
"portRange": "80-80",
"target": "projects/PROJECT_ID/global/targetHttpsProxies/l7-ilb-proxy",
"loadBalancingScheme": "INTERNAL_MANAGED",
"subnetwork": "projects/PROJECT_ID/regions/REGION_B/subnetworks/SUBNET_B",
"network": "projects/PROJECT_ID/global/networks/NETWORK",
"networkTier": "PREMIUM"
}

Test the load balancer

Create a VM instance to test connectivity

  1. Create a client VM:

    gcloud compute instances create l7-ilb-client-a \
    --image-family=debian-10 \
    --image-project=debian-cloud \
    --network=NETWORK \
    --subnet=SUBNET_A \
    --zone=ZONE_A \
    --tags=allow-ssh

    gcloud compute instances create l7-ilb-client-b \
    --image-family=debian-10 \
    --image-project=debian-cloud \
    --network=NETWORK \
    --subnet=SUBNET_B \
    --zone=ZONE_B \
    --tags=allow-ssh

  2. Use SSH to connect to each client instance.

    gcloud compute ssh l7-ilb-client-a --zone=ZONE_A

    gcloud compute ssh l7-ilb-client-b --zone=ZONE_B

  3. Verify that the IP address is serving its hostname

    • Verify that the client VM can reach both IP addresses. The command returns the name of the backend VM which served the request:

      curl 10.1.2.99

      curl 10.1.3.99

      For HTTPS testing, replace curl with the following command line:

      curl -k 'https://test.example.com:443' --connect-to test.example.com:443:10.1.2.99:443

      curl -k 'https://test.example.com:443' --connect-to test.example.com:443:10.1.3.99:443

      The -k flag causes curl to skip certificate validation.

    • Optional: Use the configured DNS record to resolve the IP address closest to the client VM. For example, DNS_NAME can be service.example.com.

      curl DNS_NAME

Run 100 requests and confirm that they are load balanced

For HTTP:

  {
    RESULTS=
    for i in {1..100}
    do
        RESULTS="$RESULTS:$(curl --silent 10.1.2.99)"
    done
    echo ""
    echo " Results of load-balancing to 10.1.2.99: "
    echo "***"
    echo "$RESULTS" | tr ':' '\n' | grep -Ev "^$" | sort | uniq -c
    echo
  }
  

  {
    RESULTS=
    for i in {1..100}
    do
      RESULTS="$RESULTS:$(curl --silent 10.1.3.99)"
    done
    echo ""
    echo " Results of load-balancing to 10.1.3.99: "
    echo "***"
    echo "$RESULTS" | tr ':' '\n' | grep -Ev "^$" | sort | uniq -c
    echo
  }

For HTTPS:

  {
    RESULTS=
    for i in {1..100}
    do
      RESULTS="$RESULTS:$(curl -k -s 'https://test.example.com:443' --connect-to test.example.com:443:10.1.2.99:443)"
    done
    echo ""
    echo " Results of load-balancing to 10.1.2.99: "
    echo "***"
    echo "$RESULTS" | tr ':' '\n' | grep -Ev "^$" | sort | uniq -c
    echo
  }
  

  {
    RESULTS=
    for i in {1..100}
    do
       RESULTS="$RESULTS:$(curl -k -s 'https://test.example.com:443' --connect-to test.example.com:443:10.1.3.99:443)"
    done
    echo ""
    echo " Results of load-balancing to 10.1.3.99: "
    echo "***"
    echo "$RESULTS" | tr ':' '\n' | grep -Ev "^$" | sort | uniq -c
    echo
  }

Test failover

  1. Verify failover to backends in the REGION_A region when backends in the REGION_B are unhealthy or unreachable. To simulate failover, remove all backends from REGION_B:

    gcloud compute backend-services remove-backend BACKEND_SERVICE_NAME \
   --balancing-mode=UTILIZATION \
   --instance-group=gl7-ilb-mig-b \
   --instance-group-zone=ZONE_B

  2. Connect using SSH to a client VM in REGION_B.

    gcloud compute ssh l7-ilb-client-b \
   --zone=ZONE_B

  3. Send requests to the load balanced IP address in the REGION_B region. The command output shows responses from backend VMs in REGION_A:

    {
RESULTS=
for i in {1..100}
do
  RESULTS="$RESULTS:$(curl -k -s 'https://test.example.com:443' --connect-to test.example.com:443:10.1.3.99:443)"
done
echo "***"
echo "*** Results of load-balancing to 10.1.3.99: "
echo "***"
echo "$RESULTS" | tr ':' '\n' | grep -Ev "^$" | sort | uniq -c
echo
}

Additional configuration options

This section expands on the configuration example to provide alternative and additional configuration options. All of the tasks are optional. You can perform them in any order.

Enable session affinity

These procedures show you how to update a backend service for the example regional internal Application Load Balancer or cross-region internal Application Load Balancer so that the backend service uses generated cookie affinity, header field affinity, or HTTP cookie affinity.

When generated cookie affinity is enabled, the load balancer issues a cookie on the first request. For each subsequent request with the same cookie, the load balancer directs the request to the same backend virtual machine (VM) instance or endpoint. In this example, the cookie is named GCILB.

When header field affinity is enabled, the load balancer routes requests to backend VMs or endpoints in a network endpoint group (NEG) based on the value of the HTTP header named in the --custom-request-header flag. Header field affinity is only valid if the load balancing locality policy is either RING_HASH or MAGLEV and the backend service's consistent hash specifies the name of the HTTP header.

When HTTP cookie affinity is enabled, the load balancer routes requests to backend VMs or endpoints in a NEG, based on an HTTP cookie named in the HTTP_COOKIE flag with the optional --affinity-cookie-ttl flag. If the client does not provide the cookie in its HTTP request, the proxy generates the cookie and returns it to the client in a Set-Cookie header. HTTP cookie affinity is only valid if the load balancing locality policy is either RING_HASH or MAGLEV and the backend service's consistent hash specifies the HTTP cookie.

Console

To enable or change session affinity for a backend service:

  1. In the Google Cloud console, go to the Load balancing page.

    Go to Load balancing

  2. Click Backends.
  3. Click gil7-backend-service (the name of the backend service you created for this example), and then click Edit.
  4. On the Backend service details page, click Advanced configuration.
  5. Under Session affinity, select the type of session affinity you want.
  6. Click Update.

gcloud

Use the following Google Cloud CLI commands to update the backend service to different types of session affinity:

    gcloud compute backend-services update gil7-backend-service \
        --session-affinity=[GENERATED_COOKIE | HEADER_FIELD | HTTP_COOKIE | CLIENT_IP] \
        --global

API

To set session affinity, make a `PATCH` request to the backendServices/patch method.

    PATCH https://compute.googleapis.com/compute/v1/projects/[PROJECT_ID]/global/backendServices/gil7-backend-service
    {
    "sessionAffinity": ["GENERATED_COOKIE" | "HEADER_FIELD" | "HTTP_COOKIE" | "CLIENT_IP" ]
    }

Restrict which clients can send traffic to the load balancer

You can restrict clients from connecting to an internal Application Load Balancer forwarding rule VIP by configuring egress firewall rules on these clients. Set these firewall rules on specific client VMs based on service accounts or tags.

You can't use firewall rules to restrict inbound traffic to specific internal Application Load Balancer forwarding rule VIPs. Any client on the same VPC network and in the same region as the forwarding rule VIP can generally send traffic to the forwarding rule VIP.

Additionally, all requests to backends come from proxies that use IP addresses in the proxy-only subnet range. It is not possible to create firewall rules that allow or deny ingress traffic on these backends based on the forwarding rule VIP used by a client.

Here are some examples of how to use egress firewall rules to restrict traffic to the load balancer's forwarding rule VIP.

Console

To identify the client VMs, tag the specific VMs you want to restrict. These tags are used to associate firewall rules with the tagged client VMs. Then, add the tag to the TARGET_TAG field in the following steps.

Use either a single firewall rule or multiple rules to set this up.

Single egress firewall rule

You can configure one firewall egress rule to deny all egress traffic going from tagged client VMs to a load balancer's VIP.

  1. In the Google Cloud console, go to the Firewall rules page.

    Go to Firewall rules

  2. Click Create firewall rule to create the rule to deny egress traffic from tagged client VMs to a load balancer's VIP.

    • Name: fr-deny-access
    • Network: lb-network
    • Priority: 100
    • Direction of traffic: Egress
    • Action on match: Deny
    • Targets: Specified target tags
    • Target tags: TARGET_TAG
    • Destination filter: IP ranges
    • Destination IP ranges: 10.1.2.99
    • Protocols and ports:
      • Choose Specified protocols and ports.
      • Select the tcp checkbox, and then enter 80 for the port number.

  3. Click Create.

Multiple egress firewall rules

A more scalable approach involves setting two rules. A default, low-priority rule that restricts all clients from accessing the load balancer's VIP. A second, higher-priority rule that allows a subset of tagged clients to access the load balancer's VIP. Only tagged VMs can access the VIP.

  1. In the Google Cloud console, go to the Firewall rules page.

    Go to Firewall rules

  2. Click Create firewall rule to create the lower priority rule to deny access by default:

    • Name: fr-deny-all-access-low-priority
    • Network: lb-network
    • Priority: 200
    • Direction of traffic: Egress
    • Action on match: Deny
    • Targets: Specified target tags
    • Target tags: TARGET_TAG
    • Destination filter: IP ranges
    • Destination IP ranges: 10.1.2.99
    • Protocols and ports:
      • Choose Specified protocols and ports.
      • Select the TCP checkbox, and then enter 80 for the port number.

  3. Click Create.

  4. Click Create firewall rule to create the higher priority rule to allow traffic from certain tagged instances.

    • Name: fr-allow-some-access-high-priority
    • Network: lb-network
    • Priority: 100
    • Direction of traffic: Egress
    • Action on match: Allow
    • Targets: Specified target tags
    • Target tags: TARGET_TAG
    • Destination filter: IP ranges
    • Destination IP ranges: 10.1.2.99
    • Protocols and ports:
      • Choose Specified protocols and ports.
      • Select the TCP checkbox, and then enter 80 for the port number.

  5. Click Create.

gcloud

To identify the client VMs, tag the specific VMs you want to restrict. Then add the tag to the TARGET_TAG field in these steps.

Use either a single firewall rule or multiple rules to set this up.

Single egress firewall rule

You can configure one firewall egress rule to deny all egress traffic going from tagged client VMs to a load balancer's VIP.

gcloud compute firewall-rules create fr-deny-access \
  --network=lb-network \
  --action=deny \
  --direction=egress \
  --rules=tcp \
  --priority=100 \
  --destination-ranges=10.1.2.99 \
  --target-tags=TARGET_TAG

Multiple egress firewall rules

A more scalable approach involves setting two rules: a default, low-priority rule that restricts all clients from accessing the load balancer's VIP, and a second, higher-priority rule that allows a subset of tagged clients to access the load balancer's VIP. Only tagged VMs can access the VIP.

  1. Create the lower-priority rule:

    gcloud compute firewall-rules create fr-deny-all-access-low-priority \
  --network=lb-network \
  --action=deny \
  --direction=egress \
  --rules=tcp \
  --priority=200 \
  --destination-ranges=10.1.2.99

  2. Create the higher priority rule:

    gcloud compute firewall-rules create fr-allow-some-access-high-priority \
  --network=lb-network \
  --action=allow \
  --direction=egress \
  --rules=tcp \
  --priority=100 \
  --destination-ranges=10.1.2.99 \
  --target-tags=TARGET_TAG

To use service accounts instead of tags to control access, use the --target-service-accounts option instead of the --target-tags flag when creating firewall rules.

Scale restricted access to internal Application Load Balancer backends based on subnets

Maintaining separate firewall rules or adding new load-balanced IP addresses to existing rules as described in the previous section becomes inconvenient as the number of forwarding rules increases. One way to prevent this is to allocate forwarding rule IP addresses from a reserved subnet. Then, traffic from tagged instances or service accounts can be allowed or blocked by using the reserved subnet as the destination range for firewall rules. This lets you effectively control access to a group of forwarding rule VIPs without having to maintain per-VIP firewall egress rules.

Here are the high-level steps to set this up, assuming that you will create all the other required load balancer resources separately.

gcloud

  1. Create a regional subnet that will be used to allocate load-balanced IP addresses for forwarding rules:

    gcloud compute networks subnets create l7-ilb-restricted-subnet \
  --network=lb-network \
  --region=us-west1 \
  --range=10.127.0.0/24

  2. Create a forwarding rule that takes an address from the subnet. The following example uses the address 10.127.0.1 from the subnet created in the previous step. 

    gcloud compute forwarding-rules create l7-ilb-forwarding-rule-restricted \
  --load-balancing-scheme=INTERNAL_MANAGED \
  --network=lb-network \
  --subnet=l7-ilb-restricted-subnet \
  --address=10.127.0.1 \
  --ports=80 \
  --global \
  --target-http-proxy=gil7-http-proxy

  3. Create a firewall rule to restrict traffic destined for the range IP addresses in the forwarding rule subnet (l7-ilb-restricted-subnet):

    gcloud compute firewall-rules create restrict-traffic-to-subnet \
  --network=lb-network \
  --action=deny \
  --direction=egress \
  --rules=tcp:80 \
  --priority=100 \
  --destination-ranges=10.127.0.0/24 \
  --target-tags=TARGET_TAG

Use the same IP address between multiple internal forwarding rules

For multiple internal forwarding rules to share the same internal IP address, you must reserve the IP address and set its --purpose flag to SHARED_LOADBALANCER_VIP.

gcloud 

gcloud compute addresses create SHARED_IP_ADDRESS_NAME \
    --region=REGION \
    --subnet=SUBNET_NAME \
    --purpose=SHARED_LOADBALANCER_VIP
If you need to redirect HTTP traffic to HTTPS, you can create two forwarding rules that use a common IP address. For more information, see Set up HTTP-to-HTTPS redirect for internal Application Load Balancers.

Configure DNS routing policies

If your clients are in multiple regions, you might want to make your cross-region internal Application Load Balancer accessible by using VIPs in these regions. This multi-region setup minimizes latency and network transit costs. In addition, it lets you set up a DNS-based, global, load balancing solution that provides resilience against regional outages. For more information, see Manage DNS routing policies and health checks.

gcloud

To create a DNS entry with a 30 second TTL, use the gcloud dns record-sets create command.

gcloud dns record-sets create DNS_ENTRY --ttl="30" \
  --type="A" --zone="service-zone" \
  --routing-policy-type="GEO" \
  --routing-policy-data="REGION_A=gil7-forwarding-rule-a@global;REGION_B=gil7-forwarding-rule-b@global" \
  --enable-health-checking

Replace the following:

  • DNS_ENTRY: DNS or domain name of the record-set

    For example, service.example.com

  • REGION_A and REGION_B: the regions where you have configured the load balancer

API

Create the DNS record by making a POST request to the ResourceRecordSets.create method. Replace PROJECT_ID with your project ID.

POST https://www.googleapis.com/dns/v1/projects/PROJECT_ID/managedZones/SERVICE_ZONE/rrsets
{
  "name": "DNS_ENTRY",
  "type": "A",
  "ttl": 30,
  "routingPolicy": {
    "geo": {
      "items": [
        {
          "location": "REGION_A",
          "healthCheckedTargets": {
            "internalLoadBalancers": [
              {
                "loadBalancerType": "globalL7ilb",
                "ipAddress": "gil7-forwarding-rule-a",
                "port": "80",
                "ipProtocol": "tcp",
                "networkUrl": "https://www.googleapis.com/compute/v1/projects/PROJECT_ID/global/networks/lb-network",
                "project": "PROJECT_ID"
              }
            ]
          }
        },
        {
          "location": "REGION_B",
          "healthCheckedTargets": {
            "internalLoadBalancers": [
              {
                "loadBalancerType": "globalL7ilb",
                "ipAddress": "gil7-forwarding-rule-b",
                "port": "80",
                "ipProtocol": "tcp",
                "networkUrl": "https://www.googleapis.com/compute/v1/projects/PROJECT_ID/global/networks/lb-network",
                "project": "PROJECT_ID"
              }
            ]
          }
        }
      ]
    }
  }
}

Enable outlier detection

You can enable outlier detection on global backend services to identify unhealthy serverless NEGs and reduce the number the requests sent to the unhealthy serverless NEGs.

Outlier detection is enabled on the backend service by using one of the following methods:

  • The consecutiveErrors method (outlierDetection.consecutiveErrors), in which a 5xx series HTTP status code qualifies as an error.
  • The consecutiveGatewayFailure method (outlierDetection.consecutiveGatewayFailure), in which only the 502, 503, and 504 HTTP status codes qualify as an error.

Use the following steps to enable outlier detection for an existing backend service. Note that even after enabling outlier detection, some requests can be sent to the unhealthy service and return a 5xx status code to the clients. To further reduce the error rate, you can configure more aggressive values for the outlier detection parameters. For more information, see the outlierDetection field.

Console

  1. In the Google Cloud console, go to the Load balancing page.

    Go to Load balancing

  2. Click the name of the load balancer whose backend service you want to edit.

  3. On the Load balancer details page, click Edit.

  4. On the Edit cross-region internal Application Load Balancer page, click Backend configuration.

  5. On the Backend configuration page, click Edit for the backend service that you want to modify.

  6. Scroll down and expand the Advanced configurations section.

  7. In the Outlier detection section, select the Enable checkbox.

  8. Click Edit to configure outlier detection.

    Verify that the following options are configured with these values:

    Property Value
    Consecutive errors 5
    Interval 1000
    Base ejection time 30000
    Max ejection percent 50
    Enforcing consecutive errors 100

    In this example, the outlier detection analysis runs every one second. If the number of consecutive HTTP 5xx status codes received by an Envoy proxy is five or more, the backend endpoint is ejected from the load-balancing pool of that Envoy proxy for 30 seconds. When the enforcing percentage is set to 100%, the backend service enforces the ejection of unhealthy endpoints from the load-balancing pools of those specific Envoy proxies every time the outlier detection analysis runs. If the ejection conditions are met, up to 50% of the backend endpoints from the load-balancing pool can be ejected.

  9. Click Save.

  10. To update the backend service, click Update.

  11. To update the load balancer, on the Edit cross-region internal Application Load Balancer page, click Update.

gcloud

  1. Export the backend service into a YAML file.

    gcloud compute backend-services export BACKEND_SERVICE_NAME \
  --destination=BACKEND_SERVICE_NAME.yaml --global

    Replace BACKEND_SERVICE_NAME with the name of the backend service.

  2. Edit the YAML configuration of the backend service to add the fields for outlier detection as highlighted in the following YAML configuration.

    In this example, the outlier detection analysis runs every one second. If the number of consecutive HTTP 5xx status codes received by an Envoy proxy is five or more, the backend endpoint is ejected from the load-balancing pool of that Envoy proxy for 30 seconds. When the enforcing percentage is set to 100%, the backend service enforces the ejection of unhealthy endpoints from the load-balancing pools of those specific Envoy proxies every time the outlier detection analysis runs. If the ejection conditions are met, up to 50% of the backend endpoints from the load-balancing pool can be ejected. 

    name: BACKEND_SERVICE_NAME
backends:
- balancingMode: UTILIZATION
  capacityScaler: 1.0
  group: https://www.googleapis.com/compute/v1/projects/PROJECT_ID/regions/REGION_A/networkEndpointGroups/SERVERLESS_NEG_NAME
- balancingMode: UTILIZATION
  capacityScaler: 1.0
  group: https://www.googleapis.com/compute/v1/projects/PROJECT_ID/regions/REGION_B/networkEndpointGroups/SERVERLESS_NEG_NAME_2
outlierDetection:
  baseEjectionTime:
    nanos: 0
    seconds: 30
  consecutiveErrors: 5
  enforcingConsecutiveErrors: 100
  interval:
    nanos: 0
    seconds: 1
  maxEjectionPercent: 50
port: 80
selfLink: https://www.googleapis.com/compute/v1/projects/PROJECT_ID/global/backendServices/BACKEND_SERVICE_NAME
sessionAffinity: NONE
timeoutSec: 30
...

    Replace the following:

    • BACKEND_SERVICE_NAME: the name of the backend service
    • PROJECT_ID: the ID of your project
    • REGION_A and REGION_B: the regions where the load balancer has been configured.
    • SERVERLESS_NEG_NAME: the name of the first serverless NEG
    • SERVERLESS_NEG_NAME_2: the name of the second serverless NEG

  3. Update the backend service by importing the latest configuration.

    gcloud compute backend-services import BACKEND_SERVICE_NAME \
  --source=BACKEND_SERVICE_NAME.yaml --global

    Outlier detection is now enabled on the backend service.

What's next