Setting up Internal TCP/UDP Load Balancing

This guide uses an example to teach the fundamentals of Google Cloud Internal TCP/UDP Load Balancing. Before following this guide, familiarize yourself with the following:

• Internal TCP/UDP Load Balancing concepts
• Firewall rules overview
• Health check concepts

Permissions

To follow this guide, you need to create instances and modify a network in a project. You should be either a project owner or editor, or you should have all of the following Compute Engine IAM roles:

Task	Required Role
Create networks, subnets, and load balancer components	Network Admin
Add and remove firewall rules	Security Admin
Create instances	Compute Instance Admin

For more information, see the following guides:

• Access control
• IAM Conditions

Setup

This guide shows you how to configure and test an internal TCP/UDP load balancer. The steps in this section describe how to configure the following:

1. A sample VPC network with custom subnets
2. Firewall rules that allow incoming connections to backend VMs
3. Four backend VMs:
   • Two VMs in an unmanaged instance group in zone us-west1-a
   • Two VMs in an unmanaged instance group in zone us-west1-c
4. One client VM to test connections
5. The following internal TCP/UDP load balancer components:
   • A health check for the backend service
   • An internal backend service in the us-west1 region to manage connection distribution to the two zonal instance groups
   • An internal forwarding rule and internal IP address for the frontend of the load balancer

The architecture for this example looks like this:

Internal TCP/UDP Load Balancing Example Configuration (click to enlarge)

Note: For simplicity, this example uses unmanaged instance groups. In production, consider using a managed instance group, as illustrated in Managed instance groups.

Configuring a network, region, and subnet

The example internal TCP/UDP load balancer described on this page is created in a custom mode VPC network named lb-network.

This example's backend VMs and load balancer's components are located in this region and subnet:

• Region: us-west1
• Subnet: lb-subnet, with primary IP address range 10.1.2.0/24

To demonstrate global access, this example creates a second test client VM in a different region and subnet:

• Region: europe-west1
• Subnet: europe-subnet, with primary IP address range 10.3.4.0/24

Note: You can change the name of the network, the region, and the parameters for the subnet; however, subsequent steps in this guide use the network, region, and subnet parameters as outlined above.

To create the example network and subnet, follow these steps.

Console

1. In the Google Cloud Console, go to the VPC networks page.
   Go to VPC networks
2. Click Create VPC network.
3. Enter a Name of lb-network.
4. In the Subnets section:
   • Set the Subnet creation mode to Custom.
   • In the New subnet section, enter the following information:
     • Name: lb-subnet
     • Region: us-west1
     • IP address range: 10.1.2.0/24
     • Click Done.
   • Click Add subnet and enter the following information:
     • Name: europe-subnet
     • Region: europe-west1
     • IP address range: 10.3.4.0/24
     • Click Done.
5. Click Create.

gcloud

1. Create the custom VPC network:

   gcloud compute networks create lb-network --subnet-mode=custom
   

   1. Within the lb-network network, create a subnet for backends in the us-west1 region and another subnet for testing global access in the europe-west1 region:

   gcloud compute networks subnets create lb-subnet \
       --network=lb-network \
       --range=10.1.2.0/24 \
       --region=us-west1
   

   gcloud compute networks subnets create europe-subnet \
       --network=lb-network \
       --range=10.3.4.0/24 \
       --region=europe-west1
   

api

Make a POST request to the networks.insert method.

POST https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/global/networks

{
  "routingConfig": {
    "routingMode": "REGIONAL"
  },
  "name": "lb-network",
  "autoCreateSubnetworks": false
}

Make two POST requests to the subnetworks.insert method.

POST https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/regions/us-west1/subnetworks

{
  "name": "lb-subnet",
  "network": "https://www.googleapis.com/compute/v1/projects/project-id/global/networks/lb-network",
  "ipCidrRange": "10.1.2.0/24",
  "privateIpGoogleAccess": false
}

POST https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/regions/europe-west1/subnetworks

{
  "name": "europe-subnet",
  "network": "https://www.googleapis.com/compute/v1/projects/project-id/global/networks/lb-network",
   "ipCidrRange": "10.3.4.0/24",
   "privateIpGoogleAccess": false
}

Configuring firewall rules

This example uses the following firewall rules:

• fw-allow-lb-access: An ingress rule, applicable to all targets in the VPC network, allowing traffic from sources in the 10.1.2.0/24 and 10.3.4.0/24 ranges. This rule allows incoming traffic from any client located in either of the two subnets. It later allows you to configure and test global access.

• fw-allow-ssh: An ingress rule, applicable to the instances being load balanced, that allows incoming SSH connectivity on TCP port 22 from any address. You can choose a more restrictive source IP range for this rule; for example, you can specify just the IP ranges of the system from which you will initiating SSH sessions. This example uses the target tag allow-ssh to identify the VMs to which it should apply.

• fw-allow-health-check: An ingress rule, applicable to the instances being load balanced, that allows traffic from the Google Cloud health checking systems (130.211.0.0/22 and 35.191.0.0/16). This example uses the target tag allow-health-check to identify the instances to which it should apply.

Without these firewall rules, the default deny ingress rule blocks incoming traffic to the backend instances.

Note: You must create a firewall rule to allow health checks from the IP ranges of Google Cloud probe systems. See probe IP ranges for more information.

Console

1. In the Google Cloud Console, go to the Firewall page.
   Go to Firewall
2. Click Create firewall rule and enter the following information to create the rule to allow subnet traffic:
   • Name: fw-allow-lb-access
   • Network: lb-network
   • Priority: 1000
   • Direction of traffic: ingress
   • Action on match: allow
   • Targets: All instances in the network
   • Source filter: IP ranges
   • Source IP ranges: 10.1.2.0/24 and 10.3.4.0/24
   • Protocols and ports: Allow all
3. Click Create.
4. Click Create firewall rule again to create the rule to allow incoming SSH connections:
   • Name: fw-allow-ssh
   • Network: lb-network
   • Priority: 1000
   • Direction of traffic: ingress
   • Action on match: allow
   • Targets: Specified target tags
   • Target tags: allow-ssh
   • Source filter: IP ranges
   • Source IP ranges: 0.0.0.0/0
   • Protocols and ports: Choose Specified protocols and ports then type: tcp:22
5. Click Create.
6. Click Create firewall rule a third time to create the rule to allow Google Cloud health checks:
   • Name: fw-allow-health-check
   • Network: lb-network
   • Priority: 1000
   • Direction of traffic: ingress
   • Action on match: allow
   • Targets: Specified target tags
   • Target tags: allow-health-check
   • Source filter: IP ranges
   • Source IP ranges: 130.211.0.0/22 and 35.191.0.0/16
   • Protocols and ports: Allow all
7. Click Create.

gcloud

1. Create the fw-allow-lb-access firewall rule to allow communication from with the subnet:

   gcloud compute firewall-rules create fw-allow-lb-access \
       --network=lb-network \
       --action=allow \
       --direction=ingress \
       --source-ranges=10.1.2.0/24,10.3.4.0/24 \
       --rules=tcp,udp,icmp
   

2. Create the fw-allow-ssh firewall rule to allow SSH connectivity to VMs with the network tag allow-ssh. When you omit source-ranges, Google Cloud interprets the rule to mean any source.

   gcloud compute firewall-rules create fw-allow-ssh \
       --network=lb-network \
       --action=allow \
       --direction=ingress \
       --target-tags=allow-ssh \
       --rules=tcp:22
   

3. Create the fw-allow-health-check rule to allow Google Cloud health checks.

   gcloud compute firewall-rules create fw-allow-health-check \
       --network=lb-network \
       --action=allow \
       --direction=ingress \
       --target-tags=allow-health-check \
       --source-ranges=130.211.0.0/22,35.191.0.0/16 \
       --rules=tcp,udp,icmp
   

api

Create the fw-allow-lb-access firewall rule by making a POST request to the firewalls.insert method.

POST https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/global/firewalls

{
  "name": "fw-allow-lb-access",
  "network": "https://www.googleapis.com/compute/v1/projects/project-id/global/networks/lb-network",
  "priority": 1000,
  "sourceRanges": [
    "10.1.2.0/24", "10.3.4.0/24"
  ],
  "allowed": [
    {
      "IPProtocol": "tcp"
    },
    {
      "IPProtocol": "udp"
    },
    {
      "IPProtocol": "icmp"
    }
  ],
  "direction": "INGRESS",
  "logConfig": {
    "enable": false
  },
  "disabled": false
}

Create the fw-allow-ssh firewall rule by making a POST request to the firewalls.insert method.

POST https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/global/firewalls

{
  "name": "fw-allow-ssh",
       "network": "https://www.googleapis.com/compute/v1/projects/project-id/global/networks/lb-network",
  "priority": 1000,
  "sourceRanges": [
    "0.0.0.0/0"
  ],
  "targetTags": [
    "allow-ssh"
  ],
  "allowed": [
   {
     "IPProtocol": "tcp",
     "ports": [
       "22"
     ]
   }
  ],
 "direction": "INGRESS",
 "logConfig": {
   "enable": false
 },
 "disabled": false
}

Create the fw-allow-health-check firewall rule by making a POST request to the firewalls.insert method.

POST https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/global/firewalls

{
  "name": "fw-allow-health-check",
  "network": "https://www.googleapis.com/compute/v1/projects/project-id/global/networks/lb-network",
  "priority": 1000,
  "sourceRanges": [
    "130.211.0.0/22",
    "35.191.0.0/16"
  ],
  "targetTags": [
    "allow-health-check"
  ],
  "allowed": [
    {
      "IPProtocol": "tcp"
    },
    {
      "IPProtocol": "udp"
    },
    {
      "IPProtocol": "icmp"
    }
  ],
  "direction": "INGRESS",
  "logConfig": {
    "enable": false
  },
  "disabled": false
}

Creating backend VMs and instance groups

This example uses two unmanaged instance groups each having two backend (server) VMs. To demonstrate the regional nature of Internal TCP/UDP Load Balancing, the two instance groups are placed in separate zones, us-west1-a and us-west1-c.

• Instance group ig-a contains these two VMs:
  • vm-a1
  • vm-a2
• Instance group ig-c contains these two VMs:
  • vm-c1
  • vm-c2

Traffic to all four of the backend VMs is load balanced.

To support this example and the additional configuration options, each of the four VMs runs an Apache web server that listens on the following TCP ports: 80, 8008, 8080, 8088, 443, and 8443.

Each VM is assigned an internal IP address in the lb-subnet and an ephemeral external (public) IP address. You can remove the external IP addresses later.

External IP address for the backend VMs are not required; however, they are useful for this example because they permit the backend VMs to download Apache from the internet, and they make it easy to connect via SSH.

By default, Apache is configured to bind to any IP address. internal TCP/UDP load balancers deliver packets by preserving the destination IP. Ensure that server software running on your backend VMs is listening on the IP address of the load balancer's internal forwarding rule. If you configure multiple internal forwarding rules, ensure that your software listens to the internal IP address associated with each one. The destination IP address of a packet delivered to a backend VM by an internal TCP/UDP load balancer is the internal IP address of the forwarding rule.

For instructional simplicity, these backend VMs run Debian Debian GNU/Linux 10.

Console

Create backend VMs

1. In the Google Cloud Console, go to the VM instances page.
   Go to VM instances
2. Repeat the following steps to create four VMs, using the following name and zone combinations.
   • Name: vm-a1, zone: us-west1-a
   • Name: vm-a2, zone: us-west1-a
   • Name: vm-c1, zone: us-west1-c
   • Name: vm-c2, zone: us-west1-c
3. Click Create instance.
4. Set the Name as indicated in step 2.
5. For the Region, choose us-west1, and choose a Zone as indicated in step 2.
6. In the Boot disk section, ensure that the Debian operating system and the 10 (buster) version are selected for the boot disk options. Click Choose to change the image if necessary.

7. Click Management, security, disks, networking, sole tenancy and make the following changes:

   • Click Networking and add the following Network tags: allow-ssh and allow-health-check
   • Click Edit under Network interfaces and make the following changes then click Done:
     • Network: lb-network
     • Subnet: lb-subnet
     • Primary internal IP: Ephemeral (automatic)
     • External IP: Ephemeral

   • Click Management. In the Startup script field, copy and paste the following script contents. The script contents are identical for all four VMs:

     #! /bin/bash
     if [ -f /etc/startup_script_completed ]; then
     exit 0
     fi
     apt-get update
     apt-get install apache2 -y
     a2ensite default-ssl
     a2enmod ssl
     file_ports="/etc/apache2/ports.conf"
     file_http_site="/etc/apache2/sites-available/000-default.conf"
     file_https_site="/etc/apache2/sites-available/default-ssl.conf"
     http_listen_prts="Listen 80\nListen 8008\nListen 8080\nListen 8088"
     http_vh_prts="*:80 *:8008 *:8080 *:8088"
     https_listen_prts="Listen 443\nListen 8443"
     https_vh_prts="*:443 *:8443"
     vm_hostname="$(curl -H "Metadata-Flavor:Google" \
     http://169.254.169.254/computeMetadata/v1/instance/name)"
     echo "Page served from: $vm_hostname" | \
     tee /var/www/html/index.html
     echo "Page served from: $vm_hostname" | \
     tee /var/www/html/index.html
     prt_conf="$(cat "$file_ports")"
     prt_conf_2="$(echo "$prt_conf" | sed "s|Listen 80|${http_listen_prts}|")"
     prt_conf="$(echo "$prt_conf_2" | sed "s|Listen 443|${https_listen_prts}|")"
     echo "$prt_conf" | tee "$file_ports"
     http_site_conf="$(cat "$file_http_site")"
     http_site_conf_2="$(echo "$http_site_conf" | sed "s|*:80|${http_vh_prts}|")"
     echo "$http_site_conf_2" | tee "$file_http_site"
     https_site_conf="$(cat "$file_https_site")"
     https_site_conf_2="$(echo "$https_site_conf" | sed "s|_default_:443|${https_vh_prts}|")"
     echo "$https_site_conf_2" | tee "$file_https_site"
     systemctl restart apache2
     touch /etc/startup_script_completed
     

8. Click Create.

Create instance groups

1. In the Google Cloud Console, go to the Instance groups page.
   Go to Instance groups
2. Repeat the following steps to create two unmanaged instance groups each with two VMs in them, using these combinations.
   • Instance group: ig-a, zone: us-west1-a, VMs: vm-a1 and vm-a2
   • Instance group: ig-c, zone: us-west1-c, VMs: vm-c1 and vm-c2
3. Click Create instance group.
4. Click New unmanaged instance group.
5. Set Name as indicated in step 2.
6. In the Location section, choose us-west1 for the Region, and then choose a Zone as indicated in step 2.
7. For Network, enter lb-network.
8. For Subnetwork, enter lb-subnet.
9. In the VM instances section, add the VMs as indicated in step 2.
10. Click Create.

gcloud

1. Create the four VMs by running the following command four times, using these four combinations for [VM-NAME] and [ZONE]. The script contents are identical for all four VMs.

   • [VM-NAME] of vm-a1 and [ZONE] of us-west1-a
   • [VM-NAME] of vm-a2 and [ZONE] of us-west1-a
   • [VM-NAME] of vm-c1 and [ZONE] of us-west1-c
   • [VM-NAME] of vm-c2 and [ZONE] of us-west1-c

   gcloud compute instances create [VM-NAME] \
       --zone=[ZONE] \
       --image-family=debian-10 \
       --image-project=debian-cloud \
       --tags=allow-ssh,allow-health-check \
       --subnet=lb-subnet \
       --metadata=startup-script='#! /bin/bash
   if [ -f /etc/startup_script_completed ]; then
   exit 0
   fi
   apt-get update
   apt-get install apache2 -y
   a2ensite default-ssl
   a2enmod ssl
   file_ports="/etc/apache2/ports.conf"
   file_http_site="/etc/apache2/sites-available/000-default.conf"
   file_https_site="/etc/apache2/sites-available/default-ssl.conf"
   http_listen_prts="Listen 80\nListen 8008\nListen 8080\nListen 8088"
   http_vh_prts="*:80 *:8008 *:8080 *:8088"
   https_listen_prts="Listen 443\nListen 8443"
   https_vh_prts="*:443 *:8443"
   vm_hostname="$(curl -H "Metadata-Flavor:Google" \
   http://169.254.169.254/computeMetadata/v1/instance/name)"
   echo "Page served from: $vm_hostname" | \
   tee /var/www/html/index.html
   prt_conf="$(cat "$file_ports")"
   prt_conf_2="$(echo "$prt_conf" | sed "s|Listen 80|${http_listen_prts}|")"
   prt_conf="$(echo "$prt_conf_2" | sed "s|Listen 443|${https_listen_prts}|")"
   echo "$prt_conf" | tee "$file_ports"
   http_site_conf="$(cat "$file_http_site")"
   http_site_conf_2="$(echo "$http_site_conf" | sed "s|*:80|${http_vh_prts}|")"
   echo "$http_site_conf_2" | tee "$file_http_site"
   https_site_conf="$(cat "$file_https_site")"
   https_site_conf_2="$(echo "$https_site_conf" | sed "s|_default_:443|${https_vh_prts}|")"
   echo "$https_site_conf_2" | tee "$file_https_site"
   systemctl restart apache2
   touch /etc/startup_script_completed'
   

2. Create the two unmanaged instance groups in each zone:

   gcloud compute instance-groups unmanaged create ig-a \
       --zone=us-west1-a
   gcloud compute instance-groups unmanaged create ig-c \
       --zone=us-west1-c
   

3. Add the VMs to the appropriate instance groups:

   gcloud compute instance-groups unmanaged add-instances ig-a \
       --zone=us-west1-a \
       --instances=vm-a1,vm-a2
   gcloud compute instance-groups unmanaged add-instances ig-c \
       --zone=us-west1-c \
       --instances=vm-c1,vm-c2
   

api

For the four VMs, use the following VM names and zones:

• [VM-NAME] of vm-a1 and [ZONE] of us-west1-a
• [VM-NAME] of vm-a2 and [ZONE] of us-west1-a
• [VM-NAME] of vm-c1 and [ZONE] of us-west1-c
• [VM-NAME] of vm-c2 and [ZONE] of us-west1-c

You can get the current DEBIAN_IMAGE_NAME by running the following gcloud command:

gcloud compute images list \
 --filter="family=debian-10"

Create four backend VMs by making four POST requests to the instances.insert method:

POST https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/zones/[ZONE]/instances

{
  "name": "[VM-NAME]",
  "tags": {
    "items": [
      "allow-health-check",
      "allow-ssh"
    ]
  },
  "machineType": "https://www.googleapis.com/compute/v1/projects/project-id/zones/[ZONE]/machineTypes/e2-standard-2",
  "canIpForward": false,
  "networkInterfaces": [
    {
      "network": "https://www.googleapis.com/compute/v1/projects/project-id/global/networks/lb-network",
      "subnetwork": "https://www.googleapis.com/compute/v1/projects/project-id/regions/us-west1/subnetworks/lb-subnet",
      "accessConfigs": [
        {
          "type": "ONE_TO_ONE_NAT",
          "name": "external-nat",
          "networkTier": "PREMIUM"
        }
      ]
    }
  ],
  "disks": [
    {
      "type": "PERSISTENT",
      "boot": true,
      "mode": "READ_WRITE",
      "autoDelete": true,
      "deviceName": "[VM-NAME]",
      "initializeParams": {
        "sourceImage": "projects/debian-cloud/global/images/debian-image-name",
        "diskType": "projects/project-id/zones/zone/diskTypes/pd-standard",
        "diskSizeGb": "10"
      }
    }
  ],
  "metadata": {
    "items": [
      {
        "key": "startup-script",
        "value": "#! /bin/bash\napt-get update\napt-get install apache2 -y\na2ensite default-ssl\na2enmod ssl\nfile_ports=\"/etc/apache2/ports.conf\"\nfile_http_site=\"/etc/apache2/sites-available/000-default.conf\"\nfile_https_site=\"/etc/apache2/sites-available/default-ssl.conf\"\nhttp_listen_prts=\"Listen 80\\nListen 8008\\nListen 8080\\nListen 8088\"\nhttp_vh_prts=\"*:80 *:8008 *:8080 *:8088\"\nhttps_listen_prts=\"Listen 443\\nListen 8443\"\nhttps_vh_prts=\"*:443 *:8443\"\nvm_hostname=\"$(curl -H \"Metadata-Flavor:Google\" \\\nhttp://169.254.169.254/computeMetadata/v1/instance/name)\"\necho \"Page served from: $vm_hostname\" | \\\ntee /var/www/html/index.html\nprt_conf=\"$(cat \"$file_ports\")\"\nprt_conf_2=\"$(echo \"$prt_conf\" | sed \"s|Listen 80|${http_listen_prts}|\")\"\nprt_conf=\"$(echo \"$prt_conf_2\" | sed \"s|Listen 443|${https_listen_prts}|\")\"\necho \"$prt_conf\" | tee \"$file_ports\"\nhttp_site_conf=\"$(cat \"$file_http_site\")\"\nhttp_site_conf_2=\"$(echo \"$http_site_conf\" | sed \"s|*:80|${http_vh_prts}|\")\"\necho \"$http_site_conf_2\" | tee \"$file_http_site\"\nhttps_site_conf=\"$(cat \"$file_https_site\")\"\nhttps_site_conf_2=\"$(echo \"$https_site_conf\" | sed \"s|_default_:443|${https_vh_prts}|\")\"\necho \"$https_site_conf_2\" | tee \"$file_https_site\"\nsystemctl restart apache2"
      }
    ]
  },
  "scheduling": {
    "preemptible": false
  },
  "deletionProtection": false
}

Create two instance groups by making a POST request to the instanceGroups.insert method.

POST https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/zones/us-west1-a/instanceGroups

{
  "name": "ig-a",
  "network": "https://www.googleapis.com/compute/v1/projects/project-id/global/networks/lb-network",
  "subnetwork": "https://www.googleapis.com/compute/v1/projects/project-id/regions/us-west1/subnetworks/lb-subnet"
}

POST https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/zones/us-west1-c/instanceGroups

{
  "name": "ig-c",
  "network": "https://www.googleapis.com/compute/v1/projects/project-id/global/networks/lb-network",
  "subnetwork": "https://www.googleapis.com/compute/v1/projects/project-id/regions/us-west1/subnetworks/lb-subnet"
}

Add instances to each instance group by making a POST request to the instanceGroups.addInstances method.

POST https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/zones/us-west1-a/instanceGroups/ig-a/addInstances

{
  "instances": [
    {
      "instance": "https://www.googleapis.com/compute/v1/projects/project-id/zones/us-west1-a/instances/vm-a1",
      "instance": "https://www.googleapis.com/compute/v1/projects/project-id/zones/us-west1-a/instances/vm-a2"
    }
  ]
}

POST https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/zones/us-west1-c/instanceGroups/ig-c/addInstances

{
  "instances": [
    {
      "instance": "https://www.googleapis.com/compute/v1/projects/project-id/zones/us-west1-c/instances/vm-c1",
      "instance": "https://www.googleapis.com/compute/v1/projects/project-id/zones/us-west1-c/instances/vm-c2"
    }
  ]
}

Creating a client VM

This example creates a client VM (vm-client) in the same region as the backend (server) VMs. The client is used to validate the load balancer's configuration and demonstrate expected behavior as described in the testing section.

Console

1. In the Google Cloud Console, go to the VM instances page.
   Go to VM instances
2. Click Create instance.
3. Set the Name to vm-client.
4. Set the Zone to us-west1-a.
5. Click Management, security, disks, networking, sole tenancy and make the following changes:
   • Click Networking and add the allow-ssh to Network tags.
   • Click the edit button under Network interfaces and make the following changes then click Done:
     • Network: lb-network
     • Subnet: lb-subnet
     • Primary internal IP: Ephemeral (automatic)
     • External IP: Ephemeral
6. Click Create.

gcloud

The client VM can be in any zone in the same region as the load balancer, and it can use any subnet in that region. In this example, the client is in the us-west1-a zone, and it uses the same subnet as the backend VMs.

gcloud compute instances create vm-client \
    --zone=us-west1-a \
    --image-family=debian-10 \
    --image-project=debian-cloud \
    --tags=allow-ssh \
    --subnet=lb-subnet

api

Make a POST request to the instances.insert method.

POST https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/zones/us-west1-a/instances

{
  "name": "vm-client",
  "tags": {
    "items": [
      "allow-ssh"
    ]
  },
  "machineType": "https://www.googleapis.com/compute/v1/projects/project-id/zones/us-west1-a/machineTypes/e2-standard-2",
  "canIpForward": false,
  "networkInterfaces": [
    {
      "network": "https://www.googleapis.com/compute/v1/projects/project-id/global/networks/lb-network",
      "subnetwork": "https://www.googleapis.com/compute/v1/projects/project-id/regions/us-west1/subnetworks/lb-subnet",
      "accessConfigs": [
        {
          "type": "ONE_TO_ONE_NAT",
          "name": "external-nat",
          "networkTier": "PREMIUM"
        }
      ]
    }
  ],
  "disks": [
    {
      "type": "PERSISTENT",
      "boot": true,
      "mode": "READ_WRITE",
      "autoDelete": true,
      "deviceName": "vm-client",
      "initializeParams": {
        "sourceImage": "projects/debian-cloud/global/images/debian-image-name",
        "diskType": "projects/project-id/zones/us-west1-a/diskTypes/pd-standard",
        "diskSizeGb": "10"
      }
    }
  ],
  "scheduling": {
    "preemptible": false
  },
  "deletionProtection": false
}

Configuring load balancer components

These steps configure all of the internal TCP/UDP load balancer components starting with the health check and backend service, and then the frontend components:

• Health check: In this example, we use an HTTP health check that simply checks for an HTTP 200 (OK) response. For more information, see the health checks section of the Internal TCP/UDP Load Balancing overview.

• Backend service: Because we need to pass HTTP traffic through the internal load balancer, we need to use TCP, not UDP.

• Forwarding rule: This example creates a single internal forwarding rule.

• Internal IP address: In this example, we specify an internal IP address, 10.1.2.99, when we create the forwarding rule.

Console

Create the load balancer and configure a backend service

1. In the Google Cloud Console, go to the Load balancing page.
   Go to Load balancing
2. Click Create load balancer.
3. Under TCP load balancing, click Start configuration.
4. Under Internet facing or internal only select Only between my VMs.
5. Click Continue.
6. Set the Name to be-ilb.
7. Click Backend configuration and make the following changes:
   1. Region: us-west1
   2. Network: lb-network
   3. Under Backends, in the New item section, select the ig-a instance group and click Done.
   4. Click Add backend. In the New item section that appears, select the ig-c instance group and click Done again.
   5. Under Health check, choose Create another health check, enter the following information, and click Save and continue:
      • Name: hc-http-80
      • Protocol: HTTP
      • Port: 80
      • Proxy protocol: NONE
      • Request path: / Note that when you use the Cloud Console to create your load balancer, the health check is global. If you want to create a regional health check, use gcloud or the API.
   6. Verify that there is a blue check mark next to Backend configuration before continuing. Review this step if not.
8. Click Frontend configuration. In the New Frontend IP and port section, make the following changes:
   1. Name: fr-ilb
   2. Subnetwork: lb-subnet
   3. From Internal IP, choose Reserve a static internal IP address, enter the following information, and click Reserve:
      • Name: ip-ilb
      • Static IP address: Let me choose
      • Custom IP address: 10.1.2.99
   4. Ports: Choose Multiple, and enter 80,8008,8080,8088 for the Port number.
   5. Verify that there is a blue check mark next to Frontend configuration before continuing. Review this step if not.
9. Click Review and finalize. Double-check your settings.
10. Click Create.

gcloud

1. Create a new regional HTTP health check to test HTTP connectivity to the VMs on 80.

   gcloud compute health-checks create http hc-http-80 \
       --region=us-west1 \
       --port=80
   

2. Create the backend service for HTTP traffic:

   gcloud compute backend-services create be-ilb \
       --load-balancing-scheme=internal \
       --protocol=tcp \
       --region=us-west1 \
       --health-checks=hc-http-80 \
       --health-checks-region=us-west1
   

3. Add the two instance groups to the backend service:

   gcloud compute backend-services add-backend be-ilb \
       --region=us-west1 \
       --instance-group=ig-a \
       --instance-group-zone=us-west1-a
   gcloud compute backend-services add-backend be-ilb \
       --region=us-west1 \
       --instance-group=ig-c \
       --instance-group-zone=us-west1-c
   

4. Create a forwarding rule for the backend service. When you create the forwarding rule, specify 10.1.2.99 for the internal IP address in the subnet.

   gcloud compute forwarding-rules create fr-ilb \
       --region=us-west1 \
       --load-balancing-scheme=internal \
       --network=lb-network \
       --subnet=lb-subnet \
       --address=10.1.2.99 \
       --ip-protocol=TCP \
       --ports=80,8008,8080,8088 \
       --backend-service=be-ilb \
       --backend-service-region=us-west1
   

api

Create the health check by making a POST request to the regionHealthChecks.insert method.

POST https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/regions/us-west1/regionHealthChecks

{
  "name": "hc-http-80",
  "type": "HTTP",
  "httpHealthCheck": {
    "port": 80
  }
}

Create the regional backend service by making a POST request to the regionBackendServices.insert method.

POST https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/regions/us-west1/backendServices

{
  "name": "be-ilb",
  "backends": [
    {
      "group": "https://www.googleapis.com/compute/v1/projects/project-id/zones/us-west1-a/instanceGroups/ig-a",
      "balancingMode": "CONNECTION"
    },
    {
      "group": "https://www.googleapis.com/compute/v1/projects/project-id/zones/us-west1-c/instanceGroups/ig-c",
      "balancingMode": "CONNECTION"
    }
  ],
  "healthChecks": [
    "https://www.googleapis.com/compute/v1/projects/project-id/regions/us-west1/healthChecks/hc-http-80"
  ],
  "loadBalancingScheme": "INTERNAL",
  "connectionDraining": {
    "drainingTimeoutSec": 0
   }
}

Create the forwarding rule by making a POST request to the forwardingRules.insert method.

POST https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/regions/us-west1/forwardingRules

{
  "name": "fr-ilb",
  "IPAddress": "10.1.2.99",
  "IPProtocol": "TCP",
  "ports": [
    "80", "8008", "8080", "8088"
  ],
  "loadBalancingScheme": "INTERNAL",
  "subnetwork": "https://www.googleapis.com/compute/v1/projects/project-id/regions/us-west1/subnetworks/lb-subnet",
  "network": "https://www.googleapis.com/compute/v1/projects/project-id/global/networks/lb-network",
  "backendService": "https://www.googleapis.com/compute/v1/projects/project-id/regions/us-west1/backendServices/be-ilb",
  "networkTier": "PREMIUM"
}

Testing

These tests show how to validate your load balancer configuration and learn about its expected behavior.

Testing load balancing

This test contacts the load balancer from a separate client VM; that is, not from a backend VM of the load balancer. The expected behavior is for traffic to be distributed among the four backend VMs because no session affinity has been configured.

1. Connect to the client VM instance.

   gcloud compute ssh vm-client --zone=us-west1-a
   

2. Make a web request to the load balancer using curl to contact its IP address. Repeat the request so you can see that responses come from different backend VMs. The name of the VM generating the response is displayed in the text in the HTML response, by virtue of the contents of /var/www/html/index.html on each backend VM. Expected responses look like: Page served from: vm-a1, Page served from: vm-a2, and so on.

   curl http://10.1.2.99
   

   The forwarding rule is configured to serve ports 80, 8008, 8080, and 8088. To send traffic to those other ports, append a colon (:) and the port number after the IP address, like this:

   curl http://10.1.2.99:8008
   

   • If you add a service label to the internal forwarding rule, you can use internal DNS to contact the load balancer using its service name.

     curl http://web-test.fr-ilb.il4.us-west1.lb.project-id.internal
     

Pinging the load balancer's IP address

This test demonstrates an expected behavior: You cannot ping the IP address of the load balancer. This is because internal TCP/UDP load balancers are implemented in virtual network programming — they are not separate devices.

1. Connect to the client VM instance.

   gcloud compute ssh vm-client --zone=us-west1-a
   

2. Attempt to ping the IP address of the load balancer. Notice that you do not get a response and that the ping command times out after 10 seconds in this example.

   timeout 10 ping 10.1.2.99
   

Sending requests from load balanced VMs

This test demonstrates that requests to the load balancer that originate from any of the backend VMs — the server VMs being load balanced — are always answered by the same VM making the request.

Internal TCP/UDP Load Balancing is implemented using virtual network programming and VM configuration in the guest OS. On Linux VMs, the Linux Guest Environment performs the local configuration by installing a route in the guest OS routing table. Because of this local route, traffic to the IP address of the load balancer stays on the load balanced VM itself. (This local route is different from the routes in the VPC network.)

Caution: Don't rely on making requests to an internal TCP/UDP load balancer from a VM being load balanced (in the backend service for that load balancer). A request is always sent to the VM that makes the request, and health check information is ignored. Further, the backend can respond to traffic sent using protocols and destination ports other than those configured on the load balancer's internal forwarding rule.

1. Connect to a backend VM, such as vm-a1:

   gcloud compute ssh vm-a1 --zone=us-west1-a
   

   Important: You won't be able to connect to a backend VM in this way if you have removed its external IP address. See connecting to instances that do not have external IP addresses for connection options if your backend VMs do not have external IP addresses.

2. Make a web request to the load balancer (by IP address or service name) using curl. Repeat the request, and note that the response is sent from the backend VM that makes the request. The expected response when testing from vm-a1 is always: Page served from: vm-a1.

   curl http://10.1.2.99
   

3. Inspect the local routing table, looking for a destination that matches the IP address of the load balancer itself, 10.1.2.99. This route is a necessary part of Internal TCP/UDP Load Balancing, but it also demonstrates why a request from a VM behind the load balancer is always responded to by the same VM.

   ip route show table local | grep 10.1.2.99
   

Additional configuration options

This section expands on the configuration example to provide alternative and additional configuration options. All of the tasks are optional. You can perform them in any order.

Enabling global access

You can enable global access for your example internal TCP/UDP load balancer to make it accessible to clients in all regions. The backends of your example load balancer must still be located in one region (us-west1).

Internal TCP/UDP Load Balancing with Global Access (click to enlarge)

To configure global access, make the following configuration changes.

Console

Edit the load balancer's forwarding rule

1. In the Google Cloud Console, go to the Load balancing page.
   Go to Load balancing

2. In the Name column, click your internal TCP/UDP load balancer. The example load balancer is named be-ilb.

3. Click Frontend configuration.

4. Click Edit .

5. Under Global access, select Enable.

6. Click Done.

7. Click Update.

On the Load balancer details page, verify that the frontend configuration says Regional (REGION) with global access.

gcloud

1. Update the example load balancer's forwarding rule, fr-ilb to include the --allow-global-access flag.

   gcloud compute forwarding-rules update fr-ilb \
      --region=us-west1 \
      --allow-global-access
   

2. Make sure that the forwarding rule allows global access.

   gcloud compute forwarding-rules describe fr-ilb \
      --region=us-west1 \
      --format="get(name,region,allowGlobalAccess)"
   

   The word True appears in the output, after the name and region of the forwarding rule, when global access is enabled.

api

Make a PATCH request to the forwardingRules/patch method.

PATCH https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/regions/us-west1/forwardingRules/fr-ilb

{
  "allowGlobalAccess": true
}

Create a VM client to test global access

Console

1. In the Google Cloud Console, go to the VM instances page.
   Go to VM instances
2. Click Create instance.
3. Set the Name to vm-client2.
4. Set the Zone to europe-west1-b.
5. Click Management, security, disks, networking, sole tenancy and make the following changes:
   • Click Networking and add the allow-ssh to Network tags.
   • Click the edit button under Network interfaces and make the following changes then click Done:
     • Network: lb-network
     • Subnet: europe-subnet
     • Primary internal IP: Ephemeral (automatic)
     • External IP: Ephemeral
6. Click Create.

gcloud

The client VM can be in any zone in the same region as the load balancer, and it can use any subnet in that region. In this example, the client is in the europe-west1-b zone, and it uses the same subnet as the backend VMs.

gcloud compute instances create vm-client2 \
    --zone=europe-west1-b \
    --image-family=debian-10 \
    --image-project=debian-cloud \
    --tags=allow-ssh \
    --subnet=europe-subnet

api

Make a POST request to the instances.insert method.

POST https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/zones/europe-west1-b/instances

{
  "name": "vm-client2",
  "tags": {
    "items": [
      "allow-ssh"
    ]
  },
  "machineType": "https://www.googleapis.com/compute/v1/projects/project-id/zones/europe-west1-b/machineTypes/e2-standard-2",
  "canIpForward": false,
  "networkInterfaces": [
    {
      "network": "https://www.googleapis.com/compute/v1/projects/project-id/global/networks/lb-network",
      "subnetwork": "https://www.googleapis.com/compute/v1/projects/project-id/regions/europe-west1/subnetworks/europe-subnet",
      "accessConfigs": [
        {
          "type": "ONE_TO_ONE_NAT",
          "name": "external-nat",
          "networkTier": "PREMIUM"
        }
      ]
    }
  ],
  "disks": [
    {
      "type": "PERSISTENT",
      "boot": true,
      "mode": "READ_WRITE",
      "autoDelete": true,
      "deviceName": "vm-client2",
      "initializeParams": {
        "sourceImage": "projects/debian-cloud/global/images/debian-image-name",
        "diskType": "projects/project-id/zones/europe-west1-b/diskTypes/pd-standard",
        "diskSizeGb": "10"
      }
    }
  ],
  "scheduling": {
    "preemptible": false
  },
  "deletionProtection": false
}

Connect to the VM client and test connectivity

gcloud compute ssh vm-client2 --zone=europe-west1-b

Test connecting to the load balancer on all configured ports, as you did from the vm-client in the us-west1 region. Test HTTP connectivity on the four ports configured on the forwarding rule:

curl http://10.1.2.99
curl http://10.1.2.99:8008
curl http://10.1.2.99:8080
curl http://10.1.2.99:8088

Configuring managed instance groups

The example configuration created two unmanaged instance groups. You can instead use managed instance groups, including zonal and regional managed instance groups, as backends for Internal TCP/UDP Load Balancing.

Managed instance groups require that you create an instance template. This procedure demonstrates how to replace the two unmanaged zonal instance groups from the example with a single, regional managed instance group. A regional managed instance group automatically creates VMs in multiple zones of the region, making it simpler to distribute production traffic among zones.

Managed instance groups also support autoscaling and autohealing. If you use autoscaling with Internal TCP/UDP Load Balancing, you cannot scale based on load balancing.

This procedure shows you how to modify the backend service for the example internal TCP/UDP load balancer so that it uses a regional managed instance group.

Console

Instance template

1. In the Google Cloud Console, go to the VM instance templates page.
   Go to VM instance templates
2. Click Create instance template.
3. Set the Name to template-vm-ilb.
4. Choose a machine type.
5. For Boot disk, click Change, choose the Debian operating system and the 10 (buster) version.
6. Click Save to confirm the options for this boot disk.

7. Click Management, security, disks, networking, sole tenancy.

   • Click Networking and make the following changes:
     • Network: lb-network
     • Subnet: lb-subnet
     • Network tags: allow-ssh and allow-health-check
   • Click Management. In the Startup script field, copy and paste the following script contents:

   #! /bin/bash
   if [ -f /etc/startup_script_completed ]; then
   exit 0
   fi
   apt-get update
   apt-get install apache2 -y
   a2ensite default-ssl
   a2enmod ssl
   file_ports="/etc/apache2/ports.conf"
   file_http_site="/etc/apache2/sites-available/000-default.conf"
   file_https_site="/etc/apache2/sites-available/default-ssl.conf"
   http_listen_prts="Listen 80\nListen 8008\nListen 8080\nListen 8088"
   http_vh_prts="*:80 *:8008 *:8080 *:8088"
   https_listen_prts="Listen 443\nListen 8443"
   https_vh_prts="*:443 *:8443"
   vm_hostname="$(curl -H "Metadata-Flavor:Google" \
   http://169.254.169.254/computeMetadata/v1/instance/name)"
   echo "Page served from: $vm_hostname" | \
   tee /var/www/html/index.html
   echo "Page served from: $vm_hostname" | \
   tee /var/www/html/index.html
   prt_conf="$(cat "$file_ports")"
   prt_conf_2="$(echo "$prt_conf" | sed "s|Listen 80|${http_listen_prts}|")"
   prt_conf="$(echo "$prt_conf_2" | sed "s|Listen 443|${https_listen_prts}|")"
   echo "$prt_conf" | tee "$file_ports"
   http_site_conf="$(cat "$file_http_site")"
   http_site_conf_2="$(echo "$http_site_conf" | sed "s|*:80|${http_vh_prts}|")"
   echo "$http_site_conf_2" | tee "$file_http_site"
   https_site_conf="$(cat "$file_https_site")"
   https_site_conf_2="$(echo "$https_site_conf" | sed "s|_default_:443|${https_vh_prts}|")"
   echo "$https_site_conf_2" | tee "$file_https_site"
   systemctl restart apache2
   touch /etc/startup_script_completed
   

8. Click Create.

Managed instance group

1. In the Google Cloud Console, go to the Instance groups page.
   Go to VM instance groups
2. Click Create instance group.
3. Set the Name to ig-ilb.
4. For Location, choose Multi-zone, and set the Region to us-west1.
5. Set the Instance template to template-vm-ilb.
6. (Optional) Configure autoscaling. You cannot autoscale the instance group based on HTTP load balancing usage because the instance group is a backend for Internal TCP/UDP Load Balancing.
7. Set the Minimum number of instances to 1 and the Maximum number of instances to 6.
8. (Optional) Configure autohealing. If you configure autohealing, use the same health check used by the backend service for the internal TCP/UDP load balancer. In this example, use hc-http-80.
9. Click Create.

gcloud

1. Create the instance template. Optionally, you can set other parameters, such as machine type, for the image template to use.

   gcloud compute instance-templates create template-vm-ilb \
       --image-family=debian-10 \
       --image-project=debian-cloud \
       --tags=allow-ssh,allow-health-check \
       --subnet=lb-subnet \
       --region=us-west1 \
       --network=lb-network \
          --metadata=startup-script='#! /bin/bash
            if [ -f /etc/startup_script_completed ]; then
            exit 0
            fi
            apt-get update
            apt-get install apache2 -y
            a2ensite default-ssl
            a2enmod ssl
            file_ports="/etc/apache2/ports.conf"
            file_http_site="/etc/apache2/sites-available/000-default.conf"
            file_https_site="/etc/apache2/sites-available/default-ssl.conf"
            http_listen_prts="Listen 80\nListen 8008\nListen 8080\nListen 8088"
            http_vh_prts="*:80 *:8008 *:8080 *:8088"
            https_listen_prts="Listen 443\nListen 8443"
            https_vh_prts="*:443 *:8443"
            vm_hostname="$(curl -H "Metadata-Flavor:Google" \
            http://169.254.169.254/computeMetadata/v1/instance/name)"
            echo "Page served from: $vm_hostname" | \
            tee /var/www/html/index.html
            prt_conf="$(cat "$file_ports")"
            prt_conf_2="$(echo "$prt_conf" | sed "s|Listen 80|${http_listen_prts}|")"
            prt_conf="$(echo "$prt_conf_2" | sed "s|Listen 443|${https_listen_prts}|")"
            echo "$prt_conf" | tee "$file_ports"
            http_site_conf="$(cat "$file_http_site")"
            http_site_conf_2="$(echo "$http_site_conf" | sed "s|*:80|${http_vh_prts}|")"
            echo "$http_site_conf_2" | tee "$file_http_site"
            https_site_conf="$(cat "$file_https_site")"
            https_site_conf_2="$(echo "$https_site_conf" | sed "s|_default_:443|${https_vh_prts}|")"
            echo "$https_site_conf_2" | tee "$file_https_site"
            systemctl restart apache2
            touch /etc/startup_script_completed'
   

2. Create one regional managed instance group using the template:

   gcloud compute instance-groups managed create ig-ilb \
       --template=template-vm-ilb \
       --region=us-west1 \
       --size=6
   

3. Add the regional managed instance group as a backend to the backend service that you already created:

   gcloud compute backend-services add-backend be-ilb \
       --region=us-west1 \
       --instance-group=ig-ilb \
       --instance-group-region=us-west1
   

4. Disconnect the two unmanaged (zonal) instance groups from the backend service:

   gcloud compute backend-services remove-backend be-ilb \
       --region=us-west1 \
       --instance-group=ig-a \
       --instance-group-zone=us-west1-a
   gcloud compute backend-services remove-backend be-ilb \
       --region=us-west1 \
       --instance-group=ig-c \
       --instance-group-zone=us-west1-c
   

Removing external IP addresses from backend VMs

When you created the backend VMs, each was assigned an ephemeral external IP address so it could download Apache via a startup script. Because the backend VMs are only used by an internal TCP/UDP load balancer, you can remove their external IP addresses. Removing external IP addresses prevents the backend VMs from accessing the internet directly.

Important: Removing the external IP address from a VM limits how you can connect to it. See connecting to instances that do not have external IP addresses for details. It's useful to leave external IP addresses on backend VMs if you need to demonstrate how requests from a backend VM to the IP address of the load balancer are handled.

Console

1. In the Google Cloud Console, go to the VM instances page.
   Go to VM instances
2. Repeat the following steps for each backend VM.
3. Click the name of the backend VM (for example, vm-a1) to view VM instance details.
4. Click Edit .
5. In the Network interfaces section, click the Edit button.
6. From the External IP pop-up, choose None, and click Done.
7. Click Save.

gcloud

1. To look up the zone for an instance – for example, if you're using a regional managed instance group – run the following command for each instance to determine its zone. Replace [SERVER-VM] with the name of the VM to look up.

   gcloud compute instances list --filter="name=[SERVER-VM]"
   

2. Repeat the following step for each backend VM. Replace [SERVER-VM] with the name of the VM, and replace and [ZONE] with the VM's zone.

   gcloud compute instances delete-access-config [SERVER-VM] \
       --zone=[ZONE] \
       --access-config-name=external-nat
   

api

Make a POST request to the instances.deleteAccessConfig method for each backend VM, replacingvm-a1 with the name of the VM, and replacing and us-west1-a with the VM's zone.

POST https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/zones/us-west1-a/instances/vm-a1/deleteAccessConfig?accessConfig=external-nat&networkInterface=None

Accepting traffic on all ports

The load balancer's forwarding rule, not its backend service, determines the port or ports on which the load balancer accepts traffic. For information about the purpose of each component, see Components.

When you created this example load balancer's forwarding rule, you configured ports 80, 8008, 8080, and 8088. The startup script that installs Apache also configures it to accept HTTPS connections on ports 443 and 8443.

To support these six ports, you can configure the forwarding rule to accept traffic on all ports. With this strategy, you can also configure the firewall rule or rules that allow incoming connections to backend VMs so that they only permit certain ports.

This procedure shows you how to replace the example load balancer's forwarding rule with one that accepts traffic on all ports.

For more information about when to use this setup, see Internal TCP/UDP Load Balancing and forwarding rules with a common IP address.

Console

Delete your forwarding rule and create a new one

1. In the Google Cloud Console, go to the Load balancing page.
   Go to Load balancing
2. Click your be-ilb load balancer and click Edit.
3. Click Frontend configuration.
4. Hover over your 10.1.2.9 forwarding rule and click the trashcan icon to delete it.
5. Click Add frontend IP and port.
6. In the New Frontend IP and port section, make the following changes:
   1. Name: fr-ilb
   2. Subnetwork: lb-subnet
   3. From Internal IP, select ip-ilb
   4. Ports: All.
   5. Click Done.
   6. Verify that there is a blue check mark next to Frontend configuration before continuing. Review this step if not.
7. Click Review and finalize. Double-check your settings.
8. Click Create.

gcloud

1. Delete your existing forwarding rule, fr-ilb.

   gcloud compute forwarding-rules delete fr-ilb \
       --region=us-west1
   

2. Create a replacement forwarding rule, with the same name, whose port configuration uses the keyword ALL. The other parameters for the forwarding rule remain the same.

   gcloud compute forwarding-rules create fr-ilb \
       --region=us-west1 \
       --load-balancing-scheme=internal \
       --network=lb-network \
       --subnet=lb-subnet \
       --address=10.1.2.99 \
       --ip-protocol=TCP \
       --ports=ALL \
       --backend-service=be-ilb \
       --backend-service-region=us-west1
   

api

Delete the forwarding rule by making a DELETE request to the forwardingRules.delete method.

DELETE https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/regions/us-west1/forwardingRules/fr-ilb

Create the forwarding rule by making a POST request to the forwardingRules.insert method.

POST https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/regions/us-west1/forwardingRules

{
  "name": "fr-ilb",
  "IPAddress": "10.1.2.99",
  "IPProtocol": "TCP",
  "allPorts": true,
  "loadBalancingScheme": "INTERNAL",
  "subnetwork": "https://www.googleapis.com/compute/v1/projects/project-id/regions/us-west1/subnetworks/lb-subnet",
  "network": "https://www.googleapis.com/compute/v1/projects/project-id/global/networks/lb-network",
  "backendService": "https://www.googleapis.com/compute/v1/projects/project-id/regions/us-west1/backendServices/be-ilb",
  "networkTier": "PREMIUM"
}

Testing the traffic on all ports setup

Connect to the client VM instance and test HTTP and HTTPS connections.

• Connect to the client VM:

  gcloud compute ssh vm-client --zone=us-west1-a
  

• Test HTTP connectivity on all four ports:

  curl http://10.1.2.99
  curl http://10.1.2.99:8008
  curl http://10.1.2.99:8080
  curl http://10.1.2.99:8088
  

• Test HTTPS connectivity on ports 443 and 8443. The --insecure flag is required because each Apache server in the example setup uses a self-signed certificate.

  curl https://10.1.2.99 --insecure
   curl https://10.1.2.99:8443 --insecure
  

• Observe that HTTP requests (on all four ports) and HTTPS requests (on both ports) are distributed among all of the backend VMs.

Accepting traffic on multiple ports using two forwarding rules

When you created this example load balancer's forwarding rule, you configured ports 80, 8008, 8080, and 8088. The startup script that installs Apache also configures it to accept HTTPS connections on ports 443 and 8443.

An alternative strategy to configuring a single forwarding rule to accept traffic on all ports is to create multiple forwarding rules, each supporting five or fewer ports.

This procedure shows you how to replace the example load balancer's forwarding rule with two forwarding rules, one handling traffic on ports 80, 8008, 8080, and 8088, and the other handling traffic on ports 443 and 8443.

For more information about when to use this setup, see Internal TCP/UDP Load Balancing and forwarding rules with a common IP address.

Console

1. In the Google Cloud Console, go to the Forwarding Rules page.
   Go to Forwarding Rules
2. Select fr-ilb.
3. Click Delete.
4. Go to the Load balancing page.
   Go to the Load balancing page
5. Select your internal TCP/UDP load balancer. In this example, be-ilb.
6. Click Edit.
7. Click Frontend configuration.
8. Click Add frontend IP and port.
9. In the New Frontend IP and port section, make the following changes:
   1. Name: fr-ilb-http
   2. Subnetwork: lb-subnet
   3. Under Purpose, select Shared.
   4. From Internal IP, choose Reserve a static internal IP address, enter the following information, and click Reserve:
      • Name: internal-10-1-2-99
      • Static IP address: Let me choose
      • Custom IP address: 10.1.2.99
      • Click Reserve.
   5. Ports: Choose Multiple, and enter 80,8008,8080,8088 for the Port number.
   6. Click Done.
10. Click Add frontend IP and port.
11. In the New Frontend IP and port section, make the following changes:
    1. Name: fr-ilb-https
    2. Subnetwork: lb-subnet
    3. Under Purpose, select Shared.
    4. Under IP address, select internal-10-1-2-99.
    5. Ports: Choose Multiple, and enter 443,8443 for the Port number.
    6. Click Done.
    7. Verify that there is a blue check mark next to Frontend configuration before continuing. Review this step if not.
12. Click Review and finalize. Double-check your settings.
13. Click Update.

gcloud

1. Delete your existing forwarding rule, fr-ilb.

   gcloud compute forwarding-rules delete fr-ilb \
       --region=us-west1
   

2. Create a static (reserved) internal IP address for 10.1.2.99 and set its --purpose flag to SHARED_LOADBALANCER_VIP. The --purpose flag is required so that two internal forwarding rules can use the same internal IP address.

   gcloud compute addresses create internal-10-1-2-99 \
       --region=us-west1 \
       --subnet=lb-subnet \
       --addresses=10.1.2.99 \
       --purpose=SHARED_LOADBALANCER_VIP
   

   1. Create two replacement forwarding rules with the following parameters:

   gcloud compute forwarding-rules create fr-ilb-http \
       --region=us-west1 \
       --load-balancing-scheme=internal \
       --network=lb-network \
       --subnet=lb-subnet \
       --address=10.1.2.99 \
       --ip-protocol=TCP \
       --ports=80,8008,8080,8088 \
       --backend-service=be-ilb \
       --backend-service-region=us-west1
   

   gcloud compute forwarding-rules create fr-ilb-https \
       --region=us-west1 \
       --load-balancing-scheme=internal \
       --network=lb-network \
       --subnet=lb-subnet \
       --address=10.1.2.99 \
       --ip-protocol=TCP \
       --ports=443,8443 \
       --backend-service=be-ilb \
       --backend-service-region=us-west1
   

api

Delete the forwarding rule by making a DELETE request to the forwardingRules.delete method.

DELETE https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/regions/us-west1/forwardingRules/fr-ilb

Create a static (reserved) internal IP address for 10.1.2.99 and set its purpose to SHARED_LOADBALANCER_VIP by making a POST request to the addresses.insert method.

POST https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/regions/us-west1/addresses

{
  "name": "internal-10-1-2-99",
  "address": "10.1.2.99",
  "prefixLength": 32,
  "addressType": INTERNAL,
  "purpose": SHARED_LOADBALANCER_VIP,
  "subnetwork": "https://www.googleapis.com/compute/v1/projects/project-id/regions/us-west1/subnetworks/lb-subnet"
}

Create two forwarding rules by making two POST requests to the forwardingRules.insert method.

POST https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/regions/us-west1/forwardingRules

{
  "name": "fr-ilb-http",
  "IPAddress": "10.1.2.99",
  "IPProtocol": "TCP",
  "ports": [
    "80", "8008", "8080",  "8088"
  ],
  "loadBalancingScheme": "INTERNAL",
  "subnetwork": "https://www.googleapis.com/compute/v1/projects/project-id/regions/us-west1/subnetworks/lb-subnet",
  "network": "https://www.googleapis.com/compute/v1/projects/project-id/global/networks/lb-network",
  "backendService": "https://www.googleapis.com/compute/v1/projects/project-id/regions/us-west1/backendServices/be-ilb",
  "networkTier": "PREMIUM"
}

{
  "name": "fr-ilb-https",
  "IPAddress": "10.1.2.99",
  "IPProtocol": "TCP",
  "ports": [
    "443", "8443"
  ],
  "loadBalancingScheme": "INTERNAL",
  "subnetwork": "https://www.googleapis.com/compute/v1/projects/project-id/regions/us-west1/subnetworks/lb-subnet",
  "network": "https://www.googleapis.com/compute/v1/projects/project-id/global/networks/lb-network",
  "backendService": "https://www.googleapis.com/compute/v1/projects/project-id/regions/us-west1/backendServices/be-ilb",
  "networkTier": "PREMIUM"
}

Testing the traffic on multiple ports setup

Connect to the client VM instance and test HTTP and HTTPS connections.

• Connect to the client VM:

  gcloud compute ssh vm-client --zone=us-west1-a
  

• Test HTTP connectivity on all four ports:

  curl http://10.1.2.99
  curl http://10.1.2.99:8008
  curl http://10.1.2.99:8080
  curl http://10.1.2.99:8088
  

• Test HTTPS connectivity on ports 443 and 8443. The --insecure flag is required because each Apache server in the example setup uses a self-signed certificate.

  curl https://10.1.2.99 --insecure
  curl https://10.1.2.99:8443 --insecure
  

• Observe that HTTP requests (on all four ports) and HTTPS requests (on both ports) are distributed among all of the backend VMs.

Using session affinity

The example configuration creates a backend service without session affinity.

This procedure shows you how to update the backend service for the example internal TCP/UDP load balancer so that it uses session affinity based on a hash created from the client's IP addresses and the IP address of the load balancer's internal forwarding rule. Note that, currently, setting session affinity for an internal UDP load balancer is not supported.

Console

1. In the Google Cloud Console, go to the Load balancing page.
   Go to Load balancing
2. Click be-ilb (the name of the backend service that you created for this example) and click Edit.
3. On the Edit Internal load balancer page, click Backend configuration.
4. Select Client IP from the Session affinity pop-up menu.
5. Click Update.

gcloud

Use the following gcloud command to update the be-ilb backend service, specifying client IP session affinity:

gcloud compute backend-services update be-ilb \
    --region=us-west1 \
    --session-affinity CLIENT_IP

api

Make a PATCH request to the regionBackendServices/patch method.

PATCH https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/regions/us-west1/backendServices/be-ilb

{
  "sessionAffinity": "CLIENT_IP"
}

For more information about using session affinity to influence traffic distribution and a description of each option, see Traffic distribution.

Creating a forwarding rule in another subnet

This procedure creates a second IP address and forwarding rule in a different subnet to demonstrate that you can create multiple forwarding rules for one internal TCP/UDP load balancer. The region for the forwarding rule must match the region of the backend service.

Subject to firewall rules, clients in any subnet in the region can contact either internal TCP/UDP load balancer IP address.

Console

Add the second subnet

1. In the Google Cloud Console, go to the VPC networks page.
   Go to VPC networks
2. Click Create VPC network.
3. Click lb-network.
4. In the Subnets section:
   • Click Add subnet.
   • In the New subnet section, enter the following information:
     • Name: second-subnet
     • Region: us-west1
     • IP address range: 10.5.6.0/24
     • Click Add.

Add the second forwarding rule

1. In the Google Cloud Console, go to the Load balancing page.
   Go to Load balancing
2. Click your be-ilb load balancer and click Edit.
3. Click Frontend configuration.
4. Click Add frontend IP and port.
5. In the New Frontend IP and port section, make the following changes:
   1. Name: fr-ilb-2
   2. Subnetwork: second-subnet
   3. From Internal IP, select ip-ilb
   4. Ports: 80, 443.
   5. Click Done.
   6. Verify that there is a blue check mark next to Frontend configuration before continuing. Review this step if not.
6. Click Review and finalize. Double-check your settings.
7. Click Create.

gcloud

1. Create a second subnet in the lb-network network in the us-west1 region:

   gcloud compute networks subnets create second-subnet \
      --network=lb-network \
      --range=10.5.6.0/24 \
      --region=us-west1
   

2. Create a second forwarding rule for ports 80 and 443. The other parameters for this rule, including IP address and backend service, are the same as for the primary forwarding rule, fr-ilb.

   gcloud compute forwarding-rules create fr-ilb-2 \
      --region=us-west1 \
      --load-balancing-scheme=internal \
      --network=lb-network \
      --subnet=second-subnet \
      --address=10.5.6.99 \
      --ip-protocol=TCP \
      --ports=80,443 \
      --backend-service=be-ilb \
      --backend-service-region=us-west1
   

api

Make a POST requests to the subnetworks.insert method.

POST https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/regions/us-west1/subnetworks

{
  "name": "second-subnet",
  "network": "https://www.googleapis.com/compute/v1/projects/project-id/global/networks/lb-network",
  "ipCidrRange": "10.5.6.0/24",
  "privateIpGoogleAccess": false
}

Create the forwarding rule by making a POST request to the forwardingRules.insert method.

POST https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/regions/us-west1/forwardingRules

{
 "name": "fr-ilb-2",
 "IPAddress": "10.5.6.99",
 "IPProtocol": "TCP",
 "ports": [
   "80", "443"
 ],
 "loadBalancingScheme": "INTERNAL",
 "subnetwork": "https://www.googleapis.com/compute/v1/projects/project-id/regions/us-west1/subnetworks/lb-subnet",
 "network": "https://www.googleapis.com/compute/v1/projects/project-id/global/networks/lb-network",
 "backendService": "https://www.googleapis.com/compute/v1/projects/project-id/regions/us-west1/backendServices/be-ilb",
 "networkTier": "PREMIUM"
}

Testing the new forwarding rule

1. Connect to the client VM instance and test HTTP and HTTPS connections to the IP addresses.

   • Connect to the client VM:

   gcloud compute ssh vm-client --zone=us-west1-a
   

   • Test HTTP connectivity to the IP addresses:

   curl http://10.1.2.99
   curl http://10.5.6.99
   

   • Test HTTPS connectivity. Use of --insecure is required because the Apache server configuration in the example setup uses self-signed certificates.

   curl https://10.1.2.99 --insecure
   curl https://10.5.6.99 --insecure
   

   • Observe that requests are handled by all of the backend VMs, regardless of the protocol (HTTP or HTTPS) or IP address used.

Using subsetting

Preview

This product or feature is covered by the Pre-GA Offerings Terms of the Google Cloud Platform Terms of Service. Pre-GA products and features may have limited support, and changes to pre-GA products and features may not be compatible with other pre-GA versions. For more information, see the launch stage descriptions.

The example configuration creates a backend service without subsetting.

This procedure shows you how to enable subsetting on the backend service for the example internal TCP/UDP load balancer so that the deployment can scale to a larger number of backend instances.

Caution: Enabling subsetting might be temporarily disruptive and might break existing TCP connections.

Subsetting is an advanced feature that is supported for instance groups only during the Preview release.

You should only enable subsetting if you need to support more than 250 backend VMs on a single load balancer.

For more information about this use case, see Subsetting.

gcloud

Use the following gcloud command to update the be-ilb backend service, specifying subsetting:

gcloud beta compute backend-services update be-ilb \
    --subsetting-policy=CONSISTENT_HASH_SUBSETTING

api

Make a PATCH request to the regionBackendServices/patch method.

PATCH https://compute.googleapis.com/compute/beta/projects/PROJECT_ID/regions/us-west1/backendServices/be-ilb

{
  "subsetting":
    {
      "policy": CONSISTENT_HASH_SUBSETTING
    }
}

What's next

• See Internal TCP/UDP Load Balancing Concepts for important fundamentals.
• See Failover concepts for Internal TCP/UDP Load Balancing for important information about failover.
• See Internal load balancing and DNS names for available DNS name options your load balancer can use.
• See Configuring failover for Internal TCP/UDP Load Balancing for configuration steps and an example internal TCP/UDP load balancer failover configuration.
• See Internal TCP/UDP Load Balancing Logging and Monitoring for information on configuring Logging and Monitoring for Internal TCP/UDP Load Balancing.
• See Internal TCP/UDP Load Balancing and Connected Networks for information about accessing internal TCP/UDP load balancers from peer networks connected to your VPC network.
• See Troubleshooting Internal TCP/UDP Load Balancing for information on how to troubleshoot issues with your internal TCP/UDP load balancer.
• Clean up the load balancer setup.