Setting Up Internal TCP/UDP Load Balancing

This guide uses an example to teach the fundamentals of Google Cloud Platform Internal TCP/UDP Load Balancing. Before following this guide, familiarize yourself with the following:

• Internal TCP/UDP Load Balancing concepts
• Firewall rules overview
• Health check concepts

Permissions

To follow this guide, you need to create instances and modify a network in a project. You should be either a project owner or editor, or you should have all of the following Compute Engine IAM roles:

Task	Required Role
Create networks, subnets, and load balancer components	Network Admin
Add and remove firewall rules	Security Admin
Create instances	Instance Admin

Setup

This guide shows you how to configure and test an internal TCP/UDP load balancer. The steps in this section describe how to configure the following:

1. A sample VPC network with custom subnets
2. Firewall rules that allow incoming connections to backend VMs
3. Four backend VMs:
   • Two VMs in an unmanaged instance group in zone us-west1-a
   • Two VMs in an unmanaged instance group in zone us-west1-c
4. One client VM to test connections
5. The following internal TCP/UDP load balancer components:
   • A health check for the backend service
   • An internal backend service in the us-west1 region to manage connection distribution to the two zonal instance groups
   • An internal forwarding rule and internal IP address for the frontend of the load balancer

The architecture for this example looks like this:

Internal TCP/UDP Load Balancing Example Configuration (click to enlarge)

Note: For simplicity, this example uses unmanaged instance groups. In production, consider using a managed instance group, as illustrated in Managed instance groups.

Configuring a network, region, and subnet

You need a VPC network with at least one subnet. An internal TCP/UDP load balancer is regional. Traffic within the VPC network is routed to the load balancer if its source is from a subnet in the same region as the load balancer.

This example uses the following VPC network, region, and subnet:

• Network: The network is a custom mode VPC network named lb-network.

• Region: The region is us-west1.

• Subnet: The subnet, lb-subnet, uses the 10.1.2.0/24 IP range.

Note: You can change the name of the network, the region, and the parameters for the subnet; however, subsequent steps in this guide use the network, region, and subnet parameters as outlined above.

To create the example network and subnet, follow these steps.

Console

1. Go to the VPC networks page in the Google Cloud Platform Console.
   Go to the VPC network page
2. Click Create VPC network.
3. Enter a Name of lb-network.
4. In the Subnets section:
   • Set the Subnet creation mode to Custom.
   • In the New subnet section, enter the following information:
     • Name: lb-subnet
     • Region: us-west1
     • IP address range: 10.1.2.0/24
     • Click Done.
5. Click Create.

gcloud

1. Create the custom VPC network:

   gcloud compute networks create lb-network --subnet-mode=custom
   

2. Create a subnet in the lb-network network in the us-west1 region:

   gcloud compute networks subnets create lb-subnet \
       --network=lb-network \
       --range=10.1.2.0/24 \
       --region=us-west1
   

api

Make a POST request to the networks.insert method.

POST https://www.googleapis.com/compute/v1/projects/[PROJECT_ID]/global/networks
{
  "routingConfig": {
    "routingMode": "REGIONAL"
  },
  "name": "lb-network",
  "autoCreateSubnetworks": false
}

Make a POST request to the subnetworks.insert method.

POST https://www.googleapis.com/compute/v1/projects/[PROJECT_ID]/regions/us-west1/subnetworks
{
  "name": "lb-subnet",
  "network": "https://www.googleapis.com/compute/v1/projects/[PROJECT_ID]/global/networks/lb-network",
  "ipCidrRange": "10.1.2.0/24",
  "privateIpGoogleAccess": false
}

Configuring firewall rules

This example uses the following firewall rules:

• fw-allow-lb-subnet: An ingress rule, applicable to all targets in the VPC network, allowing traffic from sources in the 10.1.2.0/24 range. This rules allows incoming traffic from any source within the lb-subnet to the instances (VMs) being load balanced.

• fw-allow-ssh: An ingress rule, applicable to the instances being load balanced, that allows incoming SSH connectivity on TCP port 22 from any address. You can choose a more restrictive source IP range for this rule; for example, you can specify just the IP ranges of the system from which you will initiating SSH sessions. This example uses the target tag allow-ssh to identify the VMs to which it should apply.

• fw-allow-health-check: An ingress rule, applicable to the instances being load balanced, that allows traffic from the GCP health checking systems (130.211.0.0/22 and 35.191.0.0/16). This example uses the target tag allow-health-check to identify the instances to which it should apply.

Without these firewall rules, the default deny ingress rule blocks incoming traffic to the backend instances.

Note: You must create a firewall rule to allow health checks from the IP ranges of GCP probe systems. See probe IP ranges for more information.

Console

1. Go to the Firewall rules page in the Google Cloud Platform Console.
   Go to the Firewall rules page
2. Click Create firewall rule and enter the following information to create the rule to allow subnet traffic:
   • Name: fw-allow-lb-subnet
   • Network: lb-network
   • Priority: 1000
   • Direction of traffic: ingress
   • Action on match: allow
   • Targets: All instances in the network
   • Source filter: IP ranges
   • Source IP ranges: 10.1.2.0/24
   • Protocols and ports: Allow all
3. Click Create.
4. Click Create firewall rule again to create the rule to allow incoming SSH connections:
   • Name: fw-allow-ssh
   • Network: lb-network
   • Priority: 1000
   • Direction of traffic: ingress
   • Action on match: allow
   • Targets: Specified target tags
   • Target tags: allow-ssh
   • Source filter: IP ranges
   • Source IP ranges: 0.0.0.0/0
   • Protocols and ports: Choose Specified protocols and ports then type: tcp:22
5. Click Create.
6. Click Create firewall rule a third time to create the rule to allow GCP health checks:
   • Name: fw-allow-health-check
   • Network: lb-network
   • Priority: 1000
   • Direction of traffic: ingress
   • Action on match: allow
   • Targets: Specified target tags
   • Target tags: allow-health-check
   • Source filter: IP ranges
   • Source IP ranges: 130.211.0.0/22 and 35.191.0.0/16
   • Protocols and ports: Allow all
7. Click Create.

gcloud

1. Create the fw-allow-lb-subnet firewall rule to allow communication from with the subnet:

   gcloud compute firewall-rules create fw-allow-lb-subnet \
       --network=lb-network \
       --action=allow \
       --direction=ingress \
       --source-ranges=10.1.2.0/24 \
       --rules=tcp,udp,icmp
   

2. Create the fw-allow-ssh firewall rule to allow SSH connectivity to VMs with the network tag allow-ssh. When you omit source-ranges, GCP interprets the rule to mean any source.

   gcloud compute firewall-rules create fw-allow-ssh \
       --network=lb-network \
       --action=allow \
       --direction=ingress \
       --target-tags=allow-ssh \
       --rules=tcp:22
   

3. Create the fw-allow-health-check rule to allow GCP health checks.

   gcloud compute firewall-rules create fw-allow-health-check \
       --network=lb-network \
       --action=allow \
       --direction=ingress \
       --target-tags=allow-health-check \
       --source-ranges=130.211.0.0/22,35.191.0.0/16 \
       --rules=tcp,udp,icmp
   

api

Create the fw-allow-lb-subnet firewall rule by making a POST request to the firewalls.insert method.

POST https://www.googleapis.com/compute/v1/projects/[PROJECT_ID]/global/firewalls
{
  "name": "fw-allow-lb-subnet",
  "network": "https://www.googleapis.com/compute/v1/projects/[PROJECT_ID]/global/networks/lb-network",
  "priority": 1000,
  "sourceRanges": [
    "10.1.2.0/24"
  ],
  "allowed": [
    {
      "IPProtocol": "tcp"
    },
    {
      "IPProtocol": "udp"
    },
    {
      "IPProtocol": "icmp"
    }
  ],
  "direction": "INGRESS",
  "logConfig": {
    "enable": false
  },
  "disabled": false
}

Create the fw-allow-ssh firewall rule by making a POST request to the firewalls.insert method.

POST https://www.googleapis.com/compute/v1/projects/[PROJECT_ID]/global/firewalls
{
  "name": "fw-allow-ssh",
  "network": "https://www.googleapis.com/compute/v1/projects/[PROJECT_ID]/global/networks/lb-network",
  "priority": 1000,
  "sourceRanges": [
    "0.0.0.0/0"
  ],
  "targetTags": [
    "allow-ssh"
  ],
  "allowed": [
   {
     "IPProtocol": "tcp",
     "ports": [
       "22"
     ]
   }
  ],
 "direction": "INGRESS",
 "logConfig": {
   "enable": false
 },
 "disabled": false
}

Create the fw-allow-health-check firewall rule by making a POST request to the firewalls.insert method.

POST https://www.googleapis.com/compute/v1/projects/[PROJECT_ID]/global/firewalls
{
  "name": "fw-allow-health-check",
  "network": "https://www.googleapis.com/compute/v1/projects/[PROJECT_ID]/global/networks/lb-network",
  "priority": 1000,
  "sourceRanges": [
    "130.211.0.0/22",
    "35.191.0.0/16"
  ],
  "targetTags": [
    "allow-health-check"
  ],
  "allowed": [
    {
      "IPProtocol": "tcp"
    },
    {
      "IPProtocol": "udp"
    },
    {
      "IPProtocol": "icmp"
    }
  ],
  "direction": "INGRESS",
  "logConfig": {
    "enable": false
  },
  "disabled": false
}

Creating backend VMs and instance groups

This example uses two unmanaged instance groups each having two backend (server) VMs. To demonstrate the regional nature of Internal TCP/UDP Load Balancing, the two instance groups are placed in separate zones, us-west1-a and us-west1-c.

• Instance group ig-a contains these two VMs:
  • vm-a1
  • vm-a2
• Instance group ig-c contains these two VMs:
  • vm-c1
  • vm-c2

Traffic to all four of the backend VMs is load balanced. Each of the four run an Apache web server on TCP ports 80 and 443. Each is assigned an internal IP address in the lb-subnet and an ephemeral external (public) IP address. You can remove the external IP addresses later.

External IP address for the backend VMs are not required; however, they are useful for this example because they permit the backend VMs to download Apache from the Internet, and they make it easy to connect via SSH.

By default, Apache is configured to bind to any IP address. Internal TCP/UDP load balancers deliver packets by preserving the destination IP. Ensure that server software running on your backend VMs is listening on the IP address of the load balancer's internal forwarding rule. If you configure multiple internal forwarding rules, ensure that your software listens to the internal IP address associated with each one. The destination IP address of a packet delivered to a backend VM by an internal TCP/UDP load balancer is the internal IP address of the forwarding rule.

For instructional simplicity, these backend VMs run Debian GNU/Linux 9.

Console

Create backend VMs

1. Go to the VM instances page in the Google Cloud Platform Console.
   Go to the VM instances page
2. Repeat the following steps to create four VMs, using the following name and zone combinations.
   • Name: vm-a1, zone: us-west1-a
   • Name: vm-a2, zone: us-west1-a
   • Name: vm-c1, zone: us-west1-c
   • Name: vm-c2, zone: us-west1-c
3. Click Create instance.
4. Set the Name as indicated in step 2.
5. For the Region, choose us-west1, and choose a Zone as indicated in step 2.
6. In the Boot disk section, ensure that the selected image is Debian GNU/Linux 9 Stretch. Click Choose to change the image if necessary.

7. Click Management, security, disks, networking, sole tenancy and make the following changes:

   • Click Networking and add the following Network tags: allow-ssh and allow-health-check
   • Click the edit button under Network interfaces and make the following changes then click Done:
     • Network: lb-network
     • Subnet: lb-subnet
     • Primary internal IP: Ephemeral (automatic)
     • External IP: Ephemeral

   • Click Management. In the Startup script field, copy and paste the following script contents. The script contents are identical for all four VMs:

     #! /bin/bash
     apt-get update
     apt-get install apache2 -y
     a2ensite default-ssl
     a2enmod ssl
     vm_hostname="$(curl -H "Metadata-Flavor:Google" \
     http://169.254.169.254/computeMetadata/v1/instance/name)"
     echo "Page served from: $vm_hostname" | \
     tee /var/www/html/index.html
     systemctl restart apache2
     

8. Click Create.

Create instance groups

1. Go to the Instance groups page in the Google Cloud Platform Console.
   Go to the Instance groups page
2. Repeat the following steps to create two unmanaged instance groups each with two VMs in them, using these combinations.
   • Instance group: ig-a, zone: us-west1-a, VMs: vm-a1 and vm-a2
   • Instance group: ig-c, zone: us-west1-c, VMs: vm-c1 and vm-c2
3. Click Create instance group.
4. Set Name as indicated in step 2.
5. In the Location section, select Single-zone, choose us-west1 for the Region, and then choose a Zone as indicated in step 2.
6. In the Group type section, select Unmanaged instance group.
7. For Network, enter lb-network.
8. For Subnetwork, enter lb-subnet.
9. In the VM instances section, add the VMs as indicated in step 2.
10. Click Create.

gcloud

1. Create the four VMs by running the following command four times, using these four combinations for [VM-NAME] and [ZONE]. The script contents are identical for all four VMs.

   • [VM-NAME] of vm-a1 and [ZONE] of us-west1-a
   • [VM-NAME] of vm-a2 and [ZONE] of us-west1-a
   • [VM-NAME] of vm-c1 and [ZONE] of us-west1-c
   • [VM-NAME] of vm-c2 and [ZONE] of us-west1-c

   gcloud compute instances create [VM-NAME] \
       --zone=[ZONE] \
       --image-family=debian-9 \
       --image-project=debian-cloud \
       --tags=allow-ssh,allow-health-check \
       --subnet=lb-subnet \
       --metadata=startup-script='#! /bin/bash
   apt-get update
   apt-get install apache2 -y
   a2ensite default-ssl
   a2enmod ssl
   vm_hostname="$(curl -H "Metadata-Flavor:Google" \
   http://169.254.169.254/computeMetadata/v1/instance/name)"
   echo "Page served from: $vm_hostname" | \
   tee /var/www/html/index.html
   systemctl restart apache2'
   

2. Create the two unmanaged instance groups in each zone:

   gcloud compute instance-groups unmanaged create ig-a \
       --zone=us-west1-a
   gcloud compute instance-groups unmanaged create ig-c \
       --zone=us-west1-c
   

3. Add the VMs to the appropriate instance groups:

   gcloud compute instance-groups unmanaged add-instances ig-a \
       --zone=us-west1-a \
       --instances=vm-a1,vm-a2
   gcloud compute instance-groups unmanaged add-instances ig-c \
       --zone=us-west1-c \
       --instances=vm-c1,vm-c2
   

api

For the four VMs, use the following VM names and zones:

• [VM-NAME] of vm-a1 and [ZONE] of us-west1-a
• [VM-NAME] of vm-a2 and [ZONE] of us-west1-a
• [VM-NAME] of vm-c1 and [ZONE] of us-west1-c
• [VM-NAME] of vm-c2 and [ZONE] of us-west1-c

Create four backend VMs by making four POST requests to the instances.insert method:

POST https://www.googleapis.com/compute/v1/projects/[PROJECT_ID]/zones/[ZONE]/instances
{
  "name": "[VM-NAME]",
  "tags": {
    "items": [
      "allow-health-check",
      "allow-ssh"
    ]
  },
  "machineType": "https://www.googleapis.com/compute/v1/projects/[PROJECT_ID]/zones/[ZONE]/machineTypes/n1-standard-1",
  "canIpForward": false,
  "networkInterfaces": [
    {
      "network": "https://www.googleapis.com/compute/v1/projects/[PROJECT_ID]/global/networks/lb-network",
      "subnetwork": "https://www.googleapis.com/compute/v1/projects/[PROJECT_ID]/regions/us-west1/subnetworks/lb-subnet",
      "accessConfigs": [
        {
          "type": "ONE_TO_ONE_NAT",
          "name": "external-nat",
          "networkTier": "PREMIUM"
        }
      ]
    }
  ],
  "disks": [
    {
      "type": "PERSISTENT",
      "boot": true,
      "mode": "READ_WRITE",
      "autoDelete": true,
      "deviceName": "[VM-NAME]",
      "initializeParams": {
        "sourceImage": "projects/debian-cloud/global/images/debian-9-stretch-v20190618",
        "diskType": "projects/[PROJECT_ID]/zones/us-west1-a/diskTypes/pd-standard",
        "diskSizeGb": "10"
      }
    }
  ]
  "metadata": {
    "items": [
      {
        "key": "startup-script",
        "value": "#! /bin/bash\napt-get update\napt-get install apache2 -y\na2ensite default-ssl\na2enmod ssl\nvm_hostname=\"$(curl -H \"Metadata-Flavor:Google\" \\\nhttp://169.254.169.254/computeMetadata/v1/instance/name)\"\necho \"Page served from: $vm_hostname\" | \\\ntee /var/www/html/index.html\nsystemctl restart apache2"
      }
    ]
  },
  "scheduling": {
    "preemptible": false
  },
  "deletionProtection": false
}

Create two instance groups by making a POST request to the instanceGroups.insert method.

POST https://www.googleapis.com/compute/v1/projects/[PROJECT_ID]/zones/us-west1-a/instanceGroups
{
  "name": "ig-a",
  "network": "https://www.googleapis.com/compute/v1/projects/[PROJECT_ID]/global/networks/lb-network",
  "subnetwork": "https://www.googleapis.com/compute/v1/projects/[PROJECT_ID]/regions/us-west1/subnetworks/lb-subnet"
}

POST https://www.googleapis.com/compute/v1/projects/[PROJECT_ID]/zones/us-west1-c/instanceGroups
{
  "name": "ig-c",
  "network": "https://www.googleapis.com/compute/v1/projects/[PROJECT_ID]/global/networks/lb-network",
  "subnetwork": "https://www.googleapis.com/compute/v1/projects/[PROJECT_ID]/regions/us-west1/subnetworks/lb-subnet"
}

Add instances to each instance group by making a POST request to the instanceGroups.addInstances method.

POST https://www.googleapis.com/compute/v1/projects/[PROJECT_ID]/zones/us-west1-a/instanceGroups/ig-a/addInstances
{
  "instances": [
    {
      "instance": "https://www.googleapis.com/compute/v1/projects/[PROJECT_ID]/zones/us-west1-a/instances/vm-a1",
      "instance": "https://www.googleapis.com/compute/v1/projects/[PROJECT_ID]/zones/us-west1-a/instances/vm-a2"
    }
  ]
}

POST https://www.googleapis.com/compute/v1/projects/[PROJECT_ID]/zones/us-west1-c/instanceGroups/ig-c/addInstances
{
  "instances": [
    {
      "instance": "https://www.googleapis.com/compute/v1/projects/[PROJECT_ID]/zones/us-west1-c/instances/vm-c1",
      "instance": "https://www.googleapis.com/compute/v1/projects/[PROJECT_ID]/zones/us-west1-c/instances/vm-c2"
    }
  ]
}

Creating a client VM

This example creates a client VM (vm-client) in the same region as the backend (server) VMs. The client is used to validate the load balancer's configuration and demonstrate expected behavior as described in the testing section.

Console

1. Go to the VM instances page in the Google Cloud Platform Console.
   Go to the VM instances page
2. Click Create instance.
3. Set the Name to vm-client.
4. Set the Zone to us-west1-a.
5. Click Management, security, disks, networking, sole tenancy and make the following changes:
   • Click Networking and add the allow-ssh to Network tags.
   • Click the edit button under Network interfaces and make the following changes then click Done:
     • Network: lb-network
     • Subnet: lb-subnet
     • Primary internal IP: Ephemeral (automatic)
     • External IP: Ephemeral
6. Click Create.

gcloud

The client VM can be in any zone in the same region as the load balancer, and it can use any subnet in that region. In this example, the client is in the us-west1-a zone, and it uses the same subnet as the backend VMs.

gcloud compute instances create vm-client \
    --zone=us-west1-a \
    --image-family=debian-9 \
    --image-project=debian-cloud \
    --tags=allow-ssh \
    --subnet=lb-subnet

api

Make a POST request to the instances.insert method.

POST https://www.googleapis.com/compute/v1/projects/[PROJECT_ID]/zones/us-west1-a/instances
{
  "name": "vm-client",
  "tags": {
    "items": [
      "allow-ssh"
    ]
  },
  "machineType": "https://www.googleapis.com/compute/v1/projects/[PROJECT_ID]/zones/us-west1-a/machineTypes/n1-standard-1",
  "canIpForward": false,
  "networkInterfaces": [
    {
      "network": "https://www.googleapis.com/compute/v1/projects/[PROJECT_ID]/global/networks/lb-network",
      "subnetwork": "https://www.googleapis.com/compute/v1/projects/[PROJECT_ID]/regions/us-west1/subnetworks/lb-subnet",
      "accessConfigs": [
        {
          "type": "ONE_TO_ONE_NAT",
          "name": "external-nat",
          "networkTier": "PREMIUM"
        }
      ]
    }
  ],
  "disks": [
    {
      "type": "PERSISTENT",
      "boot": true,
      "mode": "READ_WRITE",
      "autoDelete": true,
      "deviceName": "[VM-NAME]",
      "initializeParams": {
        "sourceImage": "projects/debian-cloud/global/images/debian-9-stretch-v20190618",
        "diskType": "projects/[PROJECT_ID]/zones/us-west1-a/diskTypes/pd-standard",
        "diskSizeGb": "10"
      }
    }
  ]
  "scheduling": {
    "preemptible": false
  },
  "deletionProtection": false
}

Configuring load balancer components

These steps configure all of the internal TCP/UDP load balancer components starting with the health check and backend service, and then the frontend components:

• Health check: In this example, we use an HTTP health check that simply checks for an HTTP 200 (OK) response. For more information, see the health checks section of the Internal TCP/UDP Load Balancing overview.

• Backend service: Because we need to pass HTTP traffic through the internal load balancer, we need to use TCP, not UDP.

• Forwarding rule: This example creates a single internal forwarding rule.

• Internal IP address: In this example, we specify an internal IP address, 10.1.2.99, when we create the forwarding rule.

Console

Create the load balancer and configure a backend service

1. Go to the Load balancing page in the Google Cloud Platform Console.
   Go to the Load balancing page
2. Click Create load balancer.
3. Under TCP load balancing, click Start configuration.
4. Under Internet facing or internal only select Only between my VMs.
5. Click Continue.
6. Set the Name to be-ilb.
7. Click Backend configuration and make the following changes:
   1. Region: us-west1
   2. Network: lb-network
   3. Under Backends, in the New item section, select the ig-a instance group and click Done.
   4. Click Add backend. In the New item section that appears, select the ig-c instance group and click Done again.
   5. Under Health check, choose Create another health check, enter the following information, and click Save and continue:
      • Name: hc-http-80
      • Protocol: HTTP
      • Port: 80
      • Proxy protocol: NONE
      • Request path: /
   6. Verify that there is a blue check mark next to Backend configuration before continuing. Review this step if not.
8. Click Frontend configuration. In the New Frontend IP and port section, make the following changes:
   1. Name: fr-ilb
   2. Subnetwork: ilb-subnet
   3. From Internal IP, choose Reserve a static internal IP address, enter the following information, and click Reserve:
      • Name: ip-ilb
      • Static IP address: Let me choose
      • Custom IP address: 10.1.2.99
   4. Ports: Choose Single, and enter 80 for the Port number.
   5. Verify that there is a blue check mark next to Frontend configuration before continuing. Review this step if not.
9. Click Review and finalize. Double-check your settings.
10. Click Create.

gcloud

1. Create a new HTTP health check to test TCP connectivity to the VMs on 80.

   gcloud compute health-checks create http hc-http-80 \
       --port=80
   

2. Create the backend service for HTTP traffic:

   gcloud compute backend-services create be-ilb \
       --load-balancing-scheme=internal \
       --protocol=tcp \
       --region=us-west1 \
       --health-checks=hc-http-80
   

3. Add the two instance groups to the backend service:

   gcloud compute backend-services add-backend be-ilb \
       --region=us-west1 \
       --instance-group=ig-a \
       --instance-group-zone=us-west1-a
   gcloud compute backend-services add-backend be-ilb \
       --region=us-west1 \
       --instance-group=ig-c \
       --instance-group-zone=us-west1-c
   

4. Create a forwarding rule for the backend service. When you create the forwarding rule, specify 10.1.2.99 for the internal IP in the subnet.

   gcloud compute forwarding-rules create fr-ilb \
       --region=us-west1 \
       --load-balancing-scheme=internal \
       --network=lb-network \
       --subnet=lb-subnet \
       --address=10.1.2.99 \
       --ip-protocol=TCP \
       --ports=80 \
       --backend-service=be-ilb \
       --backend-service-region=us-west1
   

api

Create the health check by making a POST request to the healthChecks.insert method.

POST https://www.googleapis.com/compute/v1/projects/[PROJECT_ID]/global/healthChecks
{
  "name": "hc-http-80",
  "type": "HTTP",
  "httpHealthCheck": {
    "port": 80
  }
}

Create the regional backend service by making a POST request to the regionBackendServices.insert method.

POST https://www.googleapis.com/compute/v1/projects/[PROJECT_ID]/regions/us-west1/backendServices
{
  "name": "be-ilb",
  "backends": [
    {
      "group": "https://www.googleapis.com/compute/v1/projects/[PROJECT_ID]/zones/us-west1-a/instanceGroups/ig-a",
      "balancingMode": "CONNECTION"
    }
    {
      "group": "https://www.googleapis.com/compute/v1/projects/[PROJECT_ID]/zones/us-west1-c/instanceGroups/ig-c",
      "balancingMode": "CONNECTION"
    }
  ],
  "healthChecks": [
    "https://www.googleapis.com/compute/v1/projects/[PROJECT_ID]/global/healthChecks/hc-http-80"
  ],
  "loadBalancingScheme": "INTERNAL",
  "connectionDraining": {
    "drainingTimeoutSec": 0
  }
}

Create the forwarding rule by making a POST request to the forwardingRules.insert method.

POST https://www.googleapis.com/compute/v1/projects/[PROJECT_ID]/regions/us-west1/forwardingRules
{
  "name": "fr-ilb",
  "IPAddress": "10.1.2.99",
  "IPProtocol": "TCP",
  "ports": [
    "80"
  ],
  "loadBalancingScheme": "INTERNAL",
  "subnetwork": "https://www.googleapis.com/compute/v1/projects/[PROJECT_ID]/regions/us-west1/subnetworks/lb-subnet",
  "network": "https://www.googleapis.com/compute/v1/projects/[PROJECT_ID]/global/networks/lb-network",
  "backendService": "https://www.googleapis.com/compute/v1/projects/[PROJECT_ID]/regions/us-west1/backendServices/be-ilb",
  "networkTier": "PREMIUM"
}

Testing

These tests show how to validate your load balancer configuration and learn about its expected behavior.

Testing load balancing

This test contacts the load balancer from a separate client VM; that is, not from a backend VM of the load balancer. The expected behavior is for traffic to be distributed among the four backend VMs because no session affinity has been configured.

1. Connect to the client VM instance.

   gcloud compute ssh vm-client --zone=us-west1-a
   

2. Make a web request to the load balancer using curl to contact its IP address. Repeat the request so you can see that responses come from different backend VMs. The name of the VM generating the response is displayed in the text in the HTML response, by virtue of the contents of /var/www/html/index.html on each backend VM. Expected responses look like: Page served from: vm-a1, Page served from: vm-a2, and so on.

   curl http://10.1.2.99
   

   • If you add a service label to the internal forwarding rule, you can use internal DNS to contact the load balancer using its service name.

     curl http://web-test.fr-ilb.il4.us-west1.lb.[PROJECT_ID].internal
     

Pinging the load balancer's IP address

This test demonstrates an expected behavior: You cannot ping the IP address of the load balancer. This is because internal TCP/UDP load balancers are implemented in virtual network programming — they are not separate devices.

1. Connect to the client VM instance.

   gcloud compute ssh vm-client --zone=us-west1-a
   

2. Attempt to ping the IP address of the load balancer. Notice that you do not get a response and that the ping command times out after 10 seconds in this example.

   timeout 10 ping 10.1.2.99
   

Sending requests from load balanced VMs

This test demonstrates that requests to the load balancer that originate from any of the backend VMs — the server VMs being load balanced — are always answered by the same VM making the request.

Internal TCP/UDP Load Balancing is implemented using virtual network programming and VM configuration in the guest OS. On Linux VMs, the Linux Guest Environment performs the local configuration by installing a route in the guest OS routing table. Because of this local route, traffic to the IP address of the load balancer stays on the load balanced VM itself. (This local route is different from the routes in the VPC network.)

Caution: Don't rely on making requests to an internal TCP/UDP load balancer from a VM being load balanced (in the backend service for that load balancer). A request is always sent to the VM that makes the request, and health check information is ignored. Further, the backend can respond to traffic sent using protocols and destination ports other than those configured on the load balancer's internal forwarding rule.

1. Connect to a backend VM, such as vm-a1:

   gcloud compute ssh vm-a1 --zone=us-west1-a
   

   Important: You won't be able to connect to a backend VM in this way if you have removed its external IP address. See connecting to instances that do not have external IP addresses for connection options if your backend VMs do not have external IP addresses.

2. Make a web request to the load balancer (by IP address or service name) using curl. Repeat the request, and note that the response is sent from the backend VM that makes the request. The expected response when testing from vm-a1 is always: Page served from: vm-a1.

   curl http://10.1.2.99
   

3. Inspect the local routing table, looking for a destination that matches the IP address of the load balancer itself, 10.1.2.99. This route is a necessary part of Internal TCP/UDP Load Balancing, but it also demonstrates why a request from a VM behind the load balancer is always responded to by the same VM.

   ip route show table local | grep 10.1.2.99
   

Additional configuration options

This section expands on the configuration example providing alternative and additional configuration options. All of the tasks are optional. You can perform them in any order.

Configuring managed instance groups

The example configuration created two unmanaged instance groups. You can instead use managed instance groups, including zonal and regional managed instance groups, as backends for Internal TCP/UDP Load Balancing.

Managed instance groups require that you create an instance template. This procedure demonstrates how to replace the two unmanaged zonal instance groups from the example with a single, regional managed instance group. A regional managed instance group automatically creates VMs in multiple zones of the region, making it simpler to distribute production traffic among zones.

Managed instance groups also support autoscaling and autohealing. If you use autoscaling with Internal TCP/UDP Load Balancing, you cannot scale based on load balancing.

This procedure shows you how to modify the backend service for the example internal TCP/UDP load balancer so that it uses a regional managed instance group.

Console

Instance template

1. Go to the VM instance templates page in the Google Cloud Platform Console.
   Go to the VM instance templates page
2. Click Create instance template.
3. Set the Name to template-vm-ilb.
4. Choose a machine type.
5. For Boot disk, click Change, choose Debian GNU/Linux 9 Stretch, then click Select.
6. Click Management, security, disks, networking, sole tenancy.
   • Click Networking and make the following changes:
     • Network: lb-network
     • Subnet: lb-subnet
     • Network tags: allow-ssh and allow-health-check
   • Click Management. In the Startup script field, copy and paste the following script contents:

     #! /bin/bash
     apt-get update
     apt-get install apache2 -y
     a2ensite default-ssl
     a2enmod ssl
     vm_hostname="$(curl -H "Metadata-Flavor:Google" 
     
     http://169.254.169.254/computeMetadata/v1/instance/name)"
     echo "Page served from: $vm_hostname" | 
     
     tee /var/www/html/index.html
     systemctl restart apache2
     

7. Click Create.

Managed instance group

1. Go to the VM instances page in the Google Cloud Platform Console.
   Go to the VM instance groups page
2. Click Create instance group.
3. Set the Name to ig-ilb.
4. For Location, choose Multi-zone, and set the Region to us-west1.
5. Set the Instance template to template-vm-ilb.
6. (Optional) Configure autoscaling. You cannot autoscale the instance group based on HTTP load balancing usage because the instance group is a backend for Internal TCP/UDP Load Balancing.
7. Set the Minimum number of instances to 1 and the Maximum number of instances to 6.
8. (Optional) Configure autohealing. If you configure autohealing, use the same health check used by the backend service for the Internal TCP/UDP Load Balancer. In this example, use hc-http-80.
9. Click Create.

gcloud

1. Create the instance template. Optionally, you can set other parameters, such as machine type, for the image template to use.

   gcloud compute instance-templates create template-vm-ilb \
       --image-family=debian-9 \
       --image-project=debian-cloud \
       --tags=allow-ssh,allow-health-check \
       --subnet=lb-subnet \
       --region=us-west1 \
       --network=lb-network \
       --metadata=startup-script='#! /bin/bash
   apt-get update
   apt-get install apache2 -y
   a2ensite default-ssl
   a2enmod ssl
   vm_hostname="$(curl -H "Metadata-Flavor:Google" \
   http://169.254.169.254/computeMetadata/v1/instance/name)"
   echo "Page served from: $vm_hostname" | \
   tee /var/www/html/index.html
   systemctl restart apache2'
   

2. Create one regional managed instance group using the template:

   gcloud compute instance-groups managed create ig-ilb \
       --template=template-vm-ilb \
       --region=us-west1 \
       --size=6
   

3. Add the regional managed instance group as a backend to the backend service you already created:

   gcloud compute backend-services add-backend be-ilb \
       --region=us-west1 \
       --instance-group=ig-ilb \
       --instance-group-region=us-west1
   

4. Disconnect the two unmanaged (zonal) instance groups from the backend service:

   gcloud compute backend-services remove-backend be-ilb \
       --region=us-west1 \
       --instance-group=ig-a \
       --instance-group-zone=us-west1-a
   gcloud compute backend-services remove-backend be-ilb \
       --region=us-west1 \
       --instance-group=ig-c \
       --instance-group-zone=us-west1-c
   

Removing external IP addresses from backend VMs

When you created the backend VMs, each was assigned an ephemeral external IP address so it could download Apache via a startup script. Because the backend VMs are only used by an internal load balancer, you can remove their external IP addresses. Removing external IP addresses prevents the backend VMs from accessing the Internet directly.

Important: Removing the external IP address from a VM limits how you can connect to it. See connecting to instances that do not have external IP addresses for details. It's useful to leave external IP addresses on backend VMs if you need to demonstrate how requests from a backend VM to the IP address of the load balancer are handled.

Console

1. Go to the VM instances page in the Google Cloud Platform Console.
   Go to the VM instances page
2. Repeat the following steps for each backend VM.
3. Click the name of the backend VM (for example, vm-a1) to view the VM instance details page.
4. Click Edit.
5. In the Network interfaces section, click the Edit button.
6. From the External IP pop-up, choose None, and click Done.
7. Click Save.

gcloud

1. To look up the zone for an instance – for example, if you're using a regional managed instance group – run the following command for each instance to determine its zone. Replace [SERVER-VM] with the name of the VM to look up.

   gcloud compute instances list --filter="name=[SERVER-VM]"
   

2. Repeat the following step for each backend VM. Replace [SERVER-VM] with the name of the VM, and replace and [ZONE] with the VM's zone.

   gcloud compute instances delete-access-config [SERVER-VM] \
       --zone=[ZONE] \
       --access-config-name=external-nat
   

api

Make a POST request to the instances.deleteAccessConfig method for each backend VM, replacingvm-a1 with the name of the VM, and replacing and us-west1-a with the VM's zone.

POST https://www.googleapis.com/compute/v1/projects/[PROJECT_ID]/zones/us-west1-a/instances/v1-a1/deleteAccessConfig?accessConfig=external-nat&networkInterface=None

Accepting traffic on multiple ports

The component that controls the port for incoming traffic is the internal forwarding rule. This procedure shows you how to replace the forwarding rule for port 80 with one that accepts traffic on ports 80 and 443. When you create the backend VMs, the startup script that installs and configures Apache also configures it to serve HTTPS traffic on port 443. You must replace a forwarding rule because GCP does not support updating forwarding rules.

Note: An internal backend service does not specify a port. For more information, see Components.

gcloud

1. Delete your existing forwarding rule, fr-ilb.

   gcloud compute forwarding-rules delete fr-ilb \
       --region=us-west1
   

2. Create a replacement forwarding rule with the same name for ports 80 and 443. The other parameters for this rule, including IP address and backend service, are the same.

   gcloud compute forwarding-rules create fr-ilb \
       --region=us-west1 \
       --load-balancing-scheme=internal \
       --network=lb-network \
       --subnet=lb-subnet \
       --address=10.1.2.99 \
       --ip-protocol=TCP \
       --ports=80,443 \
       --backend-service=be-ilb \
       --backend-service-region=us-west1
   

3. Connect to the client VM instance and test HTTP and HTTPS connections.

   • Connect to the client VM:

     gcloud compute ssh vm-client --zone=us-west1-a
     

   • Test HTTP connectivity:

     curl http://10.1.2.99
     

   • Test HTTPS connectivity. The --insecure flag is required because the Apache server configuration in the example setup uses self-signed certificates.

     curl https://10.1.2.99 --insecure
     

   • Observe that HTTP requests (on port 80) and HTTPS requests (on port 443) are handled by all of the backend VMs.

Using session affinity

The example configuration creates a backend service without session affinity.

This procedure shows you how to update the backend service for the example internal TCP/UDP load balancer so that it uses session affinity based on client IP addresses.

The load balancer directs a particular client's requests to the same backend VM based on a hash created from the client's IP address and the load balancer's IP address (the internal IP address of an internal forwarding rule)

Console

1. Go to the Load balancing page in the Google Cloud Platform Console.
   Go to the Load balancing page
2. Click be-ilb (the name of the backend service you created for this example) and click Edit.
3. On the Edit Internal load balancer page, click Backend configuration.
4. Select Client IP from the Session affinity pop-up menu.
5. Click Update.

gcloud

Use the following gcloud command to update the be-ilb backend service, specifying client IP session affinity:

gcloud compute backend-services update be-ilb \
    --session-affinity CLIENT_IP

api

Make a PATCH request to the regionBackendServices/patch method.

PATCH https://www.googleapis.com/compute/v1/projects/[PROJECT_ID]/regions/us-west1/backendServices/be-ilb
{
  "sessionAffinity": "CLIENT_IP"
}

For more information about using session affinity to influence traffic distribution and a description of each option, see Traffic distribution.

Creating a forwarding rule in another subnet

This procedure creates a second IP address and forwarding rule in a different subnet to demonstrate that you can create multiple forwarding rules for one internal TCP/UDP load balancer. The region for the forwarding rule must match the region of the backend service.

Subject to firewall rules, clients in any subnet in the region can contact either internal TCP/UDP load balancer IP address.

1. Create a second subnet in the lb-network network in the us-west1 region:

   gcloud compute networks subnets create second-subnet \
       --network=lb-network \
       --range=10.5.6.0/24 \
       --region=us-west1
   

2. Create a second forwarding rule for ports 80 and 443. The other parameters for this rule, including IP address and backend service, are the same as for the primary forwarding rule, fr-ilb.

   gcloud compute forwarding-rules create fr-ilb-2 \
       --region=us-west1 \
       --load-balancing-scheme=internal \
       --network=lb-network \
       --subnet=second-subnet \
       --address=10.5.6.99 \
       --ip-protocol=TCP \
       --ports=80,443 \
       --backend-service=be-ilb \
       --backend-service-region=us-west1
   

3. Connect to the client VM instance and test HTTP and HTTPS connections to the IP addresses.

   • Connect to the client VM:

   gcloud compute ssh vm-client --zone=us-west1-a
   

   • Test HTTP connectivity to the IP addresses:

   curl http://10.1.2.99
   curl http://10.5.6.99
   

   • Test HTTPS connectivity. Use of --insecure is required because the Apache server configuration in the example setup uses self-signed certificates.

   curl https://10.1.2.99 --insecure
   curl https://10.5.6.99 --insecure
   

   • Observe that requests are handled by all of the backend VMs, regardless of the protocol (HTTP or HTTPS) or IP address used.

What's next

• See Internal TCP/UDP Load Balancing Concepts for important fundamentals.
• See Failover concepts for Internal TCP/UDP Load Balancing for important information about failover.
• See Internal Load Balancing and DNS Names for available DNS name options your load balancer can use.
• See Configuring failover for Internal TCP/UDP Load Balancing for configuration steps and an example internal TCP/UDP load balancer failover configuration.
• See Internal TCP/UDP Load Balancing Logging and Monitoring for information on configuring Stackdriver logging and monitoring for Internal TCP/UDP Load Balancing.
• See Internal TCP/UDP Load Balancing and Connected Networks for information about accessing internal TCP/UDP load balancers from peer networks connected to your VPC network.
• See Troubleshooting Internal TCP/UDP Load Balancing for information on how to troubleshoot issues with your internal TCP/UDP load balancer.