Setting Up Internal Load Balancing

This guide provides instructions for setting up an Internal Load Balancing example on your Google Cloud Platform network. Before you start, read Internal Load Balancing Concepts for information on how Internal Load Balancing works.

An Internal Load Balancing configuration consists of several components. All these components must belong to the same VPC network and region. The client instances originating traffic must belong to the same VPC network and region as the load balancing forwarding rule IP, backend services, and instance groups. The client instances do not have to be in the same subnet as the load balanced instances.

Setting up Internal Load Balancing (click to enlarge)
Setting up Internal Load Balancing (click to enlarge)

This example sets up an internal load balancer for testing purposes.

You will create the following:

  1. Four instances spread across two zones in the same region
  2. Instance groups for holding the instances
  3. Backend components, which include the following:
    • Health check - used to monitor instance health
    • Backend service - monitors instance groups and prevents them from exceeding configured usage
    • Backends - hold the instance groups
  4. A forwarding rule, with an internal IP address, that sends user traffic to the proxy
  5. A firewall rule that allows traffic from the load balancer IP address range
  6. A standalone client instance that can be used to test the load balancer

After these are created, you'll test your configuration.

Configure a test environment

The following sections walk you through creating the components of the test environment.

Configure a VPC network and subnet

Internal Load Balancing works with auto mode VPC networks, custom mode VPC networks, and legacy networks. For this example, you'll use a custom mode VPC network.

Console


  1. Go to the VPC networks page in the Google Cloud Platform Console.
    Go to the VPC network page
  2. Click Create VPC network.
  3. Enter a Name of my-custom-network.
  4. Under subnets, enter a Name of my-custom-subnet.
  5. Set Region to us-central1.
  6. Enter an IP address range of 10.128.0.0/20.
  7. Click Create.

gcloud


Create a custom VPC network

gcloud compute networks create my-custom-network --subnet-mode custom

Created     [https://www.googleapis.com/compute/v1/projects/[PROJECT_ID]/global/networks/my-custom-network].
NAME               MODE    IPV4_RANGE  GATEWAY_IPV4
my-custom-network  custom

Create a new subnet in your custom VPC network

gcloud compute networks subnets create my-custom-subnet \
    --network my-custom-network \
    --range 10.128.0.0/20 \
    --region us-central1

Created [https://www.googleapis.com/compute/v1/projects/[PROJECT_ID]/regions/us-central1/subnetworks/my-custom-subnet].
NAME              REGION       NETWORK            RANGE
my-custom-subnet  us-central1  my-custom-network  10.128.0.0/20

  • my-custom-network - the name of the VPC network you create, in which you then create a custom subnet.
  • my-custom-subnet - the name of the subnet you are creating.
  • 10.128.0.0/20 - For the range, this can be any valid RFC 1918 range that does not overlap another range in the same network. If you are creating your subnet in an existing VPC network, you may have to choose a different range.

Configure firewall rules

Instances on this VPC network will not be reachable until you create firewall rules. For example, you can allow all internal traffic between instances and allow SSH, RDP, and ICMP traffic from all sources.

Console


Create firewall rule that allows all traffic within the subnet

  1. Go to the Firewall rules page in the Google Cloud Platform Console.
    Go to the Firewall rules page
  2. Click Create firewall rule.
  3. Enter a Name of allow-all-10-128-0-0-20.
  4. Set Network to my-custom-network.
  5. Set Source filter to IP ranges.
  6. Set Source IP ranges to 10.128.0.0/20.
  7. Set Specified protocols and ports to tcp;udp;icmp.
  8. Click Create.

Create firewall rule that allows SSH, RDP, and ICMP from anywhere

  1. Create a second firewall rule with a Name of allow-tcp22-tcp3389-icmp.
  2. Set Network to my-custom-network.
  3. Set Source filter to IP ranges.
  4. Set Source IP ranges to 0.0.0.0/0 (allow from any source).
  5. Set Specified protocols and ports to tcp:22;tcp:3389;icmp.
  6. Click Create.

gcloud


Create firewall rule that allows all traffic within the subnet

gcloud compute firewall-rules create allow-all-10-128-0-0-20 \
        --network my-custom-network \
        --allow tcp,udp,icmp \
        --source-ranges 10.128.0.0/20

Create firewall rule that allows SSH, RDP, and ICMP from anywhere

gcloud compute firewall-rules create allow-tcp22-tcp3389-icmp \
   --network my-custom-network \
   --allow tcp:22,tcp:3389,icmp

Configure instances and instance groups

A production system would normally use managed instance groups based on instance templates, but this setup is quicker for initial testing.

Create two instances in each zone

For testing purposes, install Apache on each instance. These instance are all being created with a tag of int-lb. This tag is used later by the firewall rule.

The four instances are named ig-us-central1-1 through ig-us-central1-4.

Console


Create instances

  1. Go to the VM instances page in the Google Cloud Platform Console.
    Go to the VM instances page
  2. Click Create instance.
  3. Set Name to ig-us-central1-1.
  4. Set the Zone to us-central1-b.
  5. Click Management, disk, networking, SSH keys to reveal advanced settings.
  6. Under Management, click Networking and populate the Tags field with int-lb.
  7. Under Networking, edit the network interface under Network interfaces.
  8. Under Networking, ​set Network to my-custom-network and Subnetwork to my-custom-subnet.
  9. Leave the default values for rest of the fields.
  10. Click Management and set the Startup script to
    sudo apt-get update
    sudo apt-get install apache2 -y
    sudo a2ensite default-ssl
    sudo a2enmod ssl
    sudo service apache2 restart
    echo '<!doctype html><html><body><h1>ig-us-central1-1</h1></body></html>' | sudo tee /var/www/html/index.html
  11. Create ig-us-central1-2 with the same settings, except with Startup script set to
    sudo apt-get update
    sudo apt-get install apache2 -y
    sudo a2ensite default-ssl
    sudo a2enmod ssl
    sudo service apache2 restart
    echo '<!doctype html><html><body><h1>ig-us-central1-2</h1></body></html>' | sudo tee /var/www/html/index.html
  12. Create ig-us-central1-3 with the same settings, except with Zone set to us-central1-c and Startup script set to
    sudo apt-get update
    sudo apt-get install apache2 -y
    sudo a2ensite default-ssl
    sudo a2enmod ssl
    sudo service apache2 restart
    echo '<!doctype html><html><body><h1>ig-us-central1-3</h1></body></html>' | sudo tee /var/www/html/index.html
  13. Create ig-us-central1-4 with the same settings, except with Zone set to us-central1-c and Startup script set to
    sudo apt-get update
    sudo apt-get install apache2 -y
    sudo a2ensite default-ssl
    sudo a2enmod ssl
    sudo service apache2 restart
    echo '<!doctype html><html><body><h1>ig-us-central1-4</h1></body></html>' | sudo tee /var/www/html/index.html
    

gcloud


gcloud compute instances create ig-us-central1-1 \
        --image-family debian-9 \
        --image-project debian-cloud \
        --tags int-lb \
        --zone us-central1-b \
        --subnet my-custom-subnet \
        --metadata startup-script="#! /bin/bash
          apt-get update
          apt-get install apache2 -y
          a2ensite default-ssl
          a2enmod ssl
          service apache2 restart
          echo '<!doctype html><html><body><h1>ig-us-central1-1</h1></body></html>' | tee /var/www/html/index.html
          EOF"

Created [https://www.googleapis.com/compute/v1/projects/[PROJECT_ID]/zones/us-central1-b/instances/ig-us-central1-1].
NAME             ZONE          MACHINE_TYPE  PREEMPTIBLE INTERNAL_IP EXTERNAL_IP    STATUS
ig-us-central1-1 us-central1-b n1-standard-1             10.128.0.3  23.251.150.133 RUNNING

gcloud compute instances create ig-us-central1-2 \
    --image-family debian-9 \
    --image-project debian-cloud \
    --tags int-lb \
    --zone us-central1-b \
    --subnet my-custom-subnet \
    --metadata startup-script="#! /bin/bash
      apt-get update
      apt-get install apache2 -y
      a2ensite default-ssl
      a2enmod ssl
      service apache2 restart
      echo '<!doctype html><html><body><h1>ig-us-central1-2</h1></body></html>' | tee /var/www/html/index.html
      EOF"

Created [https://www.googleapis.com/compute/v1/projects/[PROJECT_ID]/zones/us-central1-b/instances/ig-us-central1-2].
NAME             ZONE          MACHINE_TYPE  PREEMPTIBLE INTERNAL_IP EXTERNAL_IP    STATUS
ig-us-central1-2 us-central1-b n1-standard-1             10.128.0.11 23.251.148.160 RUNNING

gcloud compute instances create ig-us-central1-3 \
    --image-family debian-9 \
    --image-project debian-cloud \
    --tags int-lb \
    --zone us-central1-c \
    --subnet my-custom-subnet \
    --metadata startup-script="#! /bin/bash
      apt-get update
      apt-get install apache2 -y
      a2ensite default-ssl
      a2enmod ssl
      service apache2 restart
      echo '<!doctype html><html><body><h1>ig-us-central1-3</h1></body></html>' | tee /var/www/html/index.html
      EOF"

Created [https://www.googleapis.com/compute/v1/projects/[PROJECT_ID]/zones/us-central1-c/instances/ig-us-central1-3].
NAME             ZONE          MACHINE_TYPE  PREEMPTIBLE INTERNAL_IP EXTERNAL_IP    STATUS
ig-us-central1-3 us-central1-c n1-standard-1             10.128.0.12 104.196.31.214 RUNNING

gcloud compute instances create ig-us-central1-4 \
    --image-family debian-9 \
    --image-project debian-cloud \
    --tags int-lb \
    --zone us-central1-c \
    --subnet my-custom-subnet \
    --metadata startup-script="#! /bin/bash
      apt-get update
      apt-get install apache2 -y
      a2ensite default-ssl
      a2enmod ssl
      service apache2 restart
      echo '<!doctype html><html><body><h1>ig-us-central1-4</h1></body></html>' | tee /var/www/html/index.html
      EOF"

Created [https://www.googleapis.com/compute/v1/projects/[PROJECT_ID]/zones/us-central1-c/instances/ig-us-central1-4].
NAME             ZONE          MACHINE_TYPE  PREEMPTIBLE INTERNAL_IP EXTERNAL_IP    STATUS
ig-us-central1-4 us-central1-c n1-standard-1             10.128.0.13 104.196.25.101 RUNNING

Create an instance group for each zone and add instances

Console


  1. Go to the Instance groups page in the Google Cloud Platform Console.
    Go to the Instance groups page
  2. Click Create instance group.
  3. Set the Name to us-ig1.
  4. Set the Zone to us-central1-b.
  5. Under Group type, select Unmanaged instance group.
  6. Set Network to my-custom-network.
  7. Set Subnetwork to my-custom-subnet.
  8. From VM instances select ig-us-central1-1 and ig-us-central1-2.
  9. Leave other settings as they are.
  10. Click Create.
  11. Repeat steps, but set the following:
    • Name: us-ig2
    • Zone: us-central1-c
    • Instances: ig-us-central1-3 and ig-us-central1-4.
  12. Confirm that you now have two instance groups, each with two instances.

gcloud


  1. Create the us-ig1 instance group.

    gcloud compute instance-groups unmanaged create us-ig1 \
        --zone us-central1-b
    

    Created [https://www.googleapis.com/compute/v1/projects/[PROJECT_ID]/zones/us-central1-b/instanceGroups/us-ig1].
    NAME   ZONE          NETWORK MANAGED INSTANCES
    us-ig1 us-central1-b                 0

  2. Add ig-us-central1-1 and ig-us-central1-2 to us-ig1

    gcloud compute instance-groups unmanaged add-instances us-ig1 \
        --instances ig-us-central1-1,ig-us-central1-2 --zone us-central1-b
    

    Updated [https://www.googleapis.com/compute/v1/projects/[PROJECT_ID]/zones/us-central1-b/instanceGroups/us-ig1].

  3. Create the us-ig2 instance group.

    gcloud compute instance-groups unmanaged create us-ig2 \
        --zone us-central1-c
    

    Created [https://www.googleapis.com/compute/v1/projects/[PROJECT_ID]/zones/us-central1-c/instanceGroups/us-ig2].
    NAME   ZONE          NETWORK MANAGED INSTANCES
    us-ig2 us-central1-c                  0

  4. Add ig-us-central1-3 and ig-us-central1-4 to us-ig2

    gcloud compute instance-groups unmanaged add-instances us-ig2 \
        --instances ig-us-central1-3,ig-us-central1-4 \
        --zone us-central1-c
    

    Updated [https://www.googleapis.com/compute/v1/projects/[PROJECT_ID]/zones/ig-us-central1-c/instanceGroups/us-ig2].

Configure the load balancer

Console


Create the load balancer and configure a backend service

  1. Go to the Load balancing page in the Google Cloud Platform Console.
    Go to the Load balancing page
  2. Click Create load balancer.
  3. Under TCP load balancing, click Start configuration.
  4. Under Internet facing or internal only select Only between my VMs.
  5. Click Continue.
  6. Set the Name to my-int-lb.

Configure backend services

  1. Click Backend configuration.
  2. The Name of the backend service appears as my-int-lb.
  3. Set Region to us-central1.
  4. Set Network to my-custom-network.
  5. Under Backends, select instance group us-ig1.
  6. Click Add backend.
  7. Select instance group us-ig2.
  8. Under Health check, select Create another health check.
    1. Set the health check Name to my-tcp-health-check.
    2. Set Protocol to TCP.
    3. Leave the other settings the same.
    4. Click Save and continue.
  9. Verify that there is a blue check mark next to Backend configuration in the Google Cloud Platform Console. If not, double-check that you have completed all the steps above.

Configure frontend services

  1. Click Frontend configuration.
  2. Under Subnetwork, select my-custom-subnet.
  3. Under Internal IP, select Ephemeral (Automatic).
  4. Set Ports to 80.
  5. Verify that there is a blue check mark next to Frontend configuration in the Google Cloud Platform Console. If not, double-check that you have completed all the steps above.

Review and finalize

  1. Click Review and finalize.
  2. Double-check your settings.
  3. Click Create.

gcloud


Create a health check

gcloud compute health-checks create tcp my-tcp-health-check \
    --port 80

Created [https://www.googleapis.com/compute/v1/projects/[PROJECT_ID]/global/healthChecks/my-tcp-health-check].
NAME                PROTOCOL
my-tcp-health-check TCP

Create a backend service

gcloud compute backend-services create my-int-lb \
    --load-balancing-scheme internal \
    --region us-central1 \
    --health-checks my-tcp-health-check \
    --protocol tcp

Created [https://www.googleapis.com/compute/v1/projects/[PROJECT_ID]/regions/us-central1/backendServices/my-int-lb].
NAME               BACKENDS PROTOCOL
my-int-lb          TCP

Add instance groups to your backend service

Internal Load Balancing spreads incoming connections based on the session affinity setting. If session affinity is not set, the load balancer spreads all connections across all available instances as evenly as possible regardless of current load.

gcloud compute backend-services add-backend my-int-lb \
    --instance-group us-ig1 \
    --instance-group-zone us-central1-b \
    --region us-central1

Updated [https://www.googleapis.com/compute/v1/projects/[PROJECT_ID]/regions/us-central1/backendServices/my-int-lb].

gcloud compute backend-services add-backend my-int-lb \
    --instance-group us-ig2 \
    --instance-group-zone us-central1-c \
    --region us-central1

Updated [https://www.googleapis.com/compute/v1/projects/[PROJECT_ID]//regions/us-central1/backendServices/my-int-lb].

Create a forwarding rule

Create a forwarding rule to forward traffic to the internal load balancer. For Internal load balancing, the forwarding rule points directly to a backend service in the same region as the forwarding rule.

gcloud beta compute forwarding-rules create my-int-lb-forwarding-rule \
    --load-balancing-scheme internal \
    --address 10.128.0.6 \
    --ports 80 \
    --network my-custom-network \
    --subnet my-custom-subnet \
    --service-label my-ilb-label \
    --region us-central1 \
    --backend-service my-int-lb

Created [https://www.googleapis.com/compute/beta/projects/[PROJECT_ID]/regions/us-central1/forwardingRules/my-int-lb-forwarding-rule].

IPAddress: 10.128.0.6 IPProtocol: TCP backendService: https://www.googleapis.com/compute/v1/projects/[PROJECT_ID]/regions/us-central1/backendServices/my-int-lb creationTimestamp: '2016-06-17T12:56:44.843-07:00' description: '' id: '6922319202683054867' kind: compute#forwardingRule loadBalancingScheme: INTERNAL name: my-int-lb-forwarding-rule network: https://www.googleapis.com/compute/v1/projects/[PROJECT_ID]/global/networks/my-custom-network ports: - '80' region: us-central1 selfLink: https://www.googleapis.com/compute/v1/projects/[PROJECT_ID]/regions/us-central1/forwardingRules/my-int-lb-forwarding-rule subnetwork: https://www.googleapis.com/compute/v1/projects/[PROJECT_ID]/regions/us-central1/subnetworks/my-custom-subnet

Configure a firewall rule to allow Internal Load Balancing

Configure one firewall rule to allow traffic to the load balancer and from the load balancer to the instances. Configure another to allow health check probes from the health checker

Console


  1. Go to the Firewall rules page in the Google Cloud Platform Console.
    Go to the Firewall rules page
  2. Click Create firewall rule.
  3. Set Name to allow-internal-lb.
  4. Set Network to my-custom-network.
  5. Set Target tags to int-lb.
  6. Set Source filter to IP ranges.
  7. Set Source IP ranges to 10.128.0.0/20.
  8. Set Specified protocols and ports to tcp:80;tcp:443.
  9. Click Create.
  10. Click Create firewall rule.
  11. Set Name to allow-health-check.
  12. Set VPC network to my-custom-network.
  13. Set Target tags to int-lb.
  14. Set Source filter to IP ranges.
  15. Set Source IP ranges to 130.211.0.0/22 and 35.191.0.0/16.
  16. Set Specified protocols and ports to tcp.
  17. Click Create.

gcloud


gcloud compute firewall-rules create allow-internal-lb \
   --network my-custom-network \
    --source-ranges 10.128.0.0/20 \
    --target-tags int-lb \
    --allow tcp:80,tcp:443

Created [https://www.googleapis.com/compute/v1/projects/[PROJECT_ID]/global/firewalls/allow-internal-lb].
NAME               NETWORK            SRC_RANGES     RULES           SRC_TAGS  TARGET_TAGS
allow-internal-lb  my-custom-network  10.128.0.0/20  tcp:80,tcp:443            int-lb

gcloud compute firewall-rules create allow-health-check \
    --network my-custom-network \
    --source-ranges 130.211.0.0/22,35.191.0.0/16 \
    --target-tags int-lb \
    --allow tcp

NAME                NETWORK            SRC_RANGES                    RULES  SRC_TAGS  TARGET_TAGS
allow-health-check  my-custom-network  130.211.0.0/22,35.191.0.0/16  tcp              int-lb

Create a standalone client instance

For testing purposes, create a standalone client instance in the same region as the load balancer.

Console


  1. Go to the VM instances page in the Google Cloud Platform Console.
    Go to the VM instances page
  2. Click Create instance.
  3. Set Name to standalone-instance-1.
  4. Set the Zone to us-central1-b.
  5. Click Management, disk, networking, SSH keys to to reveal advanced settings.
  6. Under Networking, populate the Networking tags field with standalone.
  7. Under Networking, edit the network interface. Set Network to my-custom-network and Subnetwork to my-custom-subnet.
  8. Click Create.

gcloud


gcloud compute instances create standalone-instance-1 \
    --image-family debian-9 \
    --image-project debian-cloud \
    --zone us-central1-b \
    --tags standalone \
    --subnet my-custom-subnet

Created [https://www.googleapis.com/compute/v1/projects/[PROJECT_ID]/zones/us-central1-b/instances/standalone-instance-1].
NAME                  ZONE          MACHINE_TYPE  PREEMPTIBLE INTERNAL_IP EXTERNAL_IP    STATUS
standalone-instance-1 us-central1-b n1-standard-1             10.128.0.8  23.251.150.133 RUNNING

Delete the external IP addresses from your instances

When you created the instances for your instance groups, the instances needed external IP addresses to download and install Apache. Now that Apache is installed and since these instances are going to serve an internal load balancer, they do not need access to the Internet, so you can remove the external IP addresses from the instances.

Console


  1. Go to the VM instances page in the Google Cloud Platform Console.
    Go to the VM instances page
  2. Click on ig-us-central1-1 to get to the console for that instance.
  3. Click Edit.
  4. Expand "Network interfaces". Set External IP to None.
  5. Click Save.
  6. Repeat for ig-us-central1-2.
  7. Repeat for ig-us-central1-3.
  8. Repeat for ig-us-central1-4.

gcloud


gcloud compute instances delete-access-config ig-us-central1-1 \
    --access-config-name external-nat --zone us-central1-b

gcloud compute instances delete-access-config ig-us-central1-2 \
    --access-config-name external-nat --zone us-central1-b

gcloud compute instances delete-access-config ig-us-central1-3 \
    --access-config-name external-nat --zone us-central1-c

gcloud compute instances delete-access-config ig-us-central1-4 \
    --access-config-name external-nat --zone us-central1-c

Test your load balancer

From your local machine, connect to your standalone client instance.

<pre class="devsite-click-to-copy">
gcloud compute --project [PROJECT_ID] ssh --zone us-central1-b standalone-instance-1
</pre>

Use curl to contact the load balancer's internal IP address. Repeat a few times to see other instances responding.

$ curl 10.128.0.6
<!doctype html><html><body><h1>ig-us-central1-2</h1></body></html>
$ curl 10.128.0.6
<!doctype html><html><body><h1>ig-us-central1-1</h1></body></html>
$ curl 10.128.0.6
<!doctype html><html><body><h1>ig-us-central1-4</h1></body></html>
$ curl 10.128.0.6
<!doctype html><html><body><h1>ig-us-central1-4</h1></body></html>

Use curl to reach your internal load balancer via its service name.

$ curl my-ilb-label.my-int-lb-forwarding-rule.il4.us-central1.lb.[PROJECT_ID].internal
<!doctype html><html><body><h1>ig-us-central1-2</h1></body></html>
$ curl my-ilb-label.my-int-lb-forwarding-rule.il4.us-central1.lb.[PROJECT_ID].internal
<!doctype html><html><body><h1>ig-us-central1-1</h1></body></html>
$ curl my-ilb-label.my-int-lb-forwarding-rule.il4.us-central1.lb.[PROJECT_ID].internal
<!doctype html><html><body><h1>ig-us-central1-4</h1></body></html>
$ curl my-ilb-label.my-int-lb-forwarding-rule.il4.us-central1.lb.[PROJECT_ID].internal6
<!doctype html><html><body><h1>ig-us-central1-4</h1></body></html>

New gcloud command parameters for Internal load balancing

Internal Load Balancing uses the same forwarding rule, backend service, and health check resources as HTTP(S) load balancing, but with some new command parameters. This section explains only the new parameters. Existing parameters are explained in the gcloud command-line tool reference.

Forwarding rule

For Internal Load Balancing, the forwarding rule points directly to a backend service in the same region as the forwarding rule. This section describes the new parameters in the context of the create command. The commands for get, list, describe, and delete remain the same. See the forwarding-rules gcloud command for explanations of other parameters.

New forwarding-rules create parameters

gcloud compute forwarding-rules create [NAME] \
    --load-balancing-scheme INTERNAL \
    --address [ADDRESS] \
    --region [REGION] \
    --ip-protocol [IP_PROTOCOL] \
    --ports [[PORT] | [START_PORT]-[END_PORT]],[...] \
    --backend-service [SERVICE_NAME] \
    --subnet [SUBNET_NAME] \
    --network [NETWORK]
  • --load-balancing-scheme — for Internal Load Balancing, this parameter must be specified as internal. Initially, this parameter is only supported for Internal Load Balancing
  • --address — the target IP address of the internal load balancer. When the load balancing scheme is internal, this can only be an RFC 1918 IP address belonging to the subnet configured for the forwarding rule. In this case, an external address cannot be used. If this parameter is unspecified and the load balancing scheme is internal, the IP address for the forwarding rule is automatically allocated from the internal IP range of the subnet or network configured for the forwarding rule.
  • --region — the region where the forwarding rule is created. Global forwarding rules are not supported for Internal Load Balancing.
  • --ip-protocol — The protocol, either tcp or udp, that the internal load balancer will process.
  • --ports — A parameter that lets you specify 1-5 ports in a comma-separated list. Only packets destined for listed ports are forwarded.
  • --backend-service — the backend service that will receive all traffic directed to the forwarding rule.
  • --subnet — The subnet that this forwarding rule applies to. When configured for Internal Load Balancing, the internal IP of the forwarding rule is assigned from this subnet. If the network is in auto subnet mode, the subnet is optional because it can be determined from the region. However, if the network is in custom subnet mode, a subnet must be specified. Instances configured as backends for an internal load balancer may belong to different subnets in the same region.
  • --network — The network that this forwarding rule applies to. When configured for Internal Load Balancing, the internal IP of the forwarding rule is assigned from this network. If this field is not specified, the default network will be used. In the absence of the default network, this field must be specified.

Set the target of a forwarding rule

The forwarding-rules set-target command lets you change which backend service the forwarding rule directs to. For Internal Load Balancing, the target of the forwarding rule must be a backend service and the --backend-service and --region parameters are required.

Console


Use the gcloud command-line tool for this step.

gcloud


    gcloud compute forwarding-rules set-target \
        --region [REGION] \
        --backend-service [NAME]
    

Backend service

The forwarding rule directs your client request to the backend service. In turn, the backend service directs this client request to a healthy instance in the backend. To be considered healthy, and instance must have passed its health check.

New backend-services create parameters

See the backend-services gcloud command for explanations of other parameters.

gcloud compute backend-services create [NAME] \
    --load-balancing-scheme internal \
    --region \
    --health-checks \
    --protocol \
    [--session-affinity] \
    [--timeout]
  • --load-balancing-scheme internal (required) — Specifies whether the load balancer is performing internal or external load balancing. Only Internal Load Balancing is supported for this release. A value of internal is required.
  • --region (required) — The region where the backend service resides. For Internal Load Balancing, the instances, instance groups, backend service, load balancer service, and client instances must be in the same region of the same network as each other. They do not have to be in the same zone or subnet as each other.
  • --health-checks (required) — The name of the health check the backend service uses to monitor instances. The --health-checks parameter is required. The --http-health-checks and --https-health-checks parameters are not supported.
  • --protocol (default: http) — The protocol, either tcp or udp, that the load balancer will process. This parameter must match the --protocol parameter specified for the forwarding rule. The protocol is required for backend services as the default value is http and only tcp and udp are supported for Internal Load Balancing.
  • --session-affinity — If unspecified, new connections to the load balancer from the same instance are distributed evenly across backend instances. To cause connections from the same client instance to end up on the same backend instance, specify a session affinity of client_ip or client_ip_proto. See Session affinity for more details.
  • --timeout (default=30s) — Specifies how many seconds to wait for a response from the backend instance group before considering it a failed request.

New parameters for backend-services add-backend command

See the backend-services add-backend command for explanations of other parameters.

gcloud compute backend-services add-backend [INSTANCE_GROUP] \
    --balancing-mode connection \
    --instance-group-zone \
    --region
  • --balancing-mode — only a balancing mode of CONNECTION is supported for Internal Load Balancing. Balances based on the total number of outstanding connections. Instance utilization and current ongoing connections to the instance are not taken into account. Incoming connections are spread as evenly as possible across all available instances regardless of current load. Established connections stay on the same instance regardless of new incoming connections. The --max-rate and --max-rate-per-instance parameters are not supported for Internal Load Balancing.
  • --instance-group-zone — the zone of the instance group you are associating with the backend.
  • --region — you must specify the region of the backend. This is the same region as the rest of the internal load balancer components.
  • --instance-group-region — the region of the regional instance group you are associating with the backend.
  • The following parameters are not valid for Internal Load Balancing:
    • --capacity-scaler
    • --max-rate
    • --max-rate-per-instance
    • --max-utilization

NOTE: Commands for get, list, describe, remove-backend, and delete now take an additional --regions flag when used for corresponding regional backend services.

Connection draining for backend services

You can enable connection draining on backend services to ensure minimal interruption to your users when an instance is removed from an instance group, either manually or by an autoscaler. Connection draining is only available for TCP connections. Connection draining is not supported for UDP traffic. To learn more about connection draining, read the Enabling Connection Draining documentation.

Health checks

Health checks determine which instances can receive new connections. The health checker probes instances at specified intervals. Instances that do not respond successfully to the probe a specified number of times in a row are marked as UNHEALTHY. Health checks can be of type tcp, ssl, http, or https.

You can configure a TCP, SSL, HTTP, or HTTPS health check for determining the health of your instances.

  • If the service running on your backend instances is based on HTTP, use an HTTP health check
  • If the service running on your backend instances is based on HTTPS, use an HTTPS health check
  • If the service running on your backend instances uses SSL, use an SSL health check
  • Unless you have an explicit reason to use a different kind of health check, use a TCP health check

Once configured, health check probes are sent on a regular basis on the specified port to all the instances in the configured instance groups.

See Health checking for more detailed information on how health checks work.

Health-checks create parameters

gcloud compute health-checks create [tcp | ssl | http | https] my-health-check \
    ...options

If you are encrypting traffic between the load balancer and your instances, use an SSL or HTTPS health check.

  • --check-interval (default=5s) — How often to send a health check probe to an instance. For example, specifying 10s will send the probe every 10 seconds. Valid units for this flag are s for seconds and m for minutes.
  • --healthy-threshold (default=2) — The number of consecutive successful health check probes before an unhealthy instance is marked as HEALTHY.
  • --port (default=80 for TCP and HTTP, 443 for SSL and HTTPS) — The TCP port number that this health check monitors.
  • --request — Only valid for TCP and SSL health checks. An optional string of up to 1024 characters that the health checker can send to the instance. The health checker then looks for a reply from the instance of the string provided in the --response field. If --response is not configured, the health checker does not wait for a response and regards the probe as successful if the TCP or SSL handshake was successful.
  • --response — Only valid for TCP and SSL health checks. An optional string of up to 1024 characters that the health checker expects to receive from the instance. If the response is not received exactly, the health check probe fails. If --response is configured, but not --request, the health checker will wait for a response anyway. Unless your system automatically sends out a message in response to a successful handshake, always configure --response to match an explicit --request.
  • --timeout (default=5s) — If the health checker doesn't receive valid response from the instance within this interval, the probe is considered a failure. For example, specifying 10s will cause the checker to wait for 10 seconds before considering the probe a failure. Valid units for this flag are s for seconds and m for minutes.
  • --unhealthy-threshold (default=2) — The number of consecutive health check probe failures before a healthy instance is marked as UNHEALTHY.

Health check source IP addresses and firewall rules

For Internal Load Balancing, the health check probes to your load balanced instances come from addresses in the ranges 130.211.0.0/22 and 35.191.0.0/16. These are IP address ranges that the load balancer uses to connect to backend instances. Your firewall rules must allow these connections.

The section Configure a firewall rule to allow Internal Load Balancing covers this step.

If you need to determine external IP addresses at a particular time, use the instructions in the Google Compute Engine FAQ.

Internal Load Balancing with regional instance groups

You can use Internal Load Balancing with regional instance groups. The gcloud commands are as follows.

This command creates a regional instance group in us-central1, which can then be autoscaled.

    gcloud compute instance-groups managed create us-central1-ig1 \
        --region us-central1

This command creates a backend service in us-central1. A backend containing the regional instance group can then be added.

    gcloud compute backend-services create my-int-lb \
        --load-balancing-scheme internal \
        --region us-central1 \
        --health-checks my-tcp-health-check

This command adds the regional instance group to the backend service.

    gcloud compute backend-services add-backend my-int-lb \
        --instance-group us-central1-ig1 \
        --instance-group-region us-central1 \
        --region us-central1

You can then attach this backend service to a forwarding rule with load-balancing-scheme set to internal:

    gcloud compute forwarding-rules create internal-lb-forwarding-rule \
        --load-balancing-scheme internal \
        --ports 80 \
        --network my-custom-network \
        --subnet my-custom-subnet \
        --region us-central1 \
        --backend-service my-int-lb

Troubleshooting

No response from the internal load balancer

Multiple issues can manifest as no response from the internal load balancer. For example:

  • Server on backend instance listening on wrong IP address
  • Server on backend instance listening on wrong port
  • Firewall not allowing traffic

To resolve these issues, consider the following:

  • Traffic sent to an internal load balancer arrives at backend instances with the destination IP of the load balancer itself.
  • Confirm the instance software is configured to listen either on "any IP" (0.0.0.0 or ::), or on the IP address of the load balancer rather than the IP address of the instance or pod.
  • There should be at least one (1) healthy backend in the load balancer's backend service. If there are no healthy backends:
  • Confirm the configuration of the backend service.
  • Confirm the configuration of the health check.
  • Confirm that the configuration of the VPC firewall and host firewall allows health checks in accordance with the health check documentation

All traffic goes to one instance behind the internal load balancer

If traffic destined for an internal load balancer is sourced from an instance behind that same load balancer, all requests go to that same instance. Traffic sourced from outside the load balancer's backend service is properly load balanced.

Traffic goes to all instances, but is unbalanced

Session affinity can cause traffic to arrive unbalanced. If session affinity is not in use, it can be helpful to understand that traffic can be uneven at lower volumes. Higher volumes of traffic distribute more evenly. In addition, very short sample times can exacerbate the appearance of uneven distribution.

Troubleshooting access across VPN and Interconnect

Internal Load Balancing access across VPN and Interconnect combines two GCP products. To isolate any issues, we recommend that you first attempt to access the internal load balancer from a client GCP VM instance in the same network and region as the VPN tunnel.

If you can successfully access the internal load balancer from the client VM instance, check the rest of this section for common issues that you might encounter.

VPN issues

If you encounter any VPN issues when you access Internal Load Balancing over VPN, read the Cloud VPN Troubleshooting Guide.

Global routing issue

Internal Load Balancing is a regional product. All clients and backend VM instances, as well as VPN tunnels, need to be in the same region.

Because of this, Internal Load Balancing access is lost when global routing uses a tunnel outside the Internal Load Balancing region.

Access is lost because global routing can be configured with tunnels that are local to the Internal Load Balancing region and tunnels that are outside the Internal Load Balancing region. If the local tunnel is configured with a higher priority or no priority is specified, then the local tunnel is preferred, which allows access to Internal Load Balancing. However, if the local tunnel fails, then all traffic is routed to the remote tunnel and access to Internal Load Balancing is lost.

UDP and MTU

If you see large UDP packets dropped when accessing Internal Load Balancing over VPN, it is because Path MTU Discovery (PMTUD) is not supported for UDP.

To correct this issue, ensure that interface MTU is configured to take into account IPSec overhead and GCP VM instance MTU of 1460.

VPC Peering is not supported

If your on-premises clients cannot access a peered internal load balancer, it is because VPC Peering does not support VPN access. You cannot use these features together.

Troubleshooting DNS service discovery issues

ERROR: unrecognized arguments: --service-label.

The error is caused by wrong API version. The feature is not in GA yet. Use the beta API to access this feature.

ERROR: Could not fetch resource: The resource 'projects/<project>/global/networks/my-custom-network' was not found.

The error is caused by specifying a network that does not exist. The network field is optional. If you do not specify the network, the forwarding rule uses the default network. If you do specify a network, it must exist before you create the forwarding rule. See Using VPC for instructions on how to create a network.

ERROR: Could not fetch resource: The resource 'projects/<project>/regions/<region>/backendServices/my-int-lb' was not found.

This error means that the backend service name was incorrect or does not exist. See the backend service for more information about backend services.

ERROR: Could not fetch resource: Invalid value for field 'resource.ports[0]': ''. Forwarding rules with backend services must specify at least one port.

The error is caused by a missing ports parameter or invalid value for ports. The ports is a required field for forwarding rules with backend services.

ERROR: Invalid value for field 'resource.serviceLabel': ''. Must be a match of regex '(?:a-z?)'.

The error is caused by invalid service label. A valid service label must meet the following rules:

  • A service label may be composed of the following valid characters: a to z (lower case letters), 0 to 9 (numeric characters) and - (dash)
  • A service label MUST start with a lowercase letter
  • A service label MUST end with a lowercase letter or a number
  • A service label can be up to 63 characters

ERROR: Invalid value for field 'resource.serviceLabel': ''. The serviceLabel field can only be used if loadBalancingScheme is INTERNAL.

The error is caused by a missing load balancing scheme value or by specifying a value instead of INTERNAL. Service labels only work with Internal Load Balancing. Specify a load balancing scheme of INTERNAL when creating the forwarding rule.

Error: You cannot create any more internal forwarding rules in the project.

Internal forwarding rules are limited by default to 50 per project. See the documentation for Requesting Quotas for more information.

Restrictions

  • The internal load balancer IP cannot be the next-hop IP of a manually configured route. If a route is configured that has the IP of the internal load balancer as its target, traffic matching that route will be dropped.
  • Clients in a region cannot access an internal load balancer in a different region. All the instances in the Internal Load Balancing configuration must belong to the same VPC network and region, but can be in different subnets.

What's next

Was this page helpful? Let us know how we did:

Send feedback about...

Load Balancing