Setting Up Internal HTTP(S) Load Balancing for Compute Engine VMs

This document provides instructions for configuring Internal HTTP(S) Load Balancing for your services running on Compute Engine VMs.

Before you begin

Before following the instructions in this guide, review the following documents:

Configuring Internal HTTP(S) Load Balancing with a VM-based service

This section shows the configuration required for services that run on Compute Engine VMs. Client VMs connect to the IP address and port that you configure in the forwarding rule. When your client applications send traffic to this IP address and port, their requests are forwarded to your backend virtual machines (VMs) according to your internal HTTP(S) load balancer's URL map.

The example on this page explicitly sets a reserved internal IP address for the internal HTTP(S) load balancer's forwarding rule, rather than allowing an ephemeral internal IP address to be allocated. As a best practice, we recommend reserving IP addresses for forwarding rules.

For the forwardng rule's IP address, use the backend-subnet. If you try to use the proxy-only subnet, forwarding rule creation fails.

Creating a managed instance group

This section shows how to create a template and a managed instance group. The managed instance group provides VM instances running the backend servers of an example internal HTTP(S) load balancer. Traffic from clients is load balanced to these backend servers. For demonstration purposes, backends serve their own hostnames.

Console

  1. Go to the Instance groups page in the Cloud Console.

    Go to the Instance Groups page

  2. Click Create instance group.
  3. Choose New managed instance group on the left.
  4. For the Name, enter l7-ilb-backend-example.
  5. Under Location, select Single zone.
  6. For the Region, select us-west1.
  7. For the Zone, select us-west1-a.
  8. Under Instance template, select Create a new instance template.

    1. For the Name, enter l7-ilb-backend-template.
    2. Ensure that the Boot disk is set to a Debian image, such as Debian GNU/Linux 9 (stretch). These instructions use commands that are only available on Debian, such as apt-get.
    3. Under Management, security, disks, networking, sole tenancy, on the Management tab, insert the following script into the Startup script field.

      #! /bin/bash
      apt-get update
      apt-get install apache2 -y
      a2ensite default-ssl
      a2enmod ssl
      vm_hostname="$(curl -H "Metadata-Flavor:Google" \
      http://169.254.169.254/computeMetadata/v1/instance/name)"
      echo "Page served from: $vm_hostname" | \
      tee /var/www/html/index.html
      systemctl restart apache2'
      
    4. Under Networking, select lb-network as the Network, and for the Subnet, select backend-subnet.

    5. Add the following network tags: allow-ssh and load-balanced-backend.

    6. Click Save and continue.

  9. Specify the number of instances that you want to create in the group.

    For this example, under Autoscaling mode, you can select:

    • Don't configure autoscaling
    • Under Number of instances, enter 2

    Optionally, in the Autoscaling section of the UI, you can configure the instance group to automatically add or remove instances based on instance CPU usage.

  10. Click Create to create the new instance group.

gcloud

The gcloud instructions in this guide assume that you are using Cloud Shell or another environment with bash installed.

  1. Create a VM instance template with HTTP server with the gcloud compute instance-templates create command.

    gcloud compute instance-templates create l7-ilb-backend-template \
    --region=us-west1 \
    --network=lb-network \
    --subnet=backend-subnet \
    --tags=allow-ssh,load-balanced-backend \
    --image-family=debian-9 \
    --image-project=debian-cloud \
    --metadata=startup-script='#! /bin/bash
    apt-get update
    apt-get install apache2 -y
    a2ensite default-ssl
    a2enmod ssl
    vm_hostname="$(curl -H "Metadata-Flavor:Google" \
    http://169.254.169.254/computeMetadata/v1/instance/name)"
    echo "Page served from: $vm_hostname" | \
    tee /var/www/html/index.html
    systemctl restart apache2'
    
  2. Create a managed instance group in the zone with the gcloud compute instance-groups managed create command.

    gcloud compute instance-groups managed create l7-ilb-backend-example \
        --zone=us-west1-a \
        --size=2 \
        --template=l7-ilb-backend-template
    

api

Create the instance template with the instanceTemplates.insert method, replacing [project-id] with your project ID.

POST https://www.googleapis.com/compute/v1/projects/[project-id]/global/instanceTemplates
{
  "name":"l7-ilb-backend-template",
  "properties":{
     "machineType":"n1-standard-1",
     "tags":{
        "items":[
           "allow-ssh",
           "load-balanced-backend"
        ]
     },
     "metadata":{
        "kind":"compute#metadata",
        "items":[
          {
              "key":"startup-script",
              "value":"#! /bin/bash\napt-get update\napt-get install apache2 -y\na2ensite default-ssl\na2enmod ssl\nvm_hostname=\"$(curl -H \"Metadata-Flavor:Google\" \\\nhttp://169.254.169.254/computeMetadata/v1/instance/name)\"\necho \"Page served from: $vm_hostname\" | \\\ntee /var/www/html/index.html\nsystemctl restart apache2"
          }
        ]
     },
     "networkInterfaces":[
       {
           "network":"projects/[project-id]/global/networks/lb-network",
           "subnetwork":"regions/us-west1/subnetworks/backend-subnet",
           "accessConfigs":[  
             {  
                 "type":"ONE_TO_ONE_NAT"
             }
           ]
       }
     ],
     "disks":[  
       {  
           "index":0,
           "boot":true,
           "initializeParams":{  
              "sourceImage":"projects/debian-cloud/global/images/family/debian-9"
           },
           "autoDelete":true
       }
     ]
  }
}

Create a managed instance group in each zone with the instanceGroupManagers.insert method, replacing [project-id] with your project ID.

POST https://www.googleapis.com/compute/v1/projects/[project-id]/zones/{zone}/instanceGroupManagers
{
  "name": "l7-ilb-backend-example",
  "zone": "projects/[project-id]/zones/us-west1-a",
  "instanceTemplate": "projects/[project-id]/global/instanceTemplates/l7-ilb-backend-template",
  "baseInstanceName": "l7-ilb-backend-example",
  "targetSize": 2
}

Configuring the load balancer

This example shows you how to create the following internal HTTP(S) load balancer resources:

  • HTTP health check
  • Backend service with a managed instance group as the backend
  • A URL map
    • Make sure to refer to a regional URL map if a region is defined for the target HTTP(S) proxy. A regional URL map routes requests to a regional backend service based on rules that you define for the host and path of an incoming URL. A regional URL map can be referenced by a regional target proxy rule in the same region only.
  • SSL certificate (for HTTPS)
  • Target proxy
  • Forwarding rule

For the forwardng rule's IP address, use the backend-subnet. If you try to use the proxy-only subnet, forwarding rule creation fails.

Console

Select a load balancer type

  1. Go to the Load balancing page in the Google Cloud Console.
    Go to the Load balancing page
  2. Under HTTP(S) Load Balancing, click Start configuration.
  3. Select Only between my VMs. This setting means that the load balancer is internal.
  4. Click Continue.

Prepare the load balancer

  1. For the Name of the load balancer, enter l7-ilb-map.
  2. For the Region, select us-west1.
  3. For the Network, select lb-network.
  4. Keep the window open to continue.

Reserve a proxy-only subnet

For Internal HTTP(S) Load Balancing, reserve a proxy subnet:

  1. Click Reserve a Subnet.
  2. For the Name, enter proxy-only-subnet.
  3. For the IP address range, enter 10.129.0.0/26.
  4. Click Add.

Configure the backend service

  1. Click Backend configuration.
  2. From the Create or select backend services menu, select Create a backend service.
  3. Set the Name of the backend service to l7-ilb-backend-service.
  4. Set the Backend type to instance groups.
  5. In the New backend section:
    1. Set the Instance group to l7-ilb-backend-example.
    2. Set the Port numbers to 80.
    3. Set the Balancing mode to Utilization.
    4. Click Done.
  6. In the Health check section, choose Create a health check with the following parameters:
    1. Name: l7-ilb-basic-check
    2. Protocol: HTTP
    3. Port: 80
    4. Click Save and Continue.
  7. Click Create.

Configure the URL map

Click Host and path rules. Ensure that the l7-ilb-backend-service is the only backend service for any unmatched host and any unmatched path.

For information about traffic management, see Setting up traffic management.

Configure the frontend

For HTTP:

  1. Click Frontend configuration.
  2. Click Add frontend IP and port.
  3. Set the Name to l7-ilb-forwarding-rule.
  4. Set the Protocol to HTTP.
  5. Set the Subnetwork to backend-subnet.
  6. Under Internal IP, select Reserve a static internal IP address.
  7. In the panel that appears, provide the following details:
    1. Name: l7-ilb-ip
    2. In the Static IP address section, select Let me choose.
    3. In the Custom IP address section, enter 10.1.2.99.
    4. Click Reserve.
  8. Set the Port to 80.
  9. Click Done.

For HTTPS:

If you are using HTTPS between the client and the load balancer, you need one or more SSL certificate resources to configure the proxy. See SSL Certificates for information on how to create SSL certificate resources. Google-managed certificates aren't currently supported with internal HTTP(S) load balancers.

  1. Click Frontend configuration.
  2. Click Add frontend IP and port.
  3. In the Name field, enter l7-ilb-forwarding-rule.
  4. In the Protocol field, select HTTPS (includes HTTP/2).
  5. Set the Subnetwork to backend-subnet.
  6. Under Internal IP, select Reserve a static internal IP address.
  7. In the panel that appears provide the following details:
    1. Name: l7-ilb-ip
    2. In the Static IP address section, select Let me choose.
    3. In the Custom IP address section, enter 10.1.2.99.
    4. Click Reserve.
  8. Ensure that the Port is set to 443, to allow HTTPS traffic.
  9. Click the Certificate drop-down list.
    1. If you already have a self-managed SSL certificate resource you want to use as the primary SSL certificate, select it from the drop-down menu.
    2. Otherwise, select Create a new certificate.
      1. Fill in a Name of l7-ilb-cert.
      2. In the appropriate fields upload your PEM-formatted files:
        • Public key certificate
        • Certificate chain
        • Private key
      3. Click Create.
  10. To add certificate resources in addition to the primary SSL certificate resource:
    1. Click Add certificate.
    2. Select a certificate from the Certificates list or click Create a new certificate and follow the instructions above.
  11. Click Done.

Complete the configuration

Click Create.

gcloud

  1. Define the HTTP health check with the gcloud compute health-checks create http command.

    gcloud compute health-checks create http l7-ilb-basic-check \
       --region=us-west1 \
       --use-serving-port
    
  2. Define the backend service with the gcloud compute backend-services create command.

    gcloud compute backend-services create l7-ilb-backend-service \
      --load-balancing-scheme=INTERNAL_MANAGED \
      --protocol=HTTP \
      --health-checks=l7-ilb-basic-check \
      --health-checks-region=us-west1 \
      --region=us-west1
    
  3. Add backends to the backend service with the gcloud compute backend-services add-backend command.

    gcloud compute backend-services add-backend l7-ilb-backend-service \
      --balancing-mode=UTILIZATION \
      --instance-group=l7-ilb-backend-example \
      --instance-group-zone=us-west1-a \
      --region=us-west1
    
  4. Create the URL map with the gcloud compute url-maps create command.

    gcloud compute url-maps create l7-ilb-map \
      --default-service=l7-ilb-backend-service \
      --region=us-west1
    
  5. Create the target proxy.

    For HTTP:

    For an internal HTTP load balancer, create the target proxy with the gcloud compute target-http-proxies create command.

    gcloud compute target-http-proxies create l7-ilb-proxy \
      --url-map=l7-ilb-map \
      --url-map-region=us-west1 \
      --region=us-west1
    

    For HTTPS:

    See SSL Certificates for information on how to create SSL certificate resources. Google-managed certificates aren't currently supported with internal HTTP(S) load balancers.

    Assign your filepaths to variable names.

    export LB_CERT=path to PEM-formatted file
    
    export LB_PRIVATE_KEY=path to PEM-formatted file
    

    Create a regional SSL certificate using the gcloud compute ssl-certificates create command.

    gcloud compute ssl-certificates create l7-ilb-cert \
      --certificate=$LB_CERT \
      --private-key=$LB_PRIVATE_KEY \
      --region=us-west1
    

    Use the regional SSL certificate to create a target proxy with the gcloud compute target-https-proxies create command.

    gcloud compute target-https-proxies create

    gcloud compute target-https-proxies create l7-ilb-proxy \
      --url-map=l7-ilb-map \
      --region=us-west1 \
      --ssl-certificates=l7-ilb-cert
    
  6. Create the forwarding rule.

    For custom networks, you must reference the subnet in the forwarding rule. Note that this is the VM subnet, not the proxy subnet.

    For HTTP:

    Use the gcloud compute forwarding-rules create command with the correct flags.

    gcloud compute forwarding-rules create l7-ilb-forwarding-rule \
      --load-balancing-scheme=INTERNAL_MANAGED \
      --network=lb-network \
      --subnet=backend-subnet \
      --address=10.1.2.99 \
      --ports=80 \
      --region=us-west1 \
      --target-http-proxy=l7-ilb-proxy \
      --target-http-proxy-region=us-west1
    

    For HTTPS:

    Create the forwarding rule with the gcloud compute forwarding-rules create command with the correct flags.

    gcloud compute forwarding-rules create l7-ilb-forwarding-rule \
      --load-balancing-scheme=INTERNAL_MANAGED \
      --network=lb-network \
      --subnet=backend-subnet \
      --address=10.1.2.99 \
      --ports=443 \
      --region=us-west1 \
      --target-https-proxy=l7-ilb-proxy \
      --target-https-proxy-region=us-west1
    

api

Create the health check by making a POST request to the regionHealthChecks.insert method, replacing [project-id] with your project ID.

POST https://compute.googleapis.com/compute/v1/projects/[project-id]/regions/{region}/healthChecks
{
  "name": "l7-ilb-basic-check",
  "type": "HTTP",
  "httpHealthCheck": {
    "portSpecification": "USE_SERVING_PORT"
  }
}

Create the regional backend service by making a POST request to the regionBackendServices.insert method, replacing [project-id] with your project ID.

POST https://www.googleapis.com/compute/v1/projects/[project-id]/regions/us-west1/backendServices
{
  "name": "l7-ilb-backend-service",
  "backends": [
    {
      "group": "projects/[project-id]/zones/us-west1-a/instanceGroups/l7-ilb-backend-example",
      "balancingMode": "UTILIZATION"
    }
  ],
  "healthChecks": [
    "projects/[project-id]/regions/us-west1/healthChecks/l7-ilb-basic-check"
  ],
  "loadBalancingScheme": "INTERNAL_MANAGED"
}

Create the URL map by making a POST request to the regionUrlMaps.insert method, replacing [project-id] with your project ID.

POST https://compute.googleapis.com/compute/v1/projects/[project-id]/regions/us-west1/urlMaps
{
  "name": "l7-ilb-map",
  "defaultService": "projects/[project-id]/regions/us-west1/backendServices/l7-ilb-backend-service"
}

Create the target HTTP proxy by making a POST request to the regionTargetHttpProxies.insert method, replacing [project-id] with your project ID.

POST https://www.googleapis.com/compute/v1/projects/[project-id]/regions/us-west1/targetHttpProxy
{
  "name": "l7-ilb-proxy",
  "urlMap": "projects/[project-id]/global/urlMaps/l7-ilb-map",
  "region": "us-west1"
}

Create the forwarding rule by making a POST request to the forwardingRules.insert method, replacing [project-id] with your project ID.

POST https://www.googleapis.com/compute/v1/projects/[project-id]/regions/us-west1/forwardingRules
{
  "name": "l7-ilb-forwarding-rule",
  "IPAddress": "10.1.2.99",
  "IPProtocol": "TCP",
  "portRange": "80-80",
  "target": "projects/[project-id]/regions/us-west1/targetHttpProxies/l7-ilb-proxy",
  "loadBalancingScheme": "INTERNAL_MANAGED",
  "subnetwork": "projects/[project-id]/regions/us-west1/subnetworks/backend-subnet",
  "network": "projects/[project-id]/global/networks/lb-network",
  "networkTier": "PREMIUM",
}

Testing

Creating a VM instance to test connectivity

gcloud compute instances create l7-ilb-client-us-west1-a \
    --image-family=debian-9 \
    --image-project=debian-cloud \
    --network=lb-network \
    --subnet=backend-subnet \
    --zone=us-west1-a \
    --tags=allow-ssh

Testing the load balancer

Log in to the instance that you just created and test that HTTP(S) services on the backends are reachable via the internal HTTP(S) load balancer's forwarding rule IP address, and traffic is being load balanced across the backend instances.

Connecting via SSH to each client instance

gcloud compute ssh l7-ilb-client-us-west1-a \
    --zone=us-west1-a

Verifying that the IP is serving its hostname

curl 10.1.2.99

For HTTPS testing, replace curl with:

curl -k -s 'https://test.example.com:443' --connect-to test.example.com:443:10.1.2.99:443

The -k flag causes curl to skip certificate validation.

Running 100 requests and confirming that they are load balanced

For HTTP:

{
  RESULTS=
  for i in {1..100}
  do
      RESULTS="$RESULTS:$(curl --silent 10.1.2.99)"
  done
  echo "***"
  echo "*** Results of load-balancing to 10.1.2.99: "
  echo "***"
  echo "$RESULTS" | tr ':' '\n' | grep -Ev "^$" | sort | uniq -c
  echo
}

For HTTPS:

{
  RESULTS=
  for i in {1..100}
  do
      RESULTS="$RESULTS:$(curl -k -s 'https://test.example.com:443' --connect-to test.example.com:443:10.1.2.99:443)"
  done
  echo "***"
  echo "*** Results of load-balancing to 10.1.2.99: "
  echo "***"
  echo "$RESULTS" | tr ':' '\n' | grep -Ev "^$" | sort | uniq -c
  echo
}

What's next