This document provides instructions for configuring Internal HTTP(S) Load Balancing for your services that run on Compute Engine VMs.
To configure load balancing for your services running in GKE pods, see Container-native load balancing with standalone NEGs and the Attaching an internal HTTP(S) load balancer to standalone NEGs section.
To configure load balancing to access Google APIs and services using Private Service Connect, see Configuring Private Service Connect with consumer HTTP(S) service controls.
The setup for Internal HTTP(S) Load Balancing has two parts:
- Performing prerequisite tasks, such as ensuring that required accounts have the correct permissions and preparing the Virtual Private Cloud (VPC) network.
- Setting up the load balancer resources.
Before following this guide, familiarize yourself with the following:
- Internal HTTP(S) Load Balancing overview, including the Limitations section
- VPC firewall rules overview
Permissions
To follow this guide, you must be able to create instances and modify a network in a project. You must be either a project owner or editor, or you must have all of the following Compute Engine IAM roles.
Task | Required role |
---|---|
Create networks, subnets, and load balancer components | Network Admin |
Add and remove firewall rules | Security Admin |
Create instances | Instance Admin |
For more information, see the following guides:
Setup overview
You can configure Internal HTTP(S) Load Balancing as described in the following high-level configuration flow. The numbered steps refer to the numbers in the diagram.
As shown in the diagram, this example creates an internal HTTP(S) load balancer in a
VPC network in region us-west1
, with one backend service
and two backend groups.
The diagram shows the following:
A VPC network with two subnets:
One subnet is used for backends (instance groups and NEGs) and the forwarding rule. Its primary IP address range is
10.1.2.0/24
.One subnet is a proxy-only subnet in the
us-west1
region. You must create one proxy-only subnet in each region of a VPC network where you use internal HTTP(S) load balancers. The region's proxy-only subnet is shared among all internal HTTP(S) load balancers in the region. Source addresses of packets sent from the internal HTTP(S) load balancer to your service's backends are allocated from the proxy-only subnet. In this example, the proxy-only subnet for the region has a primary IP address range of10.129.0.0/23
, which is the recommended subnet size. For more information, see Proxy-only subnets.
A firewall rule that permits proxy-only subnet traffic flows in your network. This means adding one rule that allows TCP port
80
,443
, and8080
traffic from10.129.0.0/23
(the range of the proxy-only subnet in this example). Another firewall rule for the health check probes.Backend instances. This example diagram demonstrates the following backend deployments:
- Compute Engine VMs
- Google Kubernetes Engine (GKE) backends added to standalone network endpoint groups (NEGs)
Instance groups and NEGs:
- Managed or unmanaged instance groups for Compute Engine VM deployments
- NEGs for GKE deployments
In each zone, you can have a combination of backend group types based on the requirements of your deployment.
A regional health check that reports the readiness of your backends.
A regional backend service that monitors the usage and health of backends.
A regional URL map that parses the URL of a request and forwards requests to specific backend services based on the host and path of the request URL.
A regional target HTTP or HTTPS proxy, which receives a request from the user and forwards it to the URL map. For HTTPS, configure a regional SSL certificate resource. The target proxy uses the SSL certificate to decrypt SSL traffic if you configure HTTPS load balancing. The target proxy can forward traffic to your instances by using HTTP or HTTPS.
A forwarding rule, which has the internal IP address of your load balancer, to forward each incoming request to the target proxy.
The internal IP address associated with the forwarding rule can come from any subnet (in the same network and region) with its
--purpose
flag set toPRIVATE
. Note that:- The IP address can (but does not need to) come from the same subnet as the backend instance groups.
- The IP address must not come from a reserved proxy-only subnet that has
its
--purpose
flag set toREGIONAL_MANAGED_PROXY
.
Configuring the network and subnets
You need a VPC network with two subnets: one for the load balancer's backends and the other for the load balancer's proxies. An internal HTTP(S) load balancer is regional. Traffic within the VPC network is routed to the load balancer if the traffic's source is in a subnet in the same region as the load balancer.
This example uses the following VPC network, region, and subnets:
Network. The network is a custom-mode VPC network named
lb-network
.Subnet for backends. A subnet named
backend-subnet
in theus-west1
region uses10.1.2.0/24
for its primary IP range.Subnet for proxies. A subnet named
proxy-only-subnet
in theus-west1
region uses10.129.0.0/23
for its primary IP range.
Configuring the network and subnet for backends
Console
- Go to the VPC network page in the Google Cloud console.
Go to the VPC networks page - Click Create VPC network.
- For the Name, enter
lb-network
. - In the Subnets section:
- Set the Subnet creation mode to Custom.
- In the New subnet section, enter the following information:
- Name:
backend-subnet
- Region:
us-west1
- IP address range:
10.1.2.0/24
- Name:
- Click Done.
- Click Create.
gcloud
Create the custom VPC network with the
gcloud compute networks create
command:gcloud compute networks create lb-network --subnet-mode=custom
Create a subnet in the
lb-network
network in theus-west1
region with thegcloud compute networks subnets create
command:gcloud compute networks subnets create backend-subnet \ --network=lb-network \ --range=10.1.2.0/24 \ --region=us-west1
API
Make a POST
request to the
networks.insert
method,
replacing PROJECT_ID
with your project ID.
POST https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/global/networks { "routingConfig": { "routingMode": "REGIONAL" }, "name": "lb-network", "autoCreateSubnetworks": false }
Make a POST
request to the
subnetworks.insert
method,
replacing PROJECT_ID
with your project ID.
POST https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/regions/us-west1/subnetworks { "name": "backend-subnet", "network": "projects/PROJECT_ID/global/networks/lb-network", "ipCidrRange": "10.1.2.0/24", "region": "projects/PROJECT_ID/regions/us-west1", }
Configuring the proxy-only subnet
This proxy-only subnet is for all internal HTTP(S) load balancers in the us-west1
region.
Console
If you're using the Google Cloud console, you can wait and create the proxy-only subnet later on the Load balancing page.
If you want to create the proxy-only subnet now, use the following steps:
- In the Google Cloud console, go to the VPC networks page.
Go to the VPC networks page - Click the name of the Shared VPC network:
lb-network
. - Click Add subnet.
- For the Name, enter
proxy-only-subnet
. - For the Region, select
us-west1
. - Set Purpose to Regional Managed Proxy.
- For the IP address range, enter
10.129.0.0/23
. - Click Add.
gcloud
Create the proxy-only subnet with the gcloud compute networks subnets
create
command.
gcloud compute networks subnets create proxy-only-subnet \ --purpose=REGIONAL_MANAGED_PROXY \ --role=ACTIVE \ --region=us-west1 \ --network=lb-network \ --range=10.129.0.0/23
API
Create the proxy-only subnet with the
subnetworks.insert
method, replacing PROJECT_ID
with your project ID.
POST https://compute.googleapis.com/compute/projects/PROJECT_ID/regions/us-west1/subnetworks { "name": "proxy-only-subnet", "ipCidrRange": "10.129.0.0/23", "network": "projects/PROJECT_ID/global/networks/lb-network", "region": "projects/PROJECT_ID/regions/us-west1", "purpose": "REGIONAL_MANAGED_PROXY", "role": "ACTIVE" }
Configuring firewall rules
This example uses the following firewall rules:
fw-allow-ssh
. An ingress rule, applicable to the instances being load balanced, that allows incoming SSH connectivity on TCP port22
from any address. You can choose a more restrictive source IP range for this rule; for example, you can specify just the IP ranges of the system from which you initiate SSH sessions. This example uses the target tagallow-ssh
.fw-allow-health-check
. An ingress rule, applicable to the instances being load balanced, that allows all TCP traffic from the Google Cloud health checking systems (in130.211.0.0/22
and35.191.0.0/16
). This example uses the target tagload-balanced-backend
.fw-allow-proxies
. An ingress rule, applicable to the instances being load balanced, that allows TCP traffic on ports80
,443
, and8080
from the internal HTTP(S) load balancer's managed proxies. This example uses the target tagload-balanced-backend
.
Without these firewall rules, the default deny ingress rule blocks incoming traffic to the backend instances.
The target tags define the backend instances. Without the target tags, the firewall rules apply to all of your backend instances in the VPC network. When you create the backend VMs, make sure to include the specified target tags, as shown in Creating a managed instance group.
Console
- Go to the Firewall rules page in the Google Cloud console.
Go to the Firewall rules page - Click Create firewall rule to create the rule to allow incoming
SSH connections:
- Name:
fw-allow-ssh
- Network:
lb-network
- Direction of traffic: Ingress
- Action on match: Allow
- Targets: Specified target tags
- Target tags:
allow-ssh
- Source filter: IPv4 ranges
- Source IPv4 ranges:
0.0.0.0/0
- Protocols and ports:
- Choose Specified protocols and ports.
- Select the tcp checkbox, and then enter
22
for the port number.
- Name:
- Click Create.
- Click Create firewall rule a second time to create the rule to allow
Google Cloud health checks:
- Name:
fw-allow-health-check
- Network:
lb-network
- Direction of traffic: Ingress
- Action on match: Allow
- Targets: Specified target tags
- Target tags:
load-balanced-backend
- Source filter: IPv4 ranges
- Source IPv4 ranges:
130.211.0.0/22
and35.191.0.0/16
- Protocols and ports:
- Choose Specified protocols and ports.
- Select the tcp checkbox, and then enter
80
for the port number. As a best practice, limit this rule to just the protocols and ports that match those used by your health check. If you usetcp:80
for the protocol and port, Google Cloud can use HTTP on port80
to contact your VMs, but it cannot use HTTPS on port443
to contact them.
- Name:
- Click Create.
- Click Create firewall rule a third time to create the rule to allow
the load balancer's proxy servers to connect the backends:
- Name:
fw-allow-proxies
- Network:
lb-network
- Direction of traffic: Ingress
- Action on match: Allow
- Targets: Specified target tags
- Target tags:
load-balanced-backend
- Source filter: IPv4 ranges
- Source IPv4 ranges:
10.129.0.0/23
- Protocols and ports:
- Choose Specified protocols and ports.
- Select the tcp checkbox, and then enter
80, 443, 8080
for the port numbers.
- Name:
- Click Create.
gcloud
Create the
fw-allow-ssh
firewall rule to allow SSH connectivity to VMs with the network tagallow-ssh
. When you omitsource-ranges
, Google Cloud interprets the rule to mean any source.gcloud compute firewall-rules create fw-allow-ssh \ --network=lb-network \ --action=allow \ --direction=ingress \ --target-tags=allow-ssh \ --rules=tcp:22
Create the
fw-allow-health-check
rule to allow Google Cloud health checks. This example allows all TCP traffic from health check probers; however, you can configure a narrower set of ports to meet your needs.gcloud compute firewall-rules create fw-allow-health-check \ --network=lb-network \ --action=allow \ --direction=ingress \ --source-ranges=130.211.0.0/22,35.191.0.0/16 \ --target-tags=load-balanced-backend \ --rules=tcp
Create the
fw-allow-proxies
rule to allow the internal HTTP(S) load balancer's proxies to connect to your backends. Setsource-ranges
to the allocated ranges of your proxy-only subnet, for example,10.129.0.0/23
.gcloud compute firewall-rules create fw-allow-proxies \ --network=lb-network \ --action=allow \ --direction=ingress \ --source-ranges=source-range \ --target-tags=load-balanced-backend \ --rules=tcp:80,tcp:443,tcp:8080
API
Create the fw-allow-ssh
firewall rule by making a POST
request to
the firewalls.insert
method, replacing PROJECT_ID
with your project ID.
POST https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/global/firewalls { "name": "fw-allow-ssh", "network": "projects/PROJECT_ID/global/networks/lb-network", "sourceRanges": [ "0.0.0.0/0" ], "targetTags": [ "allow-ssh" ], "allowed": [ { "IPProtocol": "tcp", "ports": [ "22" ] } ], "direction": "INGRESS" }
Create the fw-allow-health-check
firewall rule by making a POST
request to
the firewalls.insert
method, replacing PROJECT_ID
with your project ID.
POST https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/global/firewalls { "name": "fw-allow-health-check", "network": "projects/PROJECT_ID/global/networks/lb-network", "sourceRanges": [ "130.211.0.0/22", "35.191.0.0/16" ], "targetTags": [ "load-balanced-backend" ], "allowed": [ { "IPProtocol": "tcp" } ], "direction": "INGRESS" }
Create the fw-allow-proxies
firewall rule to allow TCP traffic within the
proxy subnet for the
firewalls.insert
method, replacing PROJECT_ID
with your project ID.
POST https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/global/firewalls { "name": "fw-allow-proxies", "network": "projects/PROJECT_ID/global/networks/lb-network", "sourceRanges": [ "10.129.0.0/23" ], "targetTags": [ "load-balanced-backend" ], "allowed": [ { "IPProtocol": "tcp", "ports": [ "80" ] }, { "IPProtocol": "tcp", "ports": [ "443" ] }, { "IPProtocol": "tcp", "ports": [ "8080" ] } ], "direction": "INGRESS" }
Configuring Internal HTTP(S) Load Balancing with a VM-based service
This section shows the configuration required for services that run on Compute Engine VMs. Client VMs connect to the IP address and port that you configure in the forwarding rule. When your client applications send traffic to this IP address and port, their requests are forwarded to your backend virtual machines (VMs) according to your internal HTTP(S) load balancer's URL map.
The example on this page explicitly sets a reserved internal IP address for the internal HTTP(S) load balancer's forwarding rule, rather than allowing an ephemeral internal IP address to be allocated. As a best practice, we recommend reserving IP addresses for forwarding rules.
For the forwarding rule's IP address, use the backend-subnet
. If you
try to use the proxy-only
subnet, forwarding rule
creation fails.
Creating a managed instance group
This section shows how to create a template and a managed instance group. The managed instance group provides VM instances running the backend servers of an example internal HTTP(S) load balancer. For your instance group, you can define an HTTP service and map a port name to the relevant port. The load balancing service forwards traffic to the named ports Traffic from clients is load balanced to these backend servers. For demonstration purposes, backends serve their own hostnames.
Cloud console
Create an instance template. In the Google Cloud console, go to the Instance templates page.
- Click Create instance template.
- For Name, enter
l7-ilb-backend-template
. - Ensure that the Boot disk is set to a Debian image, such as
Debian GNU/Linux 9 (stretch). These instructions use commands that
are only available on Debian, such as
apt-get
. Under Management, security, disks, networking, sole tenancy, on the Management tab, insert the following script into the Startup script field.
#! /bin/bash sudo apt-get update sudo apt-get install apache2 -y sudo a2ensite default-ssl sudo a2enmod ssl sudo vm_hostname="$(curl -H "Metadata-Flavor:Google" \ http://169.254.169.254/computeMetadata/v1/instance/name)" sudo echo "Page served from: $vm_hostname" | \ tee /var/www/html/index.html
Under Networking, select
lb-network
as the Network, and for the Subnet, selectbackend-subnet
.Add the following network tags:
allow-ssh
andload-balanced-backend
.Click Create.
Create a managed instance group. Go to the Instance groups page in the Google Cloud console.
- Click Create instance group.
- Select New managed instance group (stateless). For more information, see Stateless or stateful MIGs.
- For the Name, enter
l7-ilb-backend-example
. - Under Location, select Single zone.
- For the Region, select
us-west1
. - For the Zone, select
us-west1-a
. - Under Instance template, select
l7-ilb-backend-template
. Specify the number of instances that you want to create in the group.
For this example, specify the following options under Autoscaling:
- For Autoscaling mode, select
Off:do not autoscale
. - For Maximum number of instances, enter
2
.
Optionally, in the Autoscaling section of the UI, you can configure the instance group to automatically add or remove instances based on instance CPU usage.
- For Autoscaling mode, select
Click Create.
gcloud
The gcloud
instructions in this guide assume that you are using Cloud
Shell or another environment with bash installed.
Create a VM instance template with HTTP server with the
gcloud compute instance-templates create
command.gcloud compute instance-templates create `l7-ilb-backend-template` \ --region=us-west1 \ --network=lb-network \ --subnet=backend-subnet \ --tags=allow-ssh,load-balanced-backend \ --image-family=debian-9 \ --image-project=debian-cloud \ --metadata=startup-script='#! /bin/bash sudo apt-get update sudo apt-get install apache2 -y sudo a2ensite default-ssl sudo a2enmod ssl sudo vm_hostname="$(curl -H "Metadata-Flavor:Google" \ http://169.254.169.254/computeMetadata/v1/instance/name)" sudo echo "Page served from: $vm_hostname" | \ tee /var/www/html/index.html sudo systemctl restart apache2'
Create a managed instance group in the zone with the
gcloud compute instance-groups managed create
command.gcloud compute instance-groups managed create l7-ilb-backend-example \ --zone=us-west1-a \ --size=2 \ --template=l7-ilb-backend-template
API
Create the instance template with the
instanceTemplates.insert
method, replacing PROJECT_ID
with your project ID.
POST https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/global/instanceTemplates { "name":"l7-ilb-backend-template", "properties":{ "machineType":"e2-standard-2", "tags":{ "items":[ "allow-ssh", "load-balanced-backend" ] }, "metadata":{ "kind":"compute#metadata", "items":[ { "key":"startup-script", "value":"#! /bin/bash\nsudo apt-get update\nsudo apt-get install apache2 -y\nsudo a2ensite default-ssl\nsudo a2enmod ssl\nsudo vm_hostname=\"$(curl -H \"Metadata-Flavor:Google\" \\\nhttp://169.254.169.254/computeMetadata/v1/instance/name)\"\n sudo echo \"Page served from: $vm_hostname\" | \\\ntee /var/www/html/index.html\nsudo systemctl restart apache2" } ] }, "networkInterfaces":[ { "network":"projects/PROJECT_ID/global/networks/lb-network", "subnetwork":"regions/us-west1/subnetworks/backend-subnet", "accessConfigs":[ { "type":"ONE_TO_ONE_NAT" } ] } ], "disks":[ { "index":0, "boot":true, "initializeParams":{ "sourceImage":"projects/debian-cloud/global/images/family/debian-9" }, "autoDelete":true } ] } }
Create a managed instance group in each zone with the
instanceGroupManagers.insert
method, replacing PROJECT_ID
with your project ID.
POST https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/zones/{zone}/instanceGroupManagers { "name": "l7-ilb-backend-example", "zone": "projects/PROJECT_ID/zones/us-west1-a", "instanceTemplate": "projects/PROJECT_ID/global/instanceTemplates/l7-ilb-backend-template", "baseInstanceName": "l7-ilb-backend-example", "targetSize": 2 }
Configuring the load balancer
This example shows you how to create the following internal HTTP(S) load balancer resources:
- HTTP health check
- Backend service with a managed instance group as the backend
- A URL map
- Make sure to refer to a regional URL map if a region is defined for the target HTTP(S) proxy. A regional URL map routes requests to a regional backend service based on rules that you define for the host and path of an incoming URL. A regional URL map can be referenced by a regional target proxy rule in the same region only.
- SSL certificate (for HTTPS)
- Target proxy
- Forwarding rule
For the forwarding rule's IP address, use the backend-subnet
. If you
try to use the proxy-only
subnet, forwarding rule
creation fails.
Proxy availability
Sometimes Google Cloud regions don't have enough proxy capacity for a new load balancer. If this happens, the Google Cloud console provides a proxy availability warning message when you are creating your load balancer. To resolve this issue, you can do one of the following:
- Select a different region for your load balancer. This can be a practical option if you have backends in another region.
- Select a VPC network that already has an allocated proxy-only subnet.
Wait for the capacity issue to be resolved.
Cloud console
Select a load balancer type
- Go to the Load balancing page in the Google Cloud console.
Go to the Load balancing page - Under HTTP(S) Load Balancing, click Start configuration.
- Under Internet facing or internal only, select Only between my VMs. This setting means that the load balancer is internal.
- Click Continue.
Prepare the load balancer
- For the Name of the load balancer, enter
l7-ilb-map
. - For the Region, select
us-west1
. - For the Network, select
lb-network
. - Keep the window open to continue.
Reserve a proxy-only subnet
For Internal HTTP(S) Load Balancing, reserve a proxy subnet:
- Click Reserve a Subnet.
- For the Name, enter
proxy-only-subnet
. - For the IP address range, enter
10.129.0.0/23
. - Click Add.
Configure the backend service
- Click Backend configuration.
- From the Create or select backend services menu, select Create a backend service.
- Set the Name of the backend service to l7-ilb-backend-service.
- Set the Backend type to instance groups.
- In the New backend section:
- Set the Instance group to l7-ilb-backend-example.
- Set the Port numbers to
80
. - Set the Balancing mode to Utilization.
- Click Done.
- In the Health check section, choose Create a health check with the
following parameters:
- Name: l7-ilb-basic-check
- Protocol: HTTP
- Port:
80
- Click Save and Continue.
- Click Create.
Configure the URL map
Click Host and path rules. Ensure that the l7-ilb-backend-service is the only backend service for any unmatched host and any unmatched path.
For information about traffic management, see Setting up traffic management.
Configure the frontend
For HTTP:
- Click Frontend configuration.
- Click Add frontend IP and port.
- Set the Name to l7-ilb-forwarding-rule.
- Set the Protocol to HTTP.
- Set the Subnetwork to backend-subnet.
- Under Internal IP, select Reserve a static internal IP address.
- In the panel that appears, provide the following details:
- Name: l7-ilb-ip
- In the Static IP address section, select Let me choose.
- In the Custom IP address section, enter
10.1.2.99
. - Click Reserve.
- Set the Port to
80
. - Click Done.
For HTTPS:
If you are using HTTPS between the client and the load balancer, you need one or more SSL certificate resources to configure the proxy. For information about how to create SSL certificate resources, see SSL certificates. Google-managed certificates aren't currently supported with internal HTTP(S) load balancers.
- Click Frontend configuration.
- Click Add frontend IP and port.
- In the Name field, enter
l7-ilb-forwarding-rule
. - In the Protocol field, select
HTTPS (includes HTTP/2)
. - Set the Subnetwork to backend-subnet.
- Under Internal IP, select Reserve a static internal IP address.
- In the panel that appears provide the following details:
- Name: l7-ilb-ip
- In the Static IP address section, select Let me choose.
- In the Custom IP address section, enter
10.1.2.99
. - Click Reserve.
- Ensure that the Port is set to
443
, to allow HTTPS traffic. - Click the Certificate drop-down list.
- If you already have a self-managed SSL certificate resource you want to use as the primary SSL certificate, select it from the drop-down menu.
- Otherwise, select Create a new certificate.
- Fill in a Name of
l7-ilb-cert
. - In the appropriate fields upload your PEM-formatted files:
- Public key certificate
- Certificate chain
- Private key
- Click Create.
- Fill in a Name of
- To add certificate resources in addition to
the primary SSL certificate resource:
- Click Add certificate.
- Select a certificate from the Certificates list or click Create a new certificate and follow the instructions above.
- Click Done.
Complete the configuration
Click Create.
gcloud
Define the HTTP health check with the
gcloud compute health-checks create http
command.gcloud compute health-checks create http l7-ilb-basic-check \ --region=us-west1 \ --use-serving-port
Define the backend service with the
gcloud compute backend-services create
command.gcloud compute backend-services create l7-ilb-backend-service \ --load-balancing-scheme=INTERNAL_MANAGED \ --protocol=HTTP \ --health-checks=l7-ilb-basic-check \ --health-checks-region=us-west1 \ --region=us-west1
Add backends to the backend service with the
gcloud compute backend-services add-backend
command.gcloud compute backend-services add-backend l7-ilb-backend-service \ --balancing-mode=UTILIZATION \ --instance-group=l7-ilb-backend-example \ --instance-group-zone=us-west1-a \ --region=us-west1
Create the URL map with the
gcloud compute url-maps create
command.gcloud compute url-maps create l7-ilb-map \ --default-service=l7-ilb-backend-service \ --region=us-west1
Create the target proxy.
For HTTP:
For an internal HTTP load balancer, create the target proxy with the
gcloud compute target-http-proxies create
command.gcloud compute target-http-proxies create l7-ilb-proxy \ --url-map=l7-ilb-map \ --url-map-region=us-west1 \ --region=us-west1
For HTTPS:
For information about how to create SSL certificate resources, see SSL certificates. Google-managed certificates aren't currently supported with internal HTTP(S) load balancers.
Assign your filepaths to variable names.
export LB_CERT=path to PEM-formatted file
export LB_PRIVATE_KEY=path to PEM-formatted file
Create a regional SSL certificate using the
gcloud compute ssl-certificates create
command.gcloud compute ssl-certificates create l7-ilb-cert \ --certificate=$LB_CERT \ --private-key=$LB_PRIVATE_KEY \ --region=us-west1
Use the regional SSL certificate to create a target proxy with the
gcloud compute target-https-proxies create
command.gcloud compute target-https-proxies create l7-ilb-proxy \ --url-map=l7-ilb-map \ --region=us-west1 \ --ssl-certificates=l7-ilb-cert
Create the forwarding rule.
For custom networks, you must reference the subnet in the forwarding rule. Note that this is the VM subnet, not the proxy subnet.
For HTTP:
Use the
gcloud compute forwarding-rules create
command with the correct flags.gcloud compute forwarding-rules create l7-ilb-forwarding-rule \ --load-balancing-scheme=INTERNAL_MANAGED \ --network=lb-network \ --subnet=backend-subnet \ --address=10.1.2.99 \ --ports=80 \ --region=us-west1 \ --target-http-proxy=l7-ilb-proxy \ --target-http-proxy-region=us-west1
For HTTPS:
Create the forwarding rule with the
gcloud compute forwarding-rules create
command with the correct flags.gcloud compute forwarding-rules create l7-ilb-forwarding-rule \ --load-balancing-scheme=INTERNAL_MANAGED \ --network=lb-network \ --subnet=backend-subnet \ --address=10.1.2.99 \ --ports=443 \ --region=us-west1 \ --target-https-proxy=l7-ilb-proxy \ --target-https-proxy-region=us-west1
API
Create the health check by making a POST
request to the
regionHealthChecks.insert
method, replacing PROJECT_ID
with your project ID.
POST https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/regions/{region}/healthChecks { "name": "l7-ilb-basic-check", "type": "HTTP", "httpHealthCheck": { "portSpecification": "USE_SERVING_PORT" } }
Create the regional backend service by making a POST
request to the
regionBackendServices.insert
method, replacing PROJECT_ID
with your project ID.
POST https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/regions/us-west1/backendServices { "name": "l7-ilb-backend-service", "backends": [ { "group": "projects/PROJECT_ID/zones/us-west1-a/instanceGroups/l7-ilb-backend-example", "balancingMode": "UTILIZATION" } ], "healthChecks": [ "projects/PROJECT_ID/regions/us-west1/healthChecks/l7-ilb-basic-check" ], "loadBalancingScheme": "INTERNAL_MANAGED" }
Create the URL map by making a POST
request to the
regionUrlMaps.insert
method, replacing PROJECT_ID
with your project ID.
POST https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/regions/us-west1/urlMaps { "name": "l7-ilb-map", "defaultService": "projects/PROJECT_ID/regions/us-west1/backendServices/l7-ilb-backend-service" }
For HTTP:
Create the target HTTP proxy by making a POST
request to the
regionTargetHttpProxies.insert
method, replacing PROJECT_ID
with your project ID.
POST https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/regions/us-west1/targetHttpProxy { "name": "l7-ilb-proxy", "urlMap": "projects/PROJECT_ID/global/urlMaps/l7-ilb-map", "region": "us-west1" }
Create the forwarding rule by making a POST
request to the
forwardingRules.insert
method, replacing PROJECT_ID
with your project ID.
POST https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/regions/us-west1/forwardingRules { "name": "l7-ilb-forwarding-rule", "IPAddress": "10.1.2.99", "IPProtocol": "TCP", "portRange": "80-80", "target": "projects/PROJECT_ID/regions/us-west1/targetHttpProxies/l7-ilb-proxy", "loadBalancingScheme": "INTERNAL_MANAGED", "subnetwork": "projects/PROJECT_ID/regions/us-west1/subnetworks/backend-subnet", "network": "projects/PROJECT_ID/global/networks/lb-network", "networkTier": "PREMIUM" }
For HTTPS:
Read the certificate and private key files, and then create the SSL certificate. The following example shows how to do this with Python.
Create the target HTTPS proxy by making a POST
request to the
regionTargetHttpsProxies.insert
method, replacing PROJECT_ID
with your project ID.
POST https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/regions/us-west1/regionTargetHttpsProxy { "name": "l7-ilb-proxy", "urlMap": "projects/PROJECT_ID/regions/us-west1/urlMaps/l7-ilb-map", "sslCertificates": /projects/PROJECT_ID/regions/us-west1/sslCertificates/SSL_CERT_NAME }
Create the forwarding rule by making a POST
request to the
forwardingRules.insert
method, replacing PROJECT_ID
with your project ID.
POST https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/regions/us-west1/forwardingRules { "name": "l7-ilb-forwarding-rule", "IPAddress": "10.1.2.99", "IPProtocol": "TCP", "portRange": "80-80", "target": "projects/PROJECT_ID/regions/us-west1/targetHttpsProxies/l7-ilb-proxy", "loadBalancingScheme": "INTERNAL_MANAGED", "subnetwork": "projects/PROJECT_ID/regions/us-west1/subnetworks/backend-subnet", "network": "projects/PROJECT_ID/global/networks/lb-network", "networkTier": "PREMIUM", }
Testing
Creating a VM instance to test connectivity
gcloud compute instances create l7-ilb-client-us-west1-a \ --image-family=debian-9 \ --image-project=debian-cloud \ --network=lb-network \ --subnet=backend-subnet \ --zone=us-west1-a \ --tags=allow-ssh
Testing the load balancer
Log in to the instance that you just created and test that HTTP(S) services on the backends are reachable via the internal HTTP(S) load balancer's forwarding rule IP address, and traffic is being load balanced across the backend instances.
Connecting via SSH to each client instance
gcloud compute ssh l7-ilb-client-us-west1-a \ --zone=us-west1-a
Verifying that the IP is serving its hostname
curl 10.1.2.99
For HTTPS testing, replace curl
with:
curl -k -s 'https://test.example.com:443' --connect-to test.example.com:443:10.1.2.99:443
The -k
flag causes curl to skip certificate validation.
Running 100 requests and confirming that they are load balanced
For HTTP:
{ RESULTS= for i in {1..100} do RESULTS="$RESULTS:$(curl --silent 10.1.2.99)" done echo "***" echo "*** Results of load-balancing to 10.1.2.99: " echo "***" echo "$RESULTS" | tr ':' '\n' | grep -Ev "^$" | sort | uniq -c echo }
For HTTPS:
{ RESULTS= for i in {1..100} do RESULTS="$RESULTS:$(curl -k -s 'https://test.example.com:443' --connect-to test.example.com:443:10.1.2.99:443)" done echo "***" echo "*** Results of load-balancing to 10.1.2.99: " echo "***" echo "$RESULTS" | tr ':' '\n' | grep -Ev "^$" | sort | uniq -c echo }
Additional configuration options
This section expands on the configuration example to provide alternative and additional configuration options. All of the tasks are optional. You can perform them in any order.
Enabling session affinity
These procedures show you how to update a backend service for the example internal HTTP(S) load balancer so that the backend service uses generated cookie affinity, header field affinity, or HTTP cookie affinity.
When generated cookie affinity is enabled, the load balancer issues a cookie
on the first request. For each subsequent request with the same cookie, the load
balancer directs the request to the same backend VM or endpoint. For
internal HTTP(S) load balancers, the cookie is named GCILB
.
When header field affinity is enabled, the load balancer routes requests to
backend VMs or endpoints in a NEG based on the value of the HTTP header named
in the --custom-request-header
flag. Header field affinity is only valid if
the load balancing locality policy is either RING_HASH
or MAGLEV
and the
backend service's consistent hash specifies the name of the HTTP header.
When HTTP cookie affinity is enabled, the load balancer routes requests to
backend VMs or endpoints in a NEG, based on an HTTP cookie named in the
HTTP_COOKIE
flag with the optional --affinity-cookie-ttl
flag. If the client
does not provide the cookie in its HTTP request, the proxy generates
the cookie and returns it to the client in a Set-Cookie
header. HTTP cookie
affinity is only valid if the load balancing locality policy is either
RING_HASH
or MAGLEV
and the backend service's consistent hash specifies the
HTTP cookie.
Cloud console
To enable or change session affinity for a backend service:
- Go to the Load balancing page in the Google Cloud console.
Go to the Load balancing page - Click Backends.
- Click l7-ilb-backend-service (the name of the backend service you created for this example) and click Edit.
- On the Backend service details page, click Advanced configuration.
- Under Session affinity, select the type of session affinity you want from the menu.
- Click Update.
gcloud
Use the following gcloud commands to update the l7-ilb-backend-service
backend service to different types of session affinity:
gcloud compute backend-services update l7-ilb-backend-service \ --session-affinity=[GENERATED_COOKIE | HEADER_FIELD | HTTP_COOKIE | CLIENT_IP] --region=us-west1
API
To set session affinity, make a PATCH
request to the
regionBackendServices/patch
method.
PATCH https://compute.googleapis.com/compute/v1/projects/[PROJECT_ID]/regions/us-west1/regionBackendServices/l7-ilb-backend-service { "sessionAffinity": ["GENERATED_COOKIE" | "HEADER_FIELD" | "HTTP_COOKIE" | "CLIENT_IP" ] }
Restricting which clients can send traffic to the load balancer
You can restrict clients from being able to talk to an internal HTTP(S) load balancer forwarding rule VIP by configuring egress firewall rules on these clients. Set these firewall rules on specific client VMs based on service accounts or tags.
You can't use firewall rules to restrict inbound traffic to specific internal HTTP(S) load balancer forwarding rule VIPs. Any client on the same VPC network and in the same region as the forwarding rule VIP can generally send traffic to the forwarding rule VIP.
Furthermore, all requests to backends come from proxies that use IP addresses in the proxy-only subnet range. It is not possible to create firewall rules that allow or deny ingress traffic on these backends based on the forwarding rule VIP used by a client.
Here are some examples of how to use egress firewall rules to restrict traffic to the load balancer's forwarding rule VIP.
Console
To identify the client VMs, you will need to tag the
specific VMs
you want to restrict. These tags are used to associate firewall rules with
the tagged client VMs. Then add the tag to the TARGET_TAG
field in these steps.
Use either a single firewall rule or multiple rules to set this up.
Single egress firewall rule
You can configure one firewall egress rule to deny all egress traffic going from tagged client VMs to a load balancer's VIP.
- Go to the Firewall rules page in the Google Cloud console.
Go to the Firewall rules page - Click Create firewall rule to create the rule to deny egress
traffic from tagged client VMs to a load balancer's VIP.
- Name:
fr-deny-access
- Network:
lb-network
- Priority:
100
- Direction of traffic: Egress
- Action on match: Deny
- Targets: Specified target tags
- Target tags:
TARGET_TAG
- Destination filter: IP ranges
- Destination IP ranges:
10.1.2.99
- Protocols and ports:
- Choose Specified protocols and ports.
- Select the tcp checkbox, and then enter
80
for the port number.
- Name:
- Click Create.
Multiple egress firewall rules
A more scalable approach involves setting two rules. A default, low-priority rule that restricts all clients from accessing the load balancer's VIP. A second, higher-priority rule that allows a subset of tagged clients to access the load balancer's VIP. Only tagged VMs will be able to access the VIP.
- Go to the Firewall rules page in the Google Cloud console.
Go to the Firewall rules page - Click Create firewall rule to create the lower priority rule to deny
access by default:
- Name:
fr-deny-all-access-low-priority
- Network:
lb-network
- Priority:
200
- Direction of traffic: Egress
- Action on match: Deny
- Targets: Specified target tags
- Target tags:
TARGET_TAG
- Destination filter: IP ranges
- Destination IP ranges:
10.1.2.99
- Protocols and ports:
- Choose Specified protocols and ports.
- Select the tcp checkbox, and then enter
80
for the port number.
- Name:
- Click Create.
- Click Create firewall rule to create the higher priority rule to
allow traffic from certain tagged instances.
- Name:
fr-allow-some-access-high-priority
- Network:
lb-network
- Priority:
100
- Direction of traffic: Egress
- Action on match: Allow
- Targets: Specified target tags
- Target tags:
TARGET_TAG
- Destination filter: IP ranges
- Destination IP ranges:
10.1.2.99
- Protocols and ports:
- Choose Specified protocols and ports.
- Select the tcp checkbox, and then enter
80
for the port number.
- Name:
- Click Create.
gcloud
To identify the client VMs, you will need to tag the
specific VMs
you want to restrict. Then add the tag to the TARGET_TAG
field in these steps.
Use either a single firewall rule or multiple rules to set this up.
Single egress firewall rule
You can configure one firewall egress rule to deny all egress traffic going from tagged client VMs to a load balancer's VIP.
gcloud compute firewall-rules create fr-deny-access \ --network=lb-network \ --action=deny \ --direction=egress \ --rules=tcp \ --priority=100 \ --destination-ranges=10.1.2.99 \ --target-tags=TARGET_TAG
Multiple egress firewall rules
A more scalable approach involves setting two rules. A default, low-priority rule that restricts all clients from accessing the load balancer's VIP. A second, higher-priority rule allows a subset of tagged clients to access the load balancer's VIP. Only tagged VMs will be able to access the VIP.
Create the lower-priority rule:
gcloud compute firewall-rules create fr-deny-all-access-low-priority \ --network=lb-network \ --action=deny \ --direction=egress \ --rules=tcp \ --priority=200 \ --destination-ranges=10.1.2.99
Create the higher priority rule:
gcloud compute firewall-rules create fr-allow-some-access-high-priority \ --network=lb-network \ --action=allow \ --direction=egress \ --rules=tcp \ --priority=100 \ --destination-ranges=10.1.2.99 \ --target-tags=TARGET_TAG
To use service accounts instead of tags to control access, simply use
the --target-service-accounts
instead of the --target-tags
flag when creating firewall rules.
Scaling restricted access to internal HTTP(S) load balancer backends based on subnets
Maintaining separate firewall rules or adding new load-balanced IPs to existing rules as described in the previous section becomes inconvenient as the number of forwarding rules increases. One way to prevent this is to allocate forwarding rule IP addresses from a reserved subnet. Then traffic from tagged instances or service accounts can be allowed or blocked by using the reserved subnet as the destination range for firewall rules. This allows you to effectively control access to a group of forwarding rule VIPs without having to maintain per-VIP firewall egress rules.
Here are the high-level steps to set this up, assuming that you will create all the other required load balancer resources separately.
gcloud
Create a regional subnet that will be used to allocate load-balanced IP addresses for forwarding rules:
gcloud compute networks subnets create l7-ilb-restricted-subnet \ --network=lb-network \ --region=us-west1 \ --range=10.127.0.0/24
Create a forwarding rule that takes an address from the newly-allocated subnet. For this example, we've selected
10.127.0.1
from the subnet created in the previous step.gcloud compute forwarding-rules create l7-ilb-forwarding-rule-restricted \ --load-balancing-scheme=INTERNAL_MANAGED \ --network=lb-network \ --subnet=l7-ilb-restricted-subnet \ --address=10.127.0.1 \ --ports=80 \ --region=us-west1 \ --target-http-proxy=l7-ilb-proxy \ --target-http-proxy-region=us-west1
Create a firewall rule to restrict traffic destined for the range IP addresses in the forwarding rule subnet (
l7-ilb-restricted-subnet
):gcloud compute firewall-rules create restrict-traffic-to-subnet \ --network=lb-network \ --action=deny \ --direction=egress \ --rules=tcp:80 \ --priority=100 \ --destination-ranges=10.127.0.0/24 \ --target-tags=TARGET_TAG
Configure backend subsetting
Backend subsetting improves performance and scalability by assigning a subset of backends to each of the proxy instances. When enabled for a backend service, it adjusts the number of backends utilized by each proxy instance as follows:
As the number of proxy instances participating in the load balancer increases, the subset size decreases.
When the total number of backends in a network exceeds the capacity of a single proxy instance, the subset size is reduced automatically for each service that has backend subsetting enabled.
This example shows you how to create the internal HTTP(S) load balancer resources and enable backend subsetting:
- Use the example configuration
to create a regional backend service
l7-ilb-backend-service
. Enable backend subsetting by specifying
--subsetting-policy
asCONSISTENT_HASH_SUBSETTING
. The load balancing scheme isINTERNAL_MANAGED
.gcloud
Use the following
gcloud
command to updatel7-ilb-backend-service
with backend subsetting:gcloud beta compute backend-services update l7-ilb-backend-service \ --region=us-west1 \ --subsetting-policy=CONSISTENT_HASH_SUBSETTING
api
Make a
PATCH
request to theregionBackendServices/patch
method.PATCH https://compute.googleapis.com/compute/beta/projects/
PROJECT_ID
/regions/us-west1/backendServices/l7-ilb-backend-service{ "subsetting": { "policy": CONSISTENT_HASH_SUBSETTING } }
You can also refine backend load balancing by setting the localityLbPolicy
policy.
For more information, see Traffic policies.
What's next
To learn how Internal HTTP(S) Load Balancing works, see Internal HTTP(S) Load Balancing overview.
To manage the proxy-only subnet resource required by Internal HTTP(S) Load Balancing, see Proxy-only subnet for internal HTTP(S) load balancers.
To learn how to update your internal HTTP(S) load balancer's SSL certificate, see Using self-managed SSL certificates.