This page illustrates how to deploy an internal HTTP(S) load balancer to load balance traffic to network endpoints that are on-premises or in other public clouds and are reachable via hybrid connectivity.
If you haven't already done so, review the Hybrid connectivity NEGs overview to understand the network requirements to set up hybrid load balancing.
Setup overview
The example on this page sets up the following deployment:
You must configure hybrid connectivity before setting up a hybrid load balancing deployment. This page does not include the hybrid connectivity setup.
Depending on your choice of hybrid connectivity product (either Cloud VPN or Cloud Interconnect (Dedicated or Partner)), use the relevant product documentation.
Permissions
You must have the following permissions to set up hybrid load balancing:
On Google Cloud
- Permission to establish hybrid connectivity between Google Cloud and your on-premises or other cloud environments the environments. For the list of permissions needed, see the relevant Network connectivity product documentation.
On your on-premises or other non-Google Cloud cloud environment
- Permission to configure network endpoints that
allow services on your on-premises or other cloud environments to be reachable
from Google Cloud via an
IP:Port
combination. Contact your environment's network administrator for details. - Permission to create firewall rules on your on-premises or other cloud environments to allow Google's health check probes to reach the endpoints.
- Permission to configure network endpoints that
allow services on your on-premises or other cloud environments to be reachable
from Google Cloud via an
Additionally, to follow the instructions on this page, you create a hybrid connectivity NEG, a load balancer, and zonal NEGs (and their endpoints) to serve as Google Cloud-based backends for the load balancer.
You should be either a project owner or editor, or you should have the following Compute Engine IAM roles.
Task | Required role |
---|---|
Create and modify load balancer components | Network Admin |
Create and modify NEGs | Compute Instance Admin |
Add and remove firewall rules | Security Admin |
Establish hybrid connectivity
Your Google Cloud and on-premises or other cloud environments must be connected through hybrid connectivity, using either Cloud Interconnect VLAN attachments or Cloud VPN tunnels with Cloud Router. We recommend you use a high availability connection.
A Cloud Router enabled with Global dynamic routing learns about the specific endpoint via BGP and programs it into your Google Cloud VPC network. Regional dynamic routing is not supported. Static routes are also not supported.
The Google Cloud VPC network that you use to configure either Cloud Interconnect or Cloud VPN is the same network you use to configure the hybrid load balancing deployment. Ensure that your VPC network's subnet CIDR ranges do not conflict with your remote CIDR ranges. When IP addresses overlap, subnet routes are prioritized over remote connectivity.
For instructions, see:
Do not proceed with the instructions on this page until you have set up hybrid connectivity between your environments.
Set up your on-premises or other cloud environment
Perform the following steps to set up your on-premises or other cloud environment for hybrid load balancing:
- Configure network endpoints to expose on-premises services to
Google Cloud (
IP:Port
). - Configure firewall rules on your on-premises or other cloud environment.
- Configure Cloud Router to advertise certain required routes to your private environment.
Set up network endpoints
After you have set up hybrid connectivity, you configure one or
more network endpoints within your on-premises or other cloud environments that
are reachable via Cloud Interconnect or Cloud VPN using an
IP:port
combination. This IP:port
combination will be configured as one or
more endpoints for the hybrid connectivity NEG that will be created in
Google Cloud later on in this process.
If there are multiple paths to the IP endpoint, routing will follow the behavior described in the Cloud Router overview.
Set up firewall rules
The following firewall rules must be created on your on-premises or other cloud environment:
- Ingress allow firewall rules to allow traffic from Google's health-checking
probes to your endpoints.
For external HTTP(S) load balancer, internal HTTP(S) load balancer, TCP proxy load balancer, and
SSL proxy load balancer, the ranges to be allowed are:
35.191.0.0/16
and130.211.0.0/22
. For more details, see Probe IP ranges and firewall rules. - Ingress allow firewall rules to allow traffic that is being load-balanced to reach the endpoints.
- Create an ingress allow firewall rule to allow traffic from the region's proxy-only subnet to reach the endpoints.
Advertise routes
Configure Cloud Router to advertise the following routes to your on-premises or other cloud environment:
- The ranges used by Google's health check probes:
35.191.0.0/16
and130.211.0.0/22
. - The range of the region's proxy-only subnet.
Set up Google Cloud environment
For the following steps, make sure you use the same VPC network (called NETWORK in this procedure) that was used to configure hybrid connectivity between the environments.
Additionally, make sure the region used (called REGION in this procedure) is the same as that used to create the Cloud VPN tunnel or Cloud Interconnect VLAN attachment.
Configure the proxy-only subnet
This proxy-only subnet is used for all regional HTTP(S) load balancers in the REGION region.
Console
- In the Google Cloud console, go to the VPC networks page.
Go to VPC networks - Go to the network that was used to configure hybrid connectivity between the environments.
- Click Add subnet.
- Enter a Name: PROXY_ONLY_SUBNET_NAME.
- Select a Region: REGION.
- Set Purpose to Regional Managed Proxy.
- Enter an IP address range: PROXY_ONLY_SUBNET_RANGE.
- Click Add.
gcloud
Create the proxy-only subnet with the gcloud compute networks subnets
create
command.
gcloud compute networks subnets create PROXY_ONLY_SUBNET_NAME \ --purpose=REGIONAL_MANAGED_PROXY \ --role=ACTIVE \ --region=REGION \ --network=NETWORK \ --range=PROXY_ONLY_SUBNET_RANGE
Configure the load balancer subnet
This subnet is used to create the load balancer's zonal NEG backends, the frontend, and the internal IP address.
Cloud console
- In the Google Cloud console, go to the VPC networks page.
Go to VPC networks - Go to the network that was used to configure hybrid connectivity between the environments.
- In the Subnets section:
- Set the Subnet creation mode to Custom.
- In the New subnet section, enter the following information:
- Name: LB_SUBNET_NAME
- Region: REGION
- IP address range: LB_SUBNET_RANGE
- Click Done.
- Click Create.
gcloud
Create a subnet in the network that was used to configure hybrid connectivity between the environments.
gcloud compute networks subnets create LB_SUBNET_NAME \ --network=NETWORK \ --range=LB_SUBNET_RANGE \ --region=REGION
Reserve the load balancer's IP address
Console
You can reserve a standalone internal IP address using the Google Cloud console.
- Go to the VPC networks page.
- Click the network that was used to configure hybrid connectivity between the environments.
- Click Static internal IP addresses and then click Reserve static address.
- Enter a Name: LB_IP_ADDRESS.
- For the Subnet, select LB_SUBNET_NAME.
- If you want to specify which IP address to reserve, under Static IP address, select Let me choose, and then fill in a Custom IP address. Otherwise, the system automatically assigns an IP address in the subnet for you.
- If you want to use this IP address with multiple forwarding rules, under Purpose, choose Shared.
- Click Reserve to finish the process.
gcloud
- Using the gcloud CLI, run the
compute addresses create
command:
gcloud compute addresses create LB_IP_ADDRESS \ --region=REGION \ --subnet=LB_SUBNET_NAME \ --purpose=SHARED_LOADBALANCER_VIP
Use the
compute addresses describe
command to view the allocated IP address:gcloud compute addresses describe LB_IP_ADDRESS \ --region=REGION
Create firewall rules
In this example, you create the following firewall rules:
fw-allow-health-check
: An ingress rule, applicable to the instances being load balanced, that allows traffic from the load balancer and Google Cloud health checking systems (130.211.0.0/22
and35.191.0.0/16
). This example uses the target tagallow-health-check
to identify the backend VMs to which it should apply.fw-allow-ssh
: An ingress rule that allows incoming SSH connectivity on TCP port 22 from any address. You can choose a more restrictive source IP range for this rule; for example, you can specify just the IP ranges of the systems from which you will initiate SSH sessions. This example uses the target tagallow-ssh
to identify the VMs to which it should apply.fw-allow-proxy-only-subnet
: An ingress rule that allows connections from the proxy-only subnet to reach the backends.
Console
- Go to the Firewalls page in the Google Cloud console.
Go to Firewalls - Click Create firewall rule to create the rule to allow traffic from
health check probes:
- Enter a Name of
fw-allow-health-check
. - Under Network, select NETWORK.
- Under Targets, select Specified target tags.
- Populate the Target tags field with
allow-health-check
. - Set Source filter to IPv4 ranges.
- Set Source IPv4 ranges to
130.211.0.0/22
and35.191.0.0/16
. - Under Protocols and ports, select Specified protocols and ports.
- Select the checkbox next to tcp and then enter
80
for the port number. - Click Create.
- Enter a Name of
- Click Create firewall rule again to create the rule to allow incoming
SSH connections:
- Name:
fw-allow-ssh
- Network: NETWORK
- Priority:
1000
- Direction of traffic: ingress
- Action on match: allow
- Targets: Specified target tags
- Target tags:
allow-ssh
- Source filter: IPv4 ranges
- Source IPv4 ranges:
0.0.0.0/0
- Protocols and ports: Choose Specified protocols and ports,
and then enter
tcp:22
. - Click Create.
- Name:
- Click Create firewall rule again to create the rule to allow incoming
connections from the proxy-only subnet:
- Name:
fw-allow-ssh
- Network: NETWORK
- Priority:
1000
- Direction of traffic: ingress
- Action on match: allow
- Targets: Specified target tags
- Target tags:
allow-proxy-only-subnet
- Source filter: IPv4 ranges
- Source IPv4 ranges: PROXY_ONLY_SUBNET_RANGE
- Protocols and ports: Choose Specified protocols and ports,
and then enter
tcp:80
. - Click Create.
- Name:
gcloud
Create the
fw-allow-health-check-and-proxy
rule to allow the Google Cloud health checks to reach the backend instances on TCP port80
:gcloud compute firewall-rules create fw-allow-health-check \ --network=NETWORK \ --action=allow \ --direction=ingress \ --target-tags=allow-health-check \ --source-ranges=130.211.0.0/22,35.191.0.0/16 \ --rules=tcp:80
Create the
fw-allow-ssh
firewall rule to allow SSH connectivity to VMs with the network tagallow-ssh
. When you omitsource-ranges
, Google Cloud interprets the rule to mean any source.gcloud compute firewall-rules create fw-allow-ssh \ --network=NETWORK \ --action=allow \ --direction=ingress \ --target-tags=allow-ssh \ --rules=tcp:22
Create an ingress allow firewall rule for the proxy-only subnet to allow the load balancer to communicate with backend instances on TCP port
80
:gcloud compute firewall-rules create fw-allow-proxy-only-subnet \ --network=NETWORK \ --action=allow \ --direction=ingress \ --target-tags=allow-proxy-only-subnet \ --source-ranges=PROXY_ONLY_SUBNET_RANGE \ --rules=tcp:80
Set up the zonal NEG
For Google Cloud-based backends, we recommend you configure multiple zonal NEGs in the same region where you configured hybrid connectivity.
For this example, we set up a zonal NEG (with GCE_VM_IP_PORT
type endpoints)
in the REGION region. First create the VMs. Then create a zonal NEG
and add the VMs' network endpoints to the NEG.
Create VMs
Console
- Go to the VM instances page in the Google Cloud console.
Go to VM instances - Click Create instance.
- Set the Name to
vm-a1
. - For the Region, choose REGION, and choose any Zone. This will be referred to as ZONE in this procedure.
- In the Boot disk section, ensure that the Debian operating system and the 10 (buster) version are selected for the boot disk options. Click Choose to change the image if necessary.
Click Management, security, disks, networking, sole tenancy and make the following changes:
- Click Networking and add the following Network tags:
allow-ssh
,allow-health-check
, andallow-proxy-only-subnet
. - Click Edit
- Network: NETWORK
- Subnet: LB_SUBNET_NAME
- Primary internal IP: Ephemeral (automatic)
- External IP: Ephemeral
under
Network interfaces and make the following changes then click
Done:
Click Management. In the Startup script field, copy and paste the following script contents. The script contents are identical for all four VMs:
#! /bin/bash apt-get update apt-get install apache2 -y vm_hostname="$(curl -H "Metadata-Flavor:Google" \ http://169.254.169.254/computeMetadata/v1/instance/name)" echo "Page served from: $vm_hostname" | \ tee /var/www/html/index.html
- Click Networking and add the following Network tags:
Click Create.
Repeat the following steps to create a second VM, using the following name and zone combination:
- Name:
vm-a2
, zone: ZONE
- Name:
gcloud
Create the VMs by running the following command two times, using these combinations for VM_NAME and ZONE. The script contents are identical for both VMs.
- VM_NAME of
vm-a1
and any ZONE of your choice VM_NAME of
vm-a2
and the same ZONEgcloud compute instances create VM_NAME \ --zone=ZONE \ --image-family=debian-10 \ --image-project=debian-cloud \ --tags=allow-ssh,allow-health-check,allow-proxy-only-subnet \ --subnet=LB_SUBNET_NAME \ --metadata=startup-script='#! /bin/bash apt-get update apt-get install apache2 -y vm_hostname="$(curl -H "Metadata-Flavor:Google" \ http://169.254.169.254/computeMetadata/v1/instance/name)" echo "Page served from: $vm_hostname" | \ tee /var/www/html/index.html systemctl restart apache2'
Create the zonal NEG
gcloud
Create a zonal NEG (with
GCE_VM_IP_PORT
endpoints) using thegcloud compute network-endpoint-groups create
command:gcloud compute network-endpoint-groups create GCP_NEG_NAME \ --network-endpoint-type=GCE_VM_IP_PORT \ --zone=GCP_NEG_ZONE \ --network=NETWORK \ --subnet=LB_SUBNET_NAME [--default-port=DEFAULT_PORT]
You can either specify a DEFAULT_PORT while creating the NEG, or specify a port number for each endpoint in the next step.
Add endpoints to GCP_NEG_NAME.
gcloud compute network-endpoint-groups update GCP_NEG_NAME \ --zone=GCP_NEG_ZONE \ --add-endpoint='instance=vm-a1,[port=PORT_VM_A1]' \ --add-endpoint='instance=vm-a2,[port=PORT_VM_A2]'
Set up the hybrid connectivity NEG
When creating the NEG, use a ZONE that minimizes the geographic
distance between Google Cloud and your on-premises or other cloud
environment. For example, if you are hosting a service in an on-premises
environment in Frankfurt, Germany, you can specify the europe-west3-a
Google Cloud zone when you create the NEG.
Moreover, if you're using Cloud Interconnect, the ZONE used to create the NEG should be in the same region where the Cloud Interconnect attachment was configured.
For the available regions and zones, see the Compute Engine documentation: Available regions and zones.
Console
gcloud
Create a hybrid connectivity NEG using the
gcloud compute network-endpoint-groups create
command.gcloud compute network-endpoint-groups create ON_PREM_NEG_NAME \ --network-endpoint-type=NON_GCP_PRIVATE_IP_PORT \ --zone=ON_PREM_NEG_ZONE \ --network=NETWORK
Add the on-premises backend VM endpoint to ON_PREM_NEG_NAME:
gcloud compute network-endpoint-groups update ON_PREM_NEG_NAME \ --zone=ON_PREM_NEG_ZONE \ --add-endpoint="ip=ON_PREM_IP_ADDRESS_1,port=PORT_1" \ --add-endpoint="ip=ON_PREM_IP_ADDRESS_2,port=PORT_2"
You can use this command to add the network endpoints you previously
configured on-premises or in your cloud environment.
Repeat --add-endpoint
as many times as needed.
You can repeat these steps to create multiple hybrid NEGs if needed.
Configure the load balancer
Console
gcloud
- Create a health check
for the backends.
gcloud compute health-checks create http HTTP_HEALTH_CHECK_NAME \ --region=REGION \ --use-serving-port
- Create a backend service for Google Cloud-based backends.
gcloud compute backend-services create BACKEND_SERVICE_GCP \ --load-balancing-scheme=INTERNAL_MANAGED \ --protocol=HTTP \ --health-checks=HTTP_HEALTH_CHECK_NAME \ --health-checks-region=HEALTH_CHECK_REGION \ --region=REGION
- Add the zonal NEG as a backend to the backend service. For details about
configuring the balancing mode, see the gcloud CLI documentation
for the
--max-rate-per-endpoint
parameter.gcloud compute backend-services add-backend BACKEND_SERVICE_GCP \ --region=REGION \ --balancing-mode=RATE \ --max-rate-per-endpoint=MAX_REQUEST_RATE_PER_ENDPOINT \ --network-endpoint-group=GCP_NEG_NAME \ --network-endpoint-group-zone=GCP_NEG_ZONE
- Create a backend service for the on-premises backend.
gcloud compute backend-services create BACKEND_SERVICE_ON_PREM \ --load-balancing-scheme=INTERNAL_MANAGED \ --protocol=HTTP \ --health-checks=HTTP_HEALTH_CHECK_NAME \ --health-checks-region=HEALTH_CHECK_REGION \ --region=REGION
- Add the hybrid NEG as a backend to the backend service. For details
about configuring the balancing mode, see the gcloud CLI
documentation for the
--max-rate-per-endpoint
parameter.gcloud compute backend-services add-backend BACKEND_SERVICE_ON_PREM \ --region=REGION \ --balancing-mode=RATE \ --max-rate-per-endpoint=MAX_REQUEST_RATE_PER_ENDPOINT \ --network-endpoint-group=ON_PREM_NEG_NAME \ --network-endpoint-group-zone=ON_PREM_NEG_ZONE
- Create a URL map to route incoming requests to the backend service:
gcloud compute url-maps create URL_MAP_NAME \ --default-service BACKEND_SERVICE_GCP \ --region=REGION
Configure the URL map so that requests are directed to both on-prem and Google Cloud backend services. For example, you can create a path matcher so that requests matching the/on-prem-service
path are sent to BACKEND_SERVICE_ON_PREM while all other requests are sent to BACKEND_SERVICE_GCP.gcloud compute url-maps add-path-matcher URL_MAP_NAME \ --default-service BACKEND_SERVICE_GCP \ --path-matcher-name PATH_MATCHER \ --path-rules="/on-prem-service=BACKEND_SERVICE_ON_PREM" \ --region=REGION
You can also direct traffic to specific backend services based on the host component of the HTTP(S) request. For details, see Using URL maps. -
To create an HTTPS load balancer, you must have an
SSL certificate
resource to use in the HTTPS target proxy. You can create an SSL
certificate resource using either a Google-managed SSL certificate or
a self-managed SSL certificate. Using Google-managed certificates
is recommended because Google Cloud obtains, manages, and renews
these certificates automatically.
To create a Google-managed certificate, you must have a domain. If you do not have a domain, you can use a self-signed SSL certificate for testing.
To create a Google-managed SSL certificate resource:gcloud compute ssl-certificates create SSL_CERTIFICATE_NAME \ --domains DOMAIN
To create a self-managed SSL certificate resource:gcloud compute ssl-certificates create SSL_CERTIFICATE_NAME \ --certificate CRT_FILE_PATH \ --private-key KEY_FILE_PATH
- Create a target HTTP(S) proxy to route requests to your URL map.
For an HTTP load balancer, create an HTTP target proxy:gcloud compute target-http-proxies create TARGET_HTTP_PROXY_NAME \ --url-map=URL_MAP_NAME \ --url-map-region=REGION \ --region=REGION
For an HTTPS load balancer, create an HTTPS target proxy. The proxy is the portion of the load balancer that holds the SSL certificate for HTTPS Load Balancing, so you also load your certificate in this step.gcloud compute target-https-proxies create TARGET_HTTPS_PROXY_NAME \ --ssl-certificates=SSL_CERTIFICATE_NAME \ --url-map=URL_MAP_NAME \ --url-map-region=REGION \ --region=REGION
- Create a forwarding rule to route incoming requests to the proxy. Don't
use the proxy-only subnet to create the forwarding rule.
For an HTTP load balancer:gcloud compute forwarding-rules create HTTP_FORWARDING_RULE_NAME \ --load-balancing-scheme=INTERNAL_MANAGED \ --network=NETWORK \ --subnet=LB_SUBNET_NAME \ --address=LB_IP_ADDRESS \ --ports=80 \ --region=REGION \ --target-http-proxy=TARGET_HTTP_PROXY_NAME \ --target-http-proxy-region=REGION
For an HTTPS load balancer:gcloud compute forwarding-rules create HTTPS_FORWARDING_RULE_NAME \ --load-balancing-scheme=INTERNAL_MANAGED \ --network=NETWORK \ --subnet=LB_SUBNET_NAME \ --address=LB_IP_ADDRESS \ --ports=443 \ --region=REGION \ --target-http-proxy=TARGET_HTTPS_PROXY_NAME \ --target-http-proxy-region=REGION
Connect your domain to your load balancer
After the load balancer is created, note the IP address that is associated with
the load balancer: for example, 30.90.80.100
. To point your domain to your
load balancer, create an A
record using your domain registration service. If
you added multiple domains to your SSL certificate, you must add an A
record
for each one, all pointing to the load balancer's IP address. For example, to
create A
records for www.example.com
and example.com
:
NAME TYPE DATA www A 30.90.80.100 @ A 30.90.80.100
If you are using Google Domains, see the Google Domains Help page for more information.
Testing the load balancer
To test the load balancer, create a client VM in the same region as the load balancer. Then send traffic from the client to the load balancer.
Create a client VM
This example creates a client VM (vm-client
) in the same region as the backend
NEGs. The client is used to validate the load balancer's configuration
and demonstrate expected behavior as described in the
testing section.
Console
- Go to the VM instances page in the Google Cloud console.
Go to VM instances - Click Create instance.
- Set the Name to
vm-client
. - Set the Zone to ZONE.
- Click Management, security, disks, networking, sole tenancy and make
the following changes:
- Click Networking and add the
allow-ssh
to Network tags. - Click the edit button under Network interfaces and make the
following changes then click Done:
- Network: NETWORK
- Subnet: LB_SUBNET_NAME
- Primary internal IP: Ephemeral (automatic)
- External IP: Ephemeral
- Click Networking and add the
- Click Create.
gcloud
The client VM can be in any zone in the same region as the load balancer, and it can use any subnet in that region. In this example, the client is in the ZONE zone, and it uses the same subnet as the backend VMs.
gcloud compute instances create vm-client \ --zone=ZONE \ --image-family=debian-10 \ --image-project=debian-cloud \ --tags=allow-ssh \ --subnet=LB_SUBNET_NAME
Send traffic to the load balancer
Now that you have configured your load balancer, you can start sending traffic to the load balancer's IP address.
Connect via SSH to the client instance.
gcloud compute ssh client-vm \ --zone=ZONE
Verify that the load balancer is serving backend hostnames as expected.
For HTTP testing, run:
curl LB_IP_ADDRESS
To confirm that the non-Google Cloud endpoints are responding, run:
curl LB_IP_ADDRESS/on-prem-service
For HTTPS testing, run:
curl -k -s 'https://test.example.com:443' --connect-to test.example.com:443:LB_IP_ADDRESS:443
The -k
flag causes curl to skip certificate validation.