This document provides instructions for configuring Internal HTTP(S) Load Balancing for your services running in Google Kubernetes Engine (GKE) pods.
Before you begin
Before following the instructions in this guide, review the following documents:
Refer to Preparing for Internal HTTP(S) Load Balancing setup to make sure that you have completed the prerequisites.
In the Setup overview, refer to the diagram that shows the required resources.
In the Internal HTTP(S) Load Balancing concepts, make sure that you understand the Limitations.
Configuring Internal HTTP(S) Load Balancing with a GKE-based service
This section shows the configuration required for services that run on GKE pods. Client VMs connect to the IP address and port that you configure in the forwarding rule. When your client applications send traffic to this IP address and port, their requests are forwarded to your backend GKE pods according to your internal HTTP(S) load balancer's URL map.
The example on this page explicitly sets a reserved internal IP address for the internal forwarding rule, rather than allowing an ephemeral internal IP address to be allocated. This is the recommended best practice for forwarding rules.
Configuring backends: GKE cluster
This section creates a demonstration GKE cluster and a deployment for running a simple container that runs a simple web server that that serves the node's hostname. It also creates a ClusterIP service, which creates a NEG automatically.
Console
To create a cluster using Cloud Console, perform the following steps:
Go to the Google Kubernetes Engine menu in Cloud Console.
Click Create cluster.
In the Name field, enter
l7-ilb-cluster
.For the Location type, select
Zonal
.For the Zone, select
us-west1-b
.Choose the default Master version.
Expand Availability, networking, security, and additional features:
- Ensure that the Enable VPC-native (using alias IP) box is checked.
- For the Network, select
lb-network
. - For the Node subnet, select
backend-subnet
. - Make sure Load balancing is enabled by checking the box
next to
Enable HTTP load balancing
.
For the remaining fields, keep the default values or adjust them to meet your needs.
Click Create.
When you use the Cloud Console to create the cluster, you must add the system-generated network tag to the proxy firewall filter:
Find the network tag that GKE added to the nodes in the cluster. The tag is generated from the cluster hash.
Go to the VM instances page.
In the Columns pulldown, select Network Tags.
Copy the network tag for the GKE nodes. It should look something like this:
gke-l7-ilb-cluster-id-node
Edit the
fw-allow-proxies
firewall rule and add the tag.- Go to the Firewall rules page in the Google Cloud Console.
Go to the Firewall rules page - Click the
fw-allow-proxies
firewall rule, and then click Edit. - In the Target tags field, add the network tag that you copied in the previous step.
- Click Save.
- Go to the Firewall rules page in the Google Cloud Console.
gcloud
Create a GKE cluster with the gcloud container clusters create command.
gcloud container clusters create l7-ilb-cluster \ --zone=us-west1-b \ --network=lb-network \ --subnetwork=backend-subnet \ --enable-ip-alias \ --tags=allow-ssh,load-balanced-backend
api
Create a GKE cluster with the
projects.zones.clusters.create
method, replacing [project-id] with your project ID.
POST https://container.googleapis.com/v1beta1/projects/<var>[project-id]</var>/zones/us-west1-b/clusters
{
"cluster": {
"name": "l7-ilb-cluster",
"network": "projects/<var>[project-id]</var>/global/networks/lb-network1",
"subnetwork": "projects/<var>[project-id]</var>/regions/us-west1/subnetworks/backend-subnet1",
"initialClusterVersion": "1.11",
"location": "us-west1-b",
"nodePools": [{
"name": "l7-ilb-node-pool",
"initialNodeCount": 3
}],
"nodeConfig":{
"tags":["allow-ssh","load-balanced-backend"]
}
"defaultMaxPodsConstraint": {
"maxPodsPerNode": "110"
},
"ipAllocationPolicy": {
"useIpAliases": True
}
}
}
Getting credentials to operate the cluster
Use the gcloud container clusters get-credentials command.
gcloud container clusters get-credentials l7-ilb-cluster \ --zone=us-west1-b
Defining a deployment with test containers that serve their hostname
Create hostname.yaml
with the deployment and service specification.
cat << EOF > hostname.yaml apiVersion: v1 kind: Service metadata: name: hostname annotations: cloud.google.com/neg: '{"exposed_ports":{"80":{}}}' spec: ports: - port: 80 name: host1 protocol: TCP targetPort: 8000 selector: run: hostname type: ClusterIP --- apiVersion: extensions/v1beta1 kind: Deployment metadata: labels: run: hostname name: hostname spec: replicas: 3 template: metadata: labels: run: hostname spec: containers: - image: gcr.io/kubernetes-e2e-test-images/serve-hostname-amd64:1.1 name: host1 command: - /bin/sh - -c - /serve_hostname -http=true -udp=false -port=8000 ports: - protocol: TCP containerPort: 8000 EOF
Applying the configuration
kubectl apply -f hostname.yaml
Verifying the deployment and GKE configuration
Make sure that the new service hostname is created and the application pod is running.
kubectl get svc
The example output is as follows:
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE hostname ClusterIP 10.71.9.7180/TCP 41m [..skip..]
Note that your cluster IP address is likely different because GKE automatically
creates the cluster IP range as a secondary IP range on the backend-subnet
.
If the kubectl get svc
command fails with credential/OAuth issues,
run the gcloud auth application-default login
command.
kubectl get pods
The example output is as follows:
NAME READY STATUS RESTARTS AGE hostname-6db459dcb9-896kh 1/1 Running 0 33m hostname-6db459dcb9-k6ddk 1/1 Running 0 50m hostname-6db459dcb9-x72kb 1/1 Running 0 33m [..skip..]
Note that your pod names are different.
You should see that the new service hostname was created, and pods for the hostname application are running.
Getting the name of the NEG
Look up the name of the NEG by using the gcloud compute network-endpoint-groups list command, filtering by the cluster's zone and the name of the deployment:
gcloud compute network-endpoint-groups list \ --filter="us-west1-b AND hostname" \ --format="get(name)"
Examining the NEG configuration
Examine details and list endpoints in the NEG with the gcloud compute network-endpoint-groups list-network-endpoints and gcloud compute network-endpoint-groups describe commands, replacing neg-name with the name of the NEG that you created.
gcloud compute network-endpoint-groups describe neg-name \ --zone=us-west1-b
gcloud compute network-endpoint-groups list-network-endpoints neg-name \ --zone=us-west1-b
Configuring the load balancer for GKE
The example demonstrates the following internal HTTP(S) load balancer configuration tasks:
- Creates a health check using the HTTP protocol
- Creates a regional, internal managed backend service
- Add the NEG as a backend to the backend service
- Create a URL map
- Make sure to refer to a regional URL map if a region is defined for the target HTTP(S) proxy. A regional URL map routes requests to a regional backend service based on rules that you define for the host and path of an incoming URL. A regional URL map can be referenced by a regional target proxy rule in the same region only.
- Create a target proxy
Create a forwarding rule
Console
Select a load balancer type
- Go to the Load balancing page in the Google Cloud Console.
Go to the Load balancing page - Click Create load balancer.
- Under HTTP(S) Load Balancing, click Start configuration.
- Select Only between my VMs. This setting means that the load balancer is internal.
- Click Continue.
Prepare the load balancer
- For the Name of the load balancer, enter
l7-ilb-gke-map
. - Ensure the Protocol is HTTP.
- For the Region, select
us-west1
. - For the VPC network, select
lb-network
. - Keep the window open to continue.
Reserve a proxy-only subnet
For Internal HTTP(S) Load Balancing, reserve a proxy subnet:
- Click Reserve a Subnet.
- For the Name, enter
proxy-subnet
. - For the Network, select
lb-network
. - For the Region, select
us-west1
. - For the IP address range, enter
10.129.0.0/26
. - Click Add.
Configure the backend service
- Click Backend configuration.
- From the Create or select backend services menu, select Create a backend service.
- Set the Name of the backend service to
l7-ilb-gke-backend-service
. - Under Backend type, select Network endpoint groups.
- In the New backend card of the Backends section:
- Set the Network endpoint group to the NEG was created by GKE. Refer to Getting the name of the NEG for how to determine its name.
- Enter a maximum rate of
5
RPS per endpoint. Google Cloud will exceed this maximum if necessary. - Click Done.
- In the Health check section, choose Create a health check with the
following parameters:
- Name:
l7-ilb-gke-basic-check
- Protocol: HTTP
- Port specification: Serving port
- Click Save and Continue.
- Name:
- Click Create.
Configure the URL map
- Click Host and path rules. Ensure that the l7-ilb-gke-backend-service is the only backend service for any unmatched host and any unmatched path.
Configure the frontend components
- Click Frontend configuration and edit the New frontend IP and port section.
- Set the Name to l7-ilb-gke-forwarding-rule.
- Set the Protocol to HTTP.
- Set the Subnet to backend-subnet.
- Choose Reserve a static internal IP address from the Internal IP pop-up button.
- In the panel that appears provide the following details:
- Name: l7-ilb-gke-ip
- In the Static IP address section, select Let me choose.
- In the Custom IP address section, enter
10.1.2.199
. - Click Reserve.
- Set the Port to
80
. - Click Done.
Complete the configuration
- Click Create.
gcloud
Define the HTTP health check with the gcloud compute health-checks create http command.
gcloud beta compute health-checks create http l7-ilb-gke-basic-check \ --region=us-west1 \ --use-serving-port
Define the backend service with the gcloud compute backend-services create command.
gcloud beta compute backend-services create l7-ilb-gke-backend-service \ --load-balancing-scheme=INTERNAL_MANAGED \ --protocol=HTTP \ --health-checks=l7-ilb-gke-basic-check \ --health-checks-region=us-west1 \ --region=us-west1
Add NEG backends to the backend service with the gcloud compute backend-services add-backend command.
gcloud beta compute backend-services add-backend l7-ilb-gke-backend-service \ --network-endpoint-group=neg-name \ --network-endpoint-group-zone=us-west1-b \ --region=us-west1 \ --balancing-mode=RATE \ --max-rate-per-endpoint=5
Create the URL map with the gcloud compute url-maps create command.
gcloud beta compute url-maps create l7-ilb-gke-map \ --default-service=l7-ilb-gke-backend-service \ --region=us-west1
Create the target proxy with the gcloud compute target-http-proxies create command.
gcloud beta compute target-http-proxies create l7-ilb-gke-proxy \ --url-map=l7-ilb-gke-map \ --url-map-region=us-west1 \ --region=us-west1
Create the forwarding rule with the gcloud compute forwarding-rules create command.
For custom networks, you must reference the subnet in the forwarding rule. Note that this is the VM subnet, not the proxy subnet.
gcloud beta compute forwarding-rules create l7-ilb-gke-forwarding-rule \ --load-balancing-scheme=INTERNAL_MANAGED \ --network=lb-network \ --subnet=backend-subnet \ --address=10.1.2.199 \ --ports=80 \ --region=us-west1 \ --target-http-proxy=l7-ilb-gke-proxy \ --target-http-proxy-region=us-west1
api
Create the health check by making a POST
request to the
healthChecks.insert
method, replacing [project-id] with your project ID.
POST https://www.googleapis.com/compute/beta/projects/[project-id]/global/healthChecks
{
"name": "l7-ilb-gke-basic-check",
"type": "HTTP",
"httpHealthCheck": {
"portSpecification": "USE_SERVING_PORT"
}
}
Create the regional backend service by making a POST
request to the
regionBackendServices.insert
method, replacing [project-id] with your project ID and
[neg-name] with the name of the NEG that you created.
POST https://www.googleapis.com/compute/beta/projects/[project-id]/regions/us-west1/backendServices
{
"name": "l7-ilb-gke-backend-service",
"backends": [
{
"group": "https://www.googleapis.com/compute/v1/projects/[project-id]/zones/us-west1-b/networkEndpointGroups/[neg-name]",
"balancingMode": "RATE"
"maxRatePerEndpoint": 5
}
],
"healthChecks": [
"projects/[project-id]/global/healthChecks/l7-ilb-gke-basic-check"
],
"loadBalancingScheme": "INTERNAL_MANAGED"
}
Create the URL map by making a POST
request to the
urlMap.insert
method, replacing [project-id] with your project ID.
POST https://www.googleapis.com/compute/beta/projects/[project-id]/global/urlMaps
{
"name": "l7-ilb-gke-map",
"defaultService": "projects/[project-id]/regions/us-west1/backendServices/l7-ilb-gke-backend-service"
}
Create the target HTTP proxy by making a POST
request to the
targetHttpProxies.insert
method, replacing [project-id] with your project ID.
POST https://www.googleapis.com/compute/beta/projects/[project-id]/regions/us-west1/targetHttpProxy
{
"name": "l7-ilb-gke-proxy",
"urlMap": "projects/[project-id]/global/urlMaps/l7-ilb-gke-map",
"region": "us-west1"
}
Create the forwarding rule by making a POST
request to the
forwardingRules.insert
method, replacing [project-id] with your project ID.
POST https://www.googleapis.com/compute/beta/projects/[project-id]/regions/us-west1/forwardingRules
{
"name": "l7-ilb-gke-forwarding-rule",
"IPAddress": "10.1.2.199",
"IPProtocol": "TCP",
"portRange": "80-80",
"target": "projects/[project-id]/regions/us-west1/targetHttpProxies/l7-ilb-gke-proxy",
"loadBalancingScheme": "INTERNAL_MANAGED",
"subnetwork": "projects/[project-id]/regions/us-west1/subnetworks/backend-subnet",
"network": "projects/[project-id]/global/networks/lb-network",
"networkTier": "PREMIUM",
}
Testing
Creating a VM instance in the zone to test connectivity
gcloud compute instances create l7-ilb-client-us-west1-b \ --image-family=debian-9 \ --image-project=debian-cloud \ --zone=us-west1-b \ --network=lb-network \ --subnet=backend-subnet \ --tags=l7-ilb-client
Allowing SSH access to the instance
gcloud compute firewall-rules create allow-ssh-to-l7-ilb-client \ --network=lb-network \ --target-tags=l7-ilb-client \ --allow=tcp:22
Testing the load balancer
Log in to the client instance to that HTTP(S) services on the backends are reachable via the internal HTTP(S) load balancer's forwarding rule IP address, and traffic is being load balanced among endpoints in the NEG.
Connecting via SSH to each client instance
gcloud compute ssh l7-ilb-client-us-west1-b \ --zone=us-west1-b
Verifying that the IP is serving its hostname
curl 10.1.2.199
Running 100 requests and confirm that they are load balanced
{ RESULTS= for i in {1..100} do RESULTS="$RESULTS:$(curl --silent 10.1.2.199)" done echo "***" echo "*** Results of load-balancing to 10.1.2.199: " echo "***" echo "$RESULTS" | tr ':' '\n' | grep -Ev "^$" | sort | uniq -c echo }
Note that multiple proxies perform the load balancing, one for each curl command, and they don't coordinate their selection of backends. Therefore, in this test, the backends don't receive the same number of requests. However, over the long term (in other words, thousands to millions of requests), the fraction of requests received by each backend approaches an equal distribution.
What's next
- See Internal HTTP(S) Load Balancing Concepts for information on how Internal HTTP(S) Load Balancing works.
- See Setting Up Internal HTTP(S) Load Balancing for Compute Engine VMs if you want your backend services to run on Compute Engine VMs.
- See Proxy-only subnet for internal HTTP(S) load balancers for information on managing the proxy-only subnet resource required by Internal HTTP(S) Load Balancing.
- See Troubleshooting Internal HTTP(S) Load Balancing for guidance on resolving configuration issues.