Setting Up Internal HTTP(S) Load Balancing for GKE pods

This document provides instructions for configuring Internal HTTP(S) Load Balancing for your services running in Google Kubernetes Engine (GKE) pods.

Before you begin

The instructions in this guide require you to use bash or a bash-like shell. For example, you can use Cloud Shell.

Before following the instructions in this guide, review the following documents:

Configuring Internal HTTP(S) Load Balancing with a GKE-based service

This section shows the configuration required for services that run on GKE pods. Client VMs connect to the IP address and port that you configure in the forwarding rule. When your client applications send traffic to this IP address and port, their requests are forwarded to your backend GKE pods according to your internal HTTP(S) load balancer's URL map.

The example on this page explicitly sets a reserved internal IP address for the internal forwarding rule, rather than allowing an ephemeral internal IP address to be allocated. This is the recommended best practice for forwarding rules.

For the forwardng rule's IP address, use the backend-subnet. If you try to use the proxy-only subnet, forwarding rule creation fails.

Configuring backends: GKE cluster

This section creates a demonstration GKE cluster and a deployment for running a simple container that runs a simple web server that that serves the node's hostname. It also creates a ClusterIP service, which creates a NEG automatically.

Console

To create a cluster using Cloud Console, perform the following steps:

  1. Go to the Google Kubernetes Engine menu in Cloud Console.

    Visit the Google Kubernetes Engine menu

  2. Click Create cluster.

  3. In the Name field, enter l7-ilb-cluster.

  4. For the Location type, select Zonal.

  5. For the Zone, select us-west1-b.

  6. Choose the default Master version.

  7. Expand Availability, networking, security, and additional features:

    1. Ensure that the Enable VPC-native (using alias IP) box is checked.
    2. For the Network, select lb-network.
    3. For the Node subnet, select backend-subnet.
    4. Make sure Load balancing is enabled by checking the box next to Enable HTTP load balancing.

  8. For the remaining fields, keep the default values or adjust them to meet your needs.

  9. Click Create.

When you use the Cloud Console to create the cluster, you must add the system-generated network tag to the proxy firewall filter:

  1. Find the network tag that GKE added to the nodes in the cluster. The tag is generated from the cluster hash.

    1. Go to the VM instances page.

      Go to the VM instances page

    2. In the Columns pulldown, select Network Tags.

    3. Copy the network tag for the GKE nodes. It should look something like this:

      gke-l7-ilb-cluster-id-node

  2. Edit the fw-allow-proxies firewall rule and add the tag.

    1. Go to the Firewall rules page in the Google Cloud Console.
      Go to the Firewall rules page
    2. Click the fw-allow-proxies firewall rule, and then click Edit.
    3. In the Target tags field, add the network tag that you copied in the previous step.
    4. Click Save.

gcloud

  1. Create a GKE cluster with the gcloud container clusters create command.

    gcloud container clusters create l7-ilb-cluster \
    --zone=us-west1-b \
    --network=lb-network \
    --subnetwork=backend-subnet \
    --enable-ip-alias \
    --tags=allow-ssh,load-balanced-backend

api

Create a GKE cluster with the projects.zones.clusters.create method, replacing [project-id] with your project ID.

POST https://container.googleapis.com/v1/projects/[project-id]/zones/us-west1-b/clusters
{
  "cluster": {
    "name": "l7-ilb-cluster",
    "network": "projects/[project-id]/global/networks/lb-network",
    "subnetwork": "projects/[project-id]/regions/us-west1/subnetworks/backend-subnet",
    "initialClusterVersion": "",
    "location": "us-west1-b",
    "nodePools": [{
      "name": "l7-ilb-node-pool",
      "initialNodeCount": 3,
      "config": {
        "tags":["allow-ssh","load-balanced-backend"]
      }
    }],
    "defaultMaxPodsConstraint": {
      "maxPodsPerNode": "110"
    },
    "ipAllocationPolicy": {
      "useIpAliases": true
    }
  }
}

Getting credentials to operate the cluster

Use the gcloud container clusters get-credentials command.

gcloud container clusters get-credentials l7-ilb-cluster \
    --zone=us-west1-b

Defining a deployment with test containers that serve their hostname

Create hostname.yaml with the deployment and service specification.

cat << EOF > hostname.yaml
apiVersion: v1
kind: Service
metadata:
  name: hostname
  annotations:
    cloud.google.com/neg: '{"exposed_ports":{"80":{}}}'
spec:
  ports:
  - port: 80
    name: host1
    protocol: TCP
    targetPort: 8000
  selector:
    run: hostname
  type: ClusterIP

---

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    run: hostname
  name: hostname
spec:
  replicas: 3
  selector:
    matchLabels:
      run: hostname
  template:
    metadata:
      labels:
        run: hostname
    spec:
      containers:
      - image: gcr.io/kubernetes-e2e-test-images/serve-hostname-amd64:1.1
        name: host1
        command:
        - /bin/sh
        - -c
        - /serve_hostname -http=true -udp=false -port=8000
        ports:
        - protocol: TCP
          containerPort: 8000
EOF

Applying the configuration

  1. If you aren't using a service account to authorize access to Google Cloud, run the gcloud auth application-default login command.

    gcloud auth application-default login

  2. Apply the configuration.

    kubectl apply -f hostname.yaml

Verifying the deployment and GKE configuration

Make sure that the new service hostname is created and the application pod is running.

kubectl get svc

The example output is as follows:

NAME         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
hostname     ClusterIP   10.71.9.71           80/TCP    41m
[..skip..]

Note that your cluster IP address is likely different because GKE automatically creates the cluster IP range as a secondary IP range on the backend-subnet.

kubectl get pods

The example output is as follows:

NAME                        READY     STATUS    RESTARTS   AGE
hostname-6db459dcb9-896kh   1/1       Running   0          33m
hostname-6db459dcb9-k6ddk   1/1       Running   0          50m
hostname-6db459dcb9-x72kb   1/1       Running   0          33m
[..skip..]

The output should display a new service hostname. Pods for the hostname application should have a STATUS of Running.

Note that your pod names will be different from the sample names.

Getting the name of the NEG

Look up the name of the NEG by using the gcloud compute network-endpoint-groups list command, filtering by the cluster's zone and the name of the deployment. Assign the NEG name to a variable called DEPLOYMENT_NAME:

export DEPLOYMENT_NAME=$(gcloud compute network-endpoint-groups list \
       --filter="us-west1-b AND hostname" \
       --format="get(name)")
echo $DEPLOYMENT_NAME

Examining the NEG configuration

Examine details and list endpoints in the NEG with the gcloud compute network-endpoint-groups list-network-endpoints and gcloud compute network-endpoint-groups describe commands.

gcloud compute network-endpoint-groups describe $DEPLOYMENT_NAME \
    --zone=us-west1-b

gcloud compute network-endpoint-groups list-network-endpoints $DEPLOYMENT_NAME \
    --zone=us-west1-b

Configuring the load balancer for GKE

This example shows you how to create the following internal HTTP(S) load balancer resources:

  • HTTP health check
  • Backend service with a NEG as the backend
  • A URL map
    • Make sure to refer to a regional URL map if a region is defined for the target HTTP(S) proxy. A regional URL map routes requests to a regional backend service based on rules that you define for the host and path of an incoming URL. A regional URL map can be referenced by a regional target proxy rule in the same region only.
  • SSL certificate (for HTTPS)
  • Target proxy
  • Forwarding rule

For the forwardng rule's IP address, use the backend-subnet. If you try to use the proxy-only subnet, forwarding rule creation fails.

Console

Select a load balancer type

  1. Go to the Load balancing page in the Google Cloud Console.
    Go to the Load balancing page
  2. Under HTTP(S) Load Balancing, click Start configuration.
  3. Select Only between my VMs. This setting means that the load balancer is internal.
  4. Click Continue.

Prepare the load balancer

  1. For the Name of the load balancer, enter l7-ilb-gke-map.
  2. For the Region, select us-west1.
  3. For the Network, select lb-network.
  4. Keep the window open to continue.

Reserve a proxy-only subnet

For Internal HTTP(S) Load Balancing, reserve a proxy subnet:

  1. Click Reserve a Subnet.
  2. For the Name, enter proxy-only-subnet.
  3. For the IP address range, enter 10.129.0.0/26.
  4. Click Add.

Configure the backend service

  1. Click Backend configuration.
  2. From the Create or select backend services menu, select Create a backend service.
  3. Set the Name of the backend service to l7-ilb-gke-backend-service.
  4. Under Backend type, select Network endpoint groups.
  5. In the New backend card of the Backends section:
    1. Set the Network endpoint group to the NEG was created by GKE. Refer to Getting the name of the NEG for how to determine its name.
    2. Enter a maximum rate of 5 RPS per endpoint. Google Cloud will exceed this maximum if necessary.
    3. Click Done.
  6. In the Health check section, choose Create a health check with the following parameters:
    1. Name: l7-ilb-gke-basic-check
    2. Protocol: HTTP
    3. Port specification: Serving port
    4. Click Save and Continue.
  7. Click Create.

Configure the URL map

  1. Click Host and path rules. Ensure that the l7-ilb-gke-backend-service is the only backend service for any unmatched host and any unmatched path.

Configure the frontend

For HTTP:

  1. Click Frontend configuration.
  2. Click Add frontend IP and port.
  3. Set the Name to l7-ilb-gke-forwarding-rule.
  4. Set the Protocol to HTTP.
  5. Set the Subnetwork to backend-subnet.
  6. Under Internal IP, select Reserve a static internal IP address.
  7. In the panel that appears provide the following details:
    1. Name: l7-ilb-gke-ip
    2. In the Static IP address section, select Let me choose.
    3. In the Custom IP address section, enter 10.1.2.199.
    4. Click Reserve.
  8. Set the Port to 80.
  9. Click Done.

For HTTPS:

If you are using HTTPS between the client and the load balancer, you need one or more SSL certificate resources to configure the proxy. See SSL Certificates for information on how to create SSL certificate resources. Google-managed certificates aren't currently supported with internal HTTP(S) load balancers.

  1. Click Frontend configuration.
  2. Click Add frontend IP and port.
  3. In the Name field, enter l7-ilb-gke-forwarding-rule.
  4. In the Protocol field, select HTTPS (includes HTTP/2).
  5. Set the Subnet to backend-subnet.
  6. Under Internal IP, select Reserve a static internal IP address.
  7. In the panel that appears provide the following details:
    1. Name: l7-ilb-gke-ip
    2. In the Static IP address section, select Let me choose.
    3. In the Custom IP address section, enter 10.1.2.199.
    4. Click Reserve.
  8. Ensure that the Port is set to 443, to allow HTTPS traffic.
  9. Click the Certificate drop-down list.
    1. If you already have a self-managed SSL certificate resource you want to use as the primary SSL certificate, select it from the drop-down menu.
    2. Otherwise, select Create a new certificate.
      1. Fill in a Name of l7-ilb-cert.
      2. In the appropriate fields upload your PEM-formatted files:
        • Public key certificate
        • Certificate chain
        • Private key
      3. Click Create.
  10. To add certificate resources in addition to the primary SSL certificate resource:
    1. Click Add certificate.
    2. Select a certificate from the Certificates list or click Create a new certificate and follow the instructions above.
  11. Click Done.

Complete the configuration

  1. Click Create.

gcloud

  1. Define the HTTP health check with the gcloud compute health-checks create http command.

    gcloud compute health-checks create http l7-ilb-gke-basic-check \
   --region=us-west1 \
   --use-serving-port

  2. Define the backend service with the gcloud compute backend-services create command.

    gcloud compute backend-services create l7-ilb-gke-backend-service \
  --load-balancing-scheme=INTERNAL_MANAGED \
  --protocol=HTTP \
  --health-checks=l7-ilb-gke-basic-check \
  --health-checks-region=us-west1 \
  --region=us-west1

  3. Add NEG backends to the backend service with the gcloud compute backend-services add-backend command.

    gcloud compute backend-services add-backend l7-ilb-gke-backend-service \
   --network-endpoint-group=$DEPLOYMENT_NAME \
   --network-endpoint-group-zone=us-west1-b \
   --region=us-west1 \
   --balancing-mode=RATE \
   --max-rate-per-endpoint=5

  4. Create the URL map with the gcloud compute url-maps create command.

    gcloud compute url-maps create l7-ilb-gke-map \
  --default-service=l7-ilb-gke-backend-service \
  --region=us-west1

  5. Create the target proxy.

    For HTTP:

    Use the gcloud compute target-http-proxies create command.

    gcloud compute target-http-proxies create l7-ilb-gke-proxy \
  --url-map=l7-ilb-gke-map \
  --url-map-region=us-west1 \
  --region=us-west1

    For HTTPS:

    See SSL Certificates for information on how to create SSL certificate resources. Google-managed certificates aren't currently supported with internal HTTP(S) load balancers.

    Assign your filepaths to variable names.

    `export LB_CERT=path to PEM-formatted file`

    `export LB_PRIVATE_KEY=path to PEM-formatted file`

    Create a regional SSL certificate using the gcloud compute ssl-certificates create command.

    gcloud compute ssl-certificates create

    gcloud compute ssl-certificates create l7-ilb-cert \
  --certificate=$LB_CERT \
  --private-key=$LB_PRIVATE_KEY \
  --region=us-west1

    Use the regional SSL certificate to create a target proxy with the gcloud compute target-https-proxies create command.

    gcloud compute target-https-proxies create 

    gcloud compute target-https-proxies create l7-ilb-gke-proxy \
  --url-map=l7-ilb-gke-map \
  --region=us-west1 \
  --ssl-certificates=l7-ilb-cert

  6. Create the forwarding rule.

    For custom networks, you must reference the subnet in the forwarding rule. Note that this is the VM subnet, not the proxy subnet.

    For HTTP:

    Use the gcloud compute forwarding-rules create command with the correct flags.

    gcloud compute forwarding-rules create l7-ilb-gke-forwarding-rule \
  --load-balancing-scheme=INTERNAL_MANAGED \
  --network=lb-network \
  --subnet=backend-subnet \
  --address=10.1.2.199 \
  --ports=80 \
  --region=us-west1 \
  --target-http-proxy=l7-ilb-gke-proxy \
  --target-http-proxy-region=us-west1

    For HTTPS:

    Use the gcloud compute forwarding-rules create command with the correct flags.

    gcloud compute forwarding-rules create l7-ilb-gke-forwarding-rule \
  --load-balancing-scheme=INTERNAL_MANAGED \
  --network=lb-network \
  --subnet=backend-subnet \
  --address=10.1.2.199 \
  --ports=443 \
  --region=us-west1 \
  --target-https-proxy=l7-ilb-gke-proxy \
  --target-https-proxy-region=us-west1

api

Create the health check by making a POST request to the regionHealthChecks.insert method, replacing [project-id] with your project ID.

POST https://compute.googleapis.com/compute/v1/projects/[project-id]/regions/{region}/healthChecks
{
  "name": "l7-ilb-gke-basic-check",
  "type": "HTTP",
  "httpHealthCheck": {
    "portSpecification": "USE_SERVING_PORT"
  }
}

Create the regional backend service by making a POST request to the regionBackendServices.insert method, replacing [project-id] with your project ID and [neg-name] with the name of the NEG that you created.

POST https://www.googleapis.com/compute/v1/projects/[project-id]/regions/us-west1/backendServices
{
  "name": "l7-ilb-gke-backend-service",
  "backends": [
    {
      "group": "https://www.googleapis.com/compute/v1/projects/[project-id]/zones/us-west1-b/networkEndpointGroups/[neg-name]",
      "balancingMode": "RATE",
      "maxRatePerEndpoint": 5
    }
  ],
  "healthChecks": [
    "projects/[project-id]/regions/us-west1/healthChecks/l7-ilb-gke-basic-check"
  ],
  "loadBalancingScheme": "INTERNAL_MANAGED"
}

Create the URL map by making a POST request to the regionUrlMaps.insert method, replacing [project-id] with your project ID.

POST https://compute.googleapis.com/compute/v1/projects/[project-id]/regions/us-west1/urlMaps
{
  "name": "l7-ilb-gke-map",
  "defaultService": "projects/[project-id]/regions/us-west1/backendServices/l7-ilb-gke-backend-service"
}

Create the target HTTP proxy by making a POST request to the regionTargetHttpProxies.insert method, replacing [project-id] with your project ID.

POST https://www.googleapis.com/compute/v1/projects/[project-id]/regions/us-west1/targetHttpProxy
{
  "name": "l7-ilb-gke-proxy",
  "urlMap": "projects/[project-id]/global/urlMaps/l7-ilb-gke-map",
  "region": "us-west1"
}

Create the forwarding rule by making a POST request to the forwardingRules.insert method, replacing [project-id] with your project ID.

POST https://www.googleapis.com/compute/v1/projects/[project-id]/regions/us-west1/forwardingRules
{
  "name": "l7-ilb-gke-forwarding-rule",
  "IPAddress": "10.1.2.199",
  "IPProtocol": "TCP",
  "portRange": "80-80",
  "target": "projects/[project-id]/regions/us-west1/targetHttpProxies/l7-ilb-gke-proxy",
  "loadBalancingScheme": "INTERNAL_MANAGED",
  "subnetwork": "projects/[project-id]/regions/us-west1/subnetworks/backend-subnet",
  "network": "projects/[project-id]/global/networks/lb-network",
  "networkTier": "PREMIUM",
}

Testing

Creating a VM instance in the zone to test connectivity

gcloud compute instances create l7-ilb-client-us-west1-b \
    --image-family=debian-9 \
    --image-project=debian-cloud \
    --zone=us-west1-b \
    --network=lb-network \
    --subnet=backend-subnet \
    --tags=l7-ilb-client,allow-ssh

Testing the load balancer

Log in to the client instance to that HTTP(S) services on the backends are reachable via the internal HTTP(S) load balancer's forwarding rule IP address, and traffic is being load balanced among endpoints in the NEG.

Connecting via SSH to each client instance

gcloud compute ssh l7-ilb-client-us-west1-b \
    --zone=us-west1-b

Verifying that the IP is serving its hostname

curl 10.1.2.199

For HTTPS testing, replace curl with:

curl -k -s 'https://test.example.com:443' --connect-to test.example.com:443:10.1.2.199:443

The -k flag causes curl to skip certificate validation.

Running 100 requests and confirming that they are load balanced

For HTTP:

{
RESULTS=
for i in {1..100}
do
    RESULTS="$RESULTS:$(curl --silent 10.1.2.199)"
done
echo "***"
echo "*** Results of load-balancing to 10.1.2.199: "
echo "***"
echo "$RESULTS" | tr ':' '\n' | grep -Ev "^$" | sort | uniq -c
echo
}

For HTTPS:

{
RESULTS=
for i in {1..100}
do
    RESULTS="$RESULTS:$(curl -k -s 'https://test.example.com:443' --connect-to test.example.com:443:10.1.2.199:443
)"
done
echo "***"
echo "*** Results of load-balancing to 10.1.2.199: "
echo "***"
echo "$RESULTS" | tr ':' '\n' | grep -Ev "^$" | sort | uniq -c
echo
}

Note that multiple proxies perform the load balancing, one for each curl command, and they don't coordinate their selection of backends. Therefore, in this test, the backends don't receive the same number of requests. However, over the long term (in other words, thousands to millions of requests), the fraction of requests received by each backend approaches an equal distribution.

What's next