Using container-native load balancing

This page explains how to use container-native load balancing in Google Kubernetes Engine.

Overview

Container-native load balancing enables HTTP(S) load balancers to target Pods directly and to evenly distribute their traffic to Pods.

Container-native load balancing leverages a data model called network endpoint groups (NEGs), collections of network endpoints represented by IP-port pairs.

Benefits

Container-native load balancing offers the following benefits:

Pods are first-class citizens for load balancing

kube-proxy configures nodes' iptables rules to distribute traffic to Pods. Without container-native load balancing, load balancer traffic travels to the node instance groups and gets routed via iptables rules to Pods which might or might not be in the same node. With container-native load balancing, load balancer traffic is distributed directly to the Pods which should receive the traffic, eliminating the extra network hop. Container-native load balancing also helps with improved health checking since it targets Pods directly.

Diagram comparing default behavior (left) with container-native load balancer behavior.

Improved network performance

Because the container-native load balancer talks directly with the Pods, and connections have fewer network hops, both latency and throughput are improved.

Increased visibility

With container-native load balancing, you have visibility into the round-trip time (RTT) from the client to the HTTP(S) load balancer, including Stackdriver UI support. This makes troubleshooting your services at the NEG-level easier.

Support for HTTP(S) Load Balancing features

Container-native load balancing offer native support in Google Kubernetes Engine for several features of HTTP(S) Load Balancing, such as integration with GCP services like Google Cloud Armor, Cloud CDN, and Identity-Aware Proxy. It also features load balancing algorithms for accurate traffic distribution.

Support for Traffic Director

The NEG data model is required to use Traffic Director, Google Cloud's fully managed traffic control plane for service mesh.

Pod readiness

For relevant Pods, the corresponding Ingress controller manages a readiness gate of type cloud.google.com/load-balancer-neg-ready. The Ingress controller polls the load balancer's health check status, which includes the health of all endpoints in the NEG. When the load balancer's health check status indicates that the endpoint corresponding to a particular Pod is healthy, the Ingress controller sets the Pod's readiness gate value to True. The kubelet running on each Node then computes the Pod's effective readiness, considering both the value of this readiness gate and, if defined, the Pod's readiness probe.

For container-native load balancing, Pod readiness gates are automatically enabled in:

• v1.13 GKE clusters running v1.13.8 and higher
• v1.14 GKE clusters running v1.14.4 and higher

Readiness gates control the rate of a rolling update. GKE versions listed above automatically add readiness gates to Pods. When you initiate a rolling update, as GKE creates new Pods, an endpoint for each new Pod is added to a NEG. When the endpoint is healthy from the perspective of the load balancer, the Ingress controller sets the readiness gate to True. Thus, a newly created Pod must at least pass its readiness gate before GKE removes an old Pod. This ensures that the corresponding endpoint for the Pod has already passed the load balancer's health check and that the backend capacity is maintained.

If a Pod's readiness gate never indicates that the Pod is ready, due to a bad container image or a misconfigured load balancer health check, the load balancer won't direct traffic to the new Pod. If such a failure occurs while rolling out an updated Deployment, the rollout stalls after attempting to create one new Pod because that Pod's readiness gate is never True. See the troubleshooting section for information on how to detect and fix this situation.

Without container-native load balancing and readiness gates, GKE can't detect if a load balancer's endpoints are healthy before marking Pods as ready. In previous Kubernetes versions, you control the rate that Pods are removed and replaced by specifying a delay period (minReadySeconds in the Deployment specification).

Requirements

Container-native load balancers on Google Kubernetes Engine have the following requirements:

Google Kubernetes Engine v1.13.8 or v1.14.4

Container-native load balancers are generally available in:

• v1.13 GKE clusters running v1.13.8 and higher

• v1.14 GKE clusters running v1.14.4 and higher

VPC-native

To use container-native load balancing, clusters must be VPC-native. To learn more, refer to Creating VPC-native clusters using Alias IPs.

HTTP load balancing

To use container-native load balancing, your cluster must have HTTP load-balancing enabled. GKE clusters have HTTP load-balancing enabled by default; you must not disable it.

Restrictions

Container-native load balancers do not work with legacy networks.

Limitations

Container-native load balancers do not support internal load balancers or network load balancers.

Pricing

You are charged for the HTTP(S) load balancer provisioned by the Ingress that you create in this guide. For load balancer pricing information, refer to Load balancing and forwarding rules on the Compute Engine pricing page.

Using container-native load balancing

The following sections walk you through a container-native load balancing configuration on Google Kubernetes Engine.

Creating a VPC-native cluster

To use container-native load balancing, you must create a cluster with alias IPs enabled.

For example, the following command creates a cluster, neg-demo-cluster, with an auto-provisioned subnetwork in zone us-central1-a:

gcloud container clusters create neg-demo-cluster \
    --enable-ip-alias \
    --create-subnetwork="" \
    --network=default \
    --zone=us-central1-a

Creating a Deployment

Next, deploy a workload to the cluster.

The following sample Deployment, neg-demo-app, runs a single instance of a containerized HTTP server. We recommend you use workloads that use Pod Readiness feedback. See the Pod readiness section above for more information and for GKE version requirements.

using Pod readiness feedback

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    run: neg-demo-app # Label for the Deployment
  name: neg-demo-app # Name of Deployment
spec:
  selector:
    matchLabels:
      run: neg-demo-app
  template: # Pod template
    metadata:
      labels:
        run: neg-demo-app # Labels Pods from this Deployment
    spec: # Pod specification; each Pod created by this Deployment has this specification
      containers:
      - image: k8s.gcr.io/serve_hostname:v1.4 # Application to run in Deployment's Pods
        name: hostname # Container name
  

using hardcoded delay

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    run: neg-demo-app # Label for the Deployment
  name: neg-demo-app # Name of Deployment
spec:
  minReadySeconds: 60 # Number of seconds to wait after a Pod is created and its status is Ready
  selector:
    matchLabels:
      run: neg-demo-app
  template: # Pod template
    metadata:
      labels:
        run: neg-demo-app # Labels Pods from this Deployment
    spec: # Pod specification; each Pod created by this Deployment has this specification
      containers:
      - image: k8s.gcr.io/serve_hostname:v1.4 # Application to run in Deployment's Pods
        name: hostname # Container name
      # Note: The following line is necessary only on clusters running GKE v1.11 and lower.
      # For details, see https://cloud.google.com/kubernetes-engine/docs/how-to/container-native-load-balancing#align_rollouts
      terminationGracePeriodSeconds: 60 # Number of seconds to wait for connections to terminate before shutting down Pods
  

In this Deployment, each container runs an HTTP server. The HTTP server simply returns the hostname of the application server (the name of the Pod on which the server runs) as a response.

Save this manifest as neg-demo-app.yaml, then create the Deployment by running the following command:

kubectl apply -f neg-demo-app.yaml

Creating a Service for a container-native load balancer

After you have created a Deployment, you need to group its Pods into a Service.

The following sample Service, neg-demo-svc, targets the sample Deployment you created in the previous section:

apiVersion: v1
kind: Service
metadata:
  name: neg-demo-svc # Name of Service
  annotations:
    cloud.google.com/neg: '{"ingress": true}' # Creates a NEG after an Ingress is created
spec: # Service's specification
  type: NodePort
  selector:
    run: neg-demo-app # Selects Pods labelled run: neg-demo-app
  ports:
  - port: 80 # Service's port
    protocol: TCP
    targetPort: 9376

The Service's annotation, cloud.google.com/neg: '{"ingress": true}', enables container-native load balancing. However, the load balancer is not created until you create an Ingress for the Service.

Save this manifest as neg-demo-svc.yaml, then create the Service by running the following command:

kubectl apply -f neg-demo-svc.yaml

Creating an Ingress for the Service

The following sample Ingress, neg-demo-ing, targets the Service you created:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: neg-demo-ing
spec:
  backend:
    serviceName: neg-demo-svc # Name of the Service targeted by the Ingress
    servicePort: 80 # Should match the port used by the Service

Save this manifest as neg-demo-ing.yaml, then create the Ingress by running the following command:

kubectl apply -f neg-demo-ing.yaml

Upon creating the Ingress, an HTTP(S) load balancer is created in the project, and NEGs are created in each zone in which the cluster runs. The endpoints in the NEG and the endpoints of the Service are kept in sync.

Verifying the Ingress

After you have deployed a workload, grouped its Pods into a Service, and created an Ingress for the Service, you should verify that the Ingress has provisioned the container-native load balancer successfully.

To retrieve the status of the Ingress, run the following command:

kubectl describe ingress neg-demo-ing

In the command output, look for ADD and CREATE events:

Events:
Type     Reason   Age                From                     Message
----     ------   ----               ----                     -------
Normal   ADD      16m                loadbalancer-controller  default/neg-demo-ing
Normal   Service  4s                 loadbalancer-controller  default backend set to neg-demo-svc:32524
Normal   CREATE   2s                 loadbalancer-controller  ip: 192.0.2.0

Testing load balancer functionality

The following sections explain how you can test the functionality of a container-native load balancer.

Visit Ingress IP address

Wait several minutes for the HTTP(S) load balancer to be configured.

You can verify that the container-native load balancer is functioning by visiting the Ingress' IP address.

To get the Ingress IP address, run the following command:

kubectl get ingress neg-demo-ing

In the command output, the Ingress' IP address is displayed in the ADDRESS column. Visit the IP address in a web browser.

Check backend service health status

You can also get the health status of the load balancer's backend service.

First, get a list of the backend services running in your project:

gcloud compute backend-services list

Copy of the name of the backend that includes the name of the Service, such as neg-demo-svc. Then, get the health status of the backend service:

gcloud compute backend-services get-health [BACKEND_SERVICE_NAME] --global

Verifying Ingress functionality

Another way you can test that the load balancer functions as expected is by scaling the sample Deployment, sending test requests to the Ingress, and verifying that the correct number of replicas respond.

The following command scales the neg-demo-app Deployment from one instance to two instances:

kubectl scale deployment neg-demo-app --replicas 2

Wait a few minutes for the rollout to complete. To verify that the rollout is complete, run the following command:

kubectl get deployment neg-demo-app

In the command output, verify that there are two available replicas:

NAME           DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
neg-demo-app   2         2         2            2           26m

Then, run the following command to count the number of distinct responses from the load balancer:

for i in `seq 1 100`; do \
  curl --connect-timeout 1 -s [IP_ADDRESS] && echo; \
done  | sort | uniq -c

where [IP_ADDRESS] is the Ingress' IP address. You can get the Ingress' IP address from kubectl describe ingress neg-demo-ing.

Note: If this command returns a 404 error, wait a few minutes for the load balancer to come up, then try again.

In the command output, observe that the number of distinct responses is the same as the number of replicas, indicating that all backend Pods are serving traffic:

44 neg-demo-app-7f7dfd7bc6-dcn95
56 neg-demo-app-7f7dfd7bc6-jrmzf

Cleaning up

After completing the tasks on this page, follow these steps to remove the resources to prevent unwanted charges incurring on your account:

Delete the cluster

gcloud

gcloud container clusters delete neg-demo-cluster

Console

1. Visit the Google Kubernetes Engine menu in Cloud Console.

   Visit the Google Kubernetes Engine menu

2. Select neg-demo-cluster.

3. Click Delete.

Troubleshooting

Use the techniques below to verify your networking configuration. The following sections explain how to resolve specific issues related to container-native load balancing.

• See the load balancing documentation for how to list your network endpoint groups.

• You can find the name and zones of the NEG that corresponds to a service in the neg-status annotation of the service. Get the Service specification with:

  kubectl get svc svc-name -o yaml

  The metadata:annotations:cloud.google.com/neg-status annotation lists the name of service's corresponding NEG and the zones of the NEG.

• You can check the health of the backend service that corresponds to a NEG with the following command:

  gcloud compute backend-services [--project PROJECT_NAME] \
    get-health BACKEND_SERVICE_NAME --global
  

  The backend service has the same name as its NEG.

• To print a service's event logs:

  kubectl describe svc [SERVICE_NAME]
  

  The service's name string includes the name and namespace of the corresponding GKE Service.

Cannot create a cluster with alias IPs

Symptoms

When you attempt to create a cluster with alias IPs, you might encounter the following error:

ResponseError: code=400, message=IP aliases cannot be used with a legacy network.

Potential causes

You encounter this error if you attempt to create a cluster with alias IPs that also uses a legacy network.

Resolution

Ensure that you do not create a cluster with alias IPs and a legacy network enabled simultaneously. For more information about using alias IPs, refer to Creating VPC-native clusters using Alias IPs.

Traffic does not reach endpoints

Symptoms

502 errors or rejected connections.

Potential causes

New endpoints generally become reachable after attaching them to the load balancer, provided that they respond to health checks. You might encounter 502 errors or rejected connections if traffic cannot reach the endpoints.

502 errors and rejected connections can also be caused by a container that doesn't handle SIGTERM. If a container doesn't explicitly handle SIGTERM, it immediately terminates and stops handling requests. The load balancer continues to send incoming traffic to the terminated container, leading to errors.

Resolution

Configure containers to handle SIGTERM and continue responding to requests throughout the termination grace period (30 seconds by default). Configure Pods to begin failing health checks when they receive SIGTERM. This signals the load balancer to stop sending traffic to the Pod while endpoint deprograming is in progress.

See the documentation on Pod termination and this post about Pod termination best practices for more information.

To troubleshoot traffic not reaching the endpoints, verify that firewall rules allow incoming TCP traffic to your endpoints in the 130.211.0.0/22 and 35.191.0.0/16 ranges. To learn more, refer to Adding Health Checks in the Cloud Load Balancing documentation.

View the backend services in your project. The name string of the relevant backend service includes the name and namespace of the corresponding Google Kubernetes Engine Service:

gcloud compute backend-services list

Retrieve the backend health status from the backend service:

gcloud compute backend-services get-health [BACKEND_SERVICE_NAME]

If all backends are unhealthy, your firewall, Ingress, or Service might be misconfigured.

If some backends are unhealthy for a short period of time, network programming latency might be the cause.

If some backends do not appear in the list of backend services, programming latency might be the cause. You can verify this by running the following command, where [NEG] is the name of the backend service. (NEGs and backend services share the same name):

gcloud compute network-endpoint-groups list-network-endpoints [NEG]

Check if all the expected endpoints are in the NEG.

Stalled rollout

Symptoms

Rolling out an updated Deployment stalls, and the number of up-to-date replicas does not match the desired number of replicas.

Potential causes

The deployment's health checks are failing. The container image might be bad or the health check might be misconfigured. The rolling replacement of Pods waits until the newly started Pod passes its Pod readiness gate. This only occurs if the Pod is responding to load balancer health checks. If the Pod does not respond, or if the health check is misconfigured, the readiness gate conditions can't be met and the rollout can't continue.

If you're using kubectl 1.13 or higher, you can check the status of a Pod's readiness gates with the following command:

kubectl get my-Pod -o wide

Check the READINESS GATES column.

This column doesn't exist in kubectl 1.12 and lower. A Pod that is marked as being in the READY state may have a failed readiness gate. To verify this, use the following command:

kubectl get my-pod -o yaml

The readiness gates and their status are listed in the output.

Resolution

Verify that the container image in your Deployment's Pod specification is functioning correctly and is able to respond to health checks. Verify that the health checks are correctly configured.

Known issues

Container-native load balancing on Google Kubernetes Engine has the following known issues:

Incomplete garbage collection

Google Kubernetes Engine garbage collects container-native load balancers every two minutes. If a cluster is deleted before load balancers are fully removed, you need to manually delete the load balancer's NEGs.

View the NEGs in your project by running the following command:

gcloud compute network-endpoint-groups list

In the command output, look for the relevant NEGs.

To delete a NEG, run the following command, where [NEG] is the name of the NEG:

gcloud compute network-endpoint-groups delete [NEG]

Aligning workload rollouts with endpoint propagation

Note: This issue does not occur in clusters that use Pod readiness feedback to manage workload rollouts. See the Pod readiness section for more information.

When you deploy a workload to your cluster, or when you update an existing workload, the container-native load balancer can take longer to propagate new endpoints than it takes to finish the workload rollout. The sample Deployment that you deploy in this guide uses two fields to align its rollout with the propagation of endpoints: terminationGracePeriodSeconds and minReadySeconds.

terminationGracePeriodSeconds allows the Pod to shut down gracefully by waiting for connections to terminate after a Pod is scheduled for deletion.

minReadySeconds adds a latency period after a Pod is created. You specify a minimum number of seconds for which a new Pod should be in Ready status, without any of its containers crashing, for the Pod to be considered available.

You should configure your workloads' minReadySeconds and terminationGracePeriodSeconds values to be 60 seconds or higher to ensure that the service is not disrupted due to workload rollouts.

terminationGracePeriodSeconds is available in all Pod specifications, and minReadySeconds is available for Deployments and DaemonSets.

To learn more about fine-tuning rollouts, refer to RollingUpdateStrategy.

What's next

• Learn more about NEGs.
• Learn more about VPC-native clusters.
• Learn more about HTTP(S) Load Balancing.
• Watch a KubeCon talk about Pod readiness gates.