Container-native load balancing through Ingress

This page explains how to use container-native load balancing in Google Kubernetes Engine (GKE). Container-native load balancing allows load balancers to target Kubernetes Pods directly and to evenly distribute traffic to Pods.

See Container-native load balancing for information on the benefits, requirements, and limitations of container-native load balancing.

Using container-native load balancing

The following sections walk you through a container-native load balancing configuration on GKE. Note that for GKE clusters 1.17 and up and under certain conditions, container-native load balancing is default and does not require an explicit cloud.google.com/neg: '{"ingress": true}' Service annotation.

Creating a VPC-native cluster

To use container-native load balancing, you must create a cluster with alias IPs enabled.

For example, the following command creates a cluster, neg-demo-cluster, with an auto-provisioned subnetwork in zone us-central1-a:

gcloud container clusters create neg-demo-cluster \
    --enable-ip-alias \
    --create-subnetwork="" \
    --network=default \
    --zone=us-central1-a

Creating a Deployment

Next, deploy a workload to the cluster.

The following sample Deployment, neg-demo-app, runs a single instance of a containerized HTTP server. We recommend you use workloads that use Pod Readiness feedback if it is available in the version of GKE you are using. See the Pod readiness for more information and for GKE version requirements. Consider upgrading your cluster to use Pod Readiness feedback.

using Pod readiness feedback

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    run: neg-demo-app # Label for the Deployment
  name: neg-demo-app # Name of Deployment
spec:
  selector:
    matchLabels:
      run: neg-demo-app
  template: # Pod template
    metadata:
      labels:
        run: neg-demo-app # Labels Pods from this Deployment
    spec: # Pod specification; each Pod created by this Deployment has this specification
      containers:
      - image: k8s.gcr.io/serve_hostname:v1.4 # Application to run in Deployment's Pods
        name: hostname # Container name
        ports:
        - containerPort: 9376
          protocol: TCP

using hardcoded delay

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    run: neg-demo-app # Label for the Deployment
  name: neg-demo-app # Name of Deployment
spec:
  minReadySeconds: 60 # Number of seconds to wait after a Pod is created and its status is Ready
  selector:
    matchLabels:
      run: neg-demo-app
  template: # Pod template
    metadata:
      labels:
        run: neg-demo-app # Labels Pods from this Deployment
    spec: # Pod specification; each Pod created by this Deployment has this specification
      containers:
      - image: k8s.gcr.io/serve_hostname:v1.4 # Application to run in Deployment's Pods
        name: hostname # Container name
      # Note: The following line is necessary only on clusters running GKE v1.11 and lower.
      # For details, see https://cloud.google.com/kubernetes-engine/docs/how-to/container-native-load-balancing#align_rollouts
        ports:
        - containerPort: 9376
          protocol: TCP
      terminationGracePeriodSeconds: 60 # Number of seconds to wait for connections to terminate before shutting down Pods

In this Deployment, each container runs an HTTP server. The HTTP server simply returns the hostname of the application server (the name of the Pod on which the server runs) as a response.

Save this manifest as neg-demo-app.yaml, then create the Deployment by running the following command:

kubectl apply -f neg-demo-app.yaml

Creating a Service for a container-native load balancer

After you have created a Deployment, you need to group its Pods into a Service.

The following sample Service, neg-demo-svc, targets the sample Deployment you created in the previous section:

apiVersion: v1
kind: Service
metadata:
  name: neg-demo-svc # Name of Service
  annotations:
    cloud.google.com/neg: '{"ingress": true}' # Creates a NEG after an Ingress is created
spec: # Service's specification
  type: ClusterIP
  selector:
    run: neg-demo-app # Selects Pods labelled run: neg-demo-app
  ports:
  - port: 80 # Service's port
    protocol: TCP
    targetPort: 9376

The Service's annotation, cloud.google.com/neg: '{"ingress": true}', enables container-native load balancing. However, the load balancer is not created until you create an Ingress for the Service.

Save this manifest as neg-demo-svc.yaml, then create the Service by running the following command:

kubectl apply -f neg-demo-svc.yaml

Creating an Ingress for the Service

The following sample Ingress, neg-demo-ing, targets the Service you created:

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: neg-demo-ing
spec:
  backend:
    serviceName: neg-demo-svc # Name of the Service targeted by the Ingress
    servicePort: 80 # Should match the port used by the Service

Save this manifest as neg-demo-ing.yaml, then create the Ingress by running the following command:

kubectl apply -f neg-demo-ing.yaml

Upon creating the Ingress, an HTTP(S) load balancer is created in the project, and NEGs are created in each zone in which the cluster runs. The endpoints in the NEG and the endpoints of the Service are kept in sync.

Verifying the Ingress

After you have deployed a workload, grouped its Pods into a Service, and created an Ingress for the Service, you should verify that the Ingress has provisioned the container-native load balancer successfully.

To retrieve the status of the Ingress, run the following command:

kubectl describe ingress neg-demo-ing

In the command output, look for ADD and CREATE events:

Events:
Type     Reason   Age                From                     Message
----     ------   ----               ----                     -------
Normal   ADD      16m                loadbalancer-controller  default/neg-demo-ing
Normal   Service  4s                 loadbalancer-controller  default backend set to neg-demo-svc:32524
Normal   CREATE   2s                 loadbalancer-controller  ip: 192.0.2.0

Testing load balancer functionality

The following sections explain how you can test the functionality of a container-native load balancer.

Visit Ingress IP address

Wait several minutes for the HTTP(S) load balancer to be configured.

You can verify that the container-native load balancer is functioning by visiting the Ingress' IP address.

To get the Ingress IP address, run the following command:

kubectl get ingress neg-demo-ing

In the command output, the Ingress' IP address is displayed in the ADDRESS column. Visit the IP address in a web browser.

Check backend service health status

You can also get the health status of the load balancer's backend service.

First, get a list of the backend services running in your project:

gcloud compute backend-services list

Copy of the name of the backend that includes the name of the Service, such as neg-demo-svc. Then, get the health status of the backend service:

gcloud compute backend-services get-health backend-service-name --global

Verifying Ingress functionality

Another way you can test that the load balancer functions as expected is by scaling the sample Deployment, sending test requests to the Ingress, and verifying that the correct number of replicas respond.

The following command scales the neg-demo-app Deployment from one instance to two instances:

kubectl scale deployment neg-demo-app --replicas 2

Wait a few minutes for the rollout to complete. To verify that the rollout is complete, run the following command:

kubectl get deployment neg-demo-app

In the command output, verify that there are two available replicas:

NAME           DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
neg-demo-app   2         2         2            2           26m

Then, run the following command to count the number of distinct responses from the load balancer:

for i in `seq 1 100`; do \
  curl --connect-timeout 1 -s ip-address && echo; \
done  | sort | uniq -c

where ip-address is the Ingress' IP address. You can get the Ingress' IP address from kubectl describe ingress neg-demo-ing.

In the command output, observe that the number of distinct responses is the same as the number of replicas, indicating that all backend Pods are serving traffic:

44 neg-demo-app-7f7dfd7bc6-dcn95
56 neg-demo-app-7f7dfd7bc6-jrmzf

Cleaning up

After completing the tasks on this page, follow these steps to remove the resources to prevent unwanted charges incurring on your account:

Delete the cluster

gcloud

gcloud container clusters delete neg-demo-cluster

Console

  1. Visit the Google Kubernetes Engine menu in Cloud Console.

    Visit the Google Kubernetes Engine menu

  2. Select neg-demo-cluster.

  3. Click Delete.

Troubleshooting

Use the techniques below to verify your networking configuration. The following sections explain how to resolve specific issues related to container-native load balancing.

  • See the load balancing documentation for how to list your network endpoint groups.

  • You can find the name and zones of the NEG that corresponds to a service in the neg-status annotation of the service. Get the Service specification with:

    kubectl get svc svc-name -o yaml

    The metadata:annotations:cloud.google.com/neg-status annotation lists the name of service's corresponding NEG and the zones of the NEG.

  • You can check the health of the backend service that corresponds to a NEG with the following command:

    gcloud compute backend-services [--project project-name] \
  get-health backend-service-name --global

    The backend service has the same name as its NEG.

  • To print a service's event logs:

    kubectl describe svc service-name

    The service's name string includes the name and namespace of the corresponding GKE Service.

Cannot create a cluster with alias IPs

Symptoms
When you attempt to create a cluster with alias IPs, you might encounter the following error: 
ResponseError: code=400, message=IP aliases cannot be used with a legacy network.
Potential causes
You encounter this error if you attempt to create a cluster with alias IPs that also uses a legacy network.
Resolution
Ensure that you do not create a cluster with alias IPs and a legacy network enabled simultaneously. For more information about using alias IPs, refer to Creating VPC-native clusters using Alias IPs.

Traffic does not reach endpoints

Symptoms
502 errors or rejected connections.
Potential causes

New endpoints generally become reachable after attaching them to the load balancer, provided that they respond to health checks. You might encounter 502 errors or rejected connections if traffic cannot reach the endpoints.

502 errors and rejected connections can also be caused by a container that doesn't handle SIGTERM. If a container doesn't explicitly handle SIGTERM, it immediately terminates and stops handling requests. The load balancer continues to send incoming traffic to the terminated container, leading to errors.

The container native load balancer only has one backend endpoint. During a rolling update, the old endpoint gets deprogrammed before the new endpoint gets programmed.

Resolution

Configure containers to handle SIGTERM and continue responding to requests throughout the termination grace period (30 seconds by default). Configure Pods to begin failing health checks when they receive SIGTERM. This signals the load balancer to stop sending traffic to the Pod while endpoint deprograming is in progress.

If your application does not gracefully shut down and stops responding to requests when receiving a SIGTERM, the preStop hook can be used to handle SIGTERM and keep serving traffic while endpoint deprograming is in progress.

lifecycle:
  preStop:
    exec:
      # if SIGTERM triggers a quick exit; keep serving traffic instead
      command: ["sleep","60"]

See the documentation on Pod termination and this post about Pod termination best practices for more information.

If your load balancer backend only has one instance, please configure the roll out strategy to avoid tearing down the only instance before the new instance is fully programmed. For application pods managed by Deployment workload, this can be achieved by configuring rollout strategy with maxUnavailable parameter equal to 0.

strategy:
  rollingUpdate:
    maxSurge: 1
    maxUnavailable: 0

To troubleshoot traffic not reaching the endpoints, verify that firewall rules allow incoming TCP traffic to your endpoints in the 130.211.0.0/22 and 35.191.0.0/16 ranges. To learn more, refer to Adding Health Checks in the Cloud Load Balancing documentation.

View the backend services in your project. The name string of the relevant backend service includes the name and namespace of the corresponding Google Kubernetes Engine Service:

gcloud compute backend-services list

Retrieve the backend health status from the backend service:

gcloud compute backend-services get-health backend-service-name

If all backends are unhealthy, your firewall, Ingress, or Service might be misconfigured.

If some backends are unhealthy for a short period of time, network programming latency might be the cause.

If some backends do not appear in the list of backend services, programming latency might be the cause. You can verify this by running the following command, where neg is the name of the backend service. (NEGs and backend services share the same name):

gcloud compute network-endpoint-groups list-network-endpoints neg

Check if all the expected endpoints are in the NEG.

Stalled rollout

Symptoms
Rolling out an updated Deployment stalls, and the number of up-to-date replicas does not match the desired number of replicas.
Potential causes

The deployment's health checks are failing. The container image might be bad or the health check might be misconfigured. The rolling replacement of Pods waits until the newly started Pod passes its Pod readiness gate. This only occurs if the Pod is responding to load balancer health checks. If the Pod does not respond, or if the health check is misconfigured, the readiness gate conditions can't be met and the rollout can't continue.

If you're using kubectl 1.13 or higher, you can check the status of a Pod's readiness gates with the following command:

kubectl get my-pod -o wide

Check the READINESS GATES column.

This column doesn't exist in kubectl 1.12 and lower. A Pod that is marked as being in the READY state may have a failed readiness gate. To verify this, use the following command:

kubectl get my-pod -o yaml

The readiness gates and their status are listed in the output.

Resolution

Verify that the container image in your Deployment's Pod specification is functioning correctly and is able to respond to health checks. Verify that the health checks are correctly configured.

Known issues

Container-native load balancing on Google Kubernetes Engine has the following known issues:

Incomplete garbage collection

Google Kubernetes Engine garbage collects container-native load balancers every two minutes. If a cluster is deleted before load balancers are fully removed, you need to manually delete the load balancer's NEGs.

View the NEGs in your project by running the following command:

gcloud compute network-endpoint-groups list

In the command output, look for the relevant NEGs.

To delete a NEG, run the following command, where neg is the name of the NEG:

gcloud compute network-endpoint-groups delete neg

Aligning workload rollouts with endpoint propagation

When you deploy a workload to your cluster, or when you update an existing workload, the container-native load balancer can take longer to propagate new endpoints than it takes to finish the workload rollout. The sample Deployment that you deploy in this guide uses two fields to align its rollout with the propagation of endpoints: terminationGracePeriodSeconds and minReadySeconds.

terminationGracePeriodSeconds allows the Pod to shut down gracefully by waiting for connections to terminate after a Pod is scheduled for deletion.

minReadySeconds adds a latency period after a Pod is created. You specify a minimum number of seconds for which a new Pod should be in Ready status, without any of its containers crashing, for the Pod to be considered available.

You should configure your workloads' minReadySeconds and terminationGracePeriodSeconds values to be 60 seconds or higher to ensure that the service is not disrupted due to workload rollouts.

terminationGracePeriodSeconds is available in all Pod specifications, and minReadySeconds is available for Deployments and DaemonSets.

To learn more about fine-tuning rollouts, refer to RollingUpdateStrategy.

initialDelaySeconds in Pod readinessProbe not respected

You might expect the initialDelaySeconds configuration in the Pod's readinessProbe to be respected by the container-native load balancer; however, readinessProbe is implemented by kubelet, and the initialDelaySeconds configuration controls the kubelet health check, not the container-native load balancer. Container-native load balancing has its own load balancing health check.

What's next