This page explains how to use container-native load balancing in Google Kubernetes Engine (GKE). Container-native load balancing allows load balancers to target Kubernetes Pods directly and to evenly distribute traffic to Pods.
See Container-native load balancing for information on the benefits, requirements, and limitations of container-native load balancing.
Using container-native load balancing
The following sections walk you through a container-native load balancing
configuration on GKE. Note that for GKE clusters 1.17 and up and under certain conditions, container-native load balancing is default and does not require an explicit cloud.google.com/neg: '{"ingress": true}'
Service annotation.
Creating a VPC-native cluster
To use container-native load balancing, you must create a cluster with alias IPs enabled.
For example, the following command creates a cluster, neg-demo-cluster
, with
an auto-provisioned subnetwork in zone us-central1-a
:
gcloud container clusters create neg-demo-cluster \ --enable-ip-alias \ --create-subnetwork="" \ --network=default \ --zone=us-central1-a
Creating a Deployment
Next, deploy a workload to the cluster.
The following sample Deployment, neg-demo-app
, runs a single instance of a
containerized HTTP server. We recommend you use workloads that use Pod Readiness
feedback if it is available in the version of GKE you are
using. See the Pod
readiness
for more information and for GKE version requirements.
Consider upgrading your
cluster to use Pod Readiness
feedback.
using Pod readiness feedback
apiVersion: apps/v1 kind: Deployment metadata: labels: run: neg-demo-app # Label for the Deployment name: neg-demo-app # Name of Deployment spec: selector: matchLabels: run: neg-demo-app template: # Pod template metadata: labels: run: neg-demo-app # Labels Pods from this Deployment spec: # Pod specification; each Pod created by this Deployment has this specification containers: - image: k8s.gcr.io/serve_hostname:v1.4 # Application to run in Deployment's Pods name: hostname # Container name ports: - containerPort: 9376 protocol: TCP
using hardcoded delay
apiVersion: apps/v1 kind: Deployment metadata: labels: run: neg-demo-app # Label for the Deployment name: neg-demo-app # Name of Deployment spec: minReadySeconds: 60 # Number of seconds to wait after a Pod is created and its status is Ready selector: matchLabels: run: neg-demo-app template: # Pod template metadata: labels: run: neg-demo-app # Labels Pods from this Deployment spec: # Pod specification; each Pod created by this Deployment has this specification containers: - image: k8s.gcr.io/serve_hostname:v1.4 # Application to run in Deployment's Pods name: hostname # Container name # Note: The following line is necessary only on clusters running GKE v1.11 and lower. # For details, see https://cloud.google.com/kubernetes-engine/docs/how-to/container-native-load-balancing#align_rollouts ports: - containerPort: 9376 protocol: TCP terminationGracePeriodSeconds: 60 # Number of seconds to wait for connections to terminate before shutting down Pods
In this Deployment, each container runs an HTTP server. The HTTP server simply returns the hostname of the application server (the name of the Pod on which the server runs) as a response.
Save this manifest as neg-demo-app.yaml
, then create the Deployment by running
the following command:
kubectl apply -f neg-demo-app.yaml
Creating a Service for a container-native load balancer
After you have created a Deployment, you need to group its Pods into a Service.
The following sample Service, neg-demo-svc
, targets the sample Deployment you
created in the previous section:
apiVersion: v1 kind: Service metadata: name: neg-demo-svc # Name of Service annotations: cloud.google.com/neg: '{"ingress": true}' # Creates a NEG after an Ingress is created spec: # Service's specification type: ClusterIP selector: run: neg-demo-app # Selects Pods labelled run: neg-demo-app ports: - port: 80 # Service's port protocol: TCP targetPort: 9376
The Service's annotation, cloud.google.com/neg: '{"ingress": true}'
, enables
container-native load balancing. However, the load balancer is not created until
you create an Ingress for the Service.
Save this manifest as neg-demo-svc.yaml
, then create the Service by running
the following command:
kubectl apply -f neg-demo-svc.yaml
Creating an Ingress for the Service
The following sample Ingress, neg-demo-ing
, targets the Service
you created:
apiVersion: networking.k8s.io/v1beta1 kind: Ingress metadata: name: neg-demo-ing spec: backend: serviceName: neg-demo-svc # Name of the Service targeted by the Ingress servicePort: 80 # Should match the port used by the Service
Save this manifest as neg-demo-ing.yaml
, then create the Ingress by running
the following command:
kubectl apply -f neg-demo-ing.yaml
Upon creating the Ingress, an HTTP(S) load balancer is created in the project, and NEGs are created in each zone in which the cluster runs. The endpoints in the NEG and the endpoints of the Service are kept in sync.
Verifying the Ingress
After you have deployed a workload, grouped its Pods into a Service, and created an Ingress for the Service, you should verify that the Ingress has provisioned the container-native load balancer successfully.
To retrieve the status of the Ingress, run the following command:
kubectl describe ingress neg-demo-ing
In the command output, look for ADD
and CREATE
events:
Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal ADD 16m loadbalancer-controller default/neg-demo-ing Normal Service 4s loadbalancer-controller default backend set to neg-demo-svc:32524 Normal CREATE 2s loadbalancer-controller ip: 192.0.2.0
Testing load balancer functionality
The following sections explain how you can test the functionality of a container-native load balancer.
Visit Ingress IP address
Wait several minutes for the HTTP(S) load balancer to be configured.
You can verify that the container-native load balancer is functioning by visiting the Ingress' IP address.
To get the Ingress IP address, run the following command:
kubectl get ingress neg-demo-ing
In the command output, the Ingress' IP address is displayed in the ADDRESS
column. Visit the IP address in a web browser.
Check backend service health status
You can also get the health status of the load balancer's backend service.
First, get a list of the backend services running in your project:
gcloud compute backend-services list
Copy of the name of the backend that includes the name of the Service, such as
neg-demo-svc
. Then, get the health status of the backend service:
gcloud compute backend-services get-health backend-service-name --global
Verifying Ingress functionality
Another way you can test that the load balancer functions as expected is by scaling the sample Deployment, sending test requests to the Ingress, and verifying that the correct number of replicas respond.
The following command scales the neg-demo-app
Deployment from one instance to
two instances:
kubectl scale deployment neg-demo-app --replicas 2
Wait a few minutes for the rollout to complete. To verify that the rollout is complete, run the following command:
kubectl get deployment neg-demo-app
In the command output, verify that there are two available replicas:
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE neg-demo-app 2 2 2 2 26m
Then, run the following command to count the number of distinct responses from the load balancer:
for i in `seq 1 100`; do \ curl --connect-timeout 1 -s ip-address && echo; \ done | sort | uniq -c
where ip-address is the Ingress' IP address. You can get the
Ingress' IP address from kubectl describe ingress neg-demo-ing
.
In the command output, observe that the number of distinct responses is the same as the number of replicas, indicating that all backend Pods are serving traffic:
44 neg-demo-app-7f7dfd7bc6-dcn95 56 neg-demo-app-7f7dfd7bc6-jrmzf
Cleaning up
After completing the tasks on this page, follow these steps to remove the resources to prevent unwanted charges incurring on your account:
Delete the cluster
gcloud
gcloud container clusters delete neg-demo-cluster
Console
Visit the Google Kubernetes Engine menu in Cloud Console.
Select neg-demo-cluster.
Click Delete.
Troubleshooting
Use the techniques below to verify your networking configuration. The following sections explain how to resolve specific issues related to container-native load balancing.
See the load balancing documentation for how to list your network endpoint groups.
You can find the name and zones of the NEG that corresponds to a service in the
neg-status
annotation of the service. Get the Service specification with:kubectl get svc svc-name -o yaml
The
metadata:annotations:cloud.google.com/neg-status
annotation lists the name of service's corresponding NEG and the zones of the NEG.You can check the health of the backend service that corresponds to a NEG with the following command:
gcloud compute backend-services [--project project-name] \ get-health backend-service-name --global
The backend service has the same name as its NEG.
To print a service's event logs:
kubectl describe svc service-name
The service's name string includes the name and namespace of the corresponding GKE Service.
Cannot create a cluster with alias IPs
- Symptoms
- When you attempt to create a cluster with alias IPs, you might encounter the
following error:
ResponseError: code=400, message=IP aliases cannot be used with a legacy network.
- Potential causes
- You encounter this error if you attempt to create a cluster with alias IPs that also uses a legacy network.
- Resolution
- Ensure that you do not create a cluster with alias IPs and a legacy network enabled simultaneously. For more information about using alias IPs, refer to Creating VPC-native clusters using Alias IPs.
Traffic does not reach endpoints
- Symptoms
- 502 errors or rejected connections.
- Potential causes
New endpoints generally become reachable after attaching them to the load balancer, provided that they respond to health checks. You might encounter 502 errors or rejected connections if traffic cannot reach the endpoints.
502 errors and rejected connections can also be caused by a container that doesn't handle
SIGTERM
. If a container doesn't explicitly handleSIGTERM
, it immediately terminates and stops handling requests. The load balancer continues to send incoming traffic to the terminated container, leading to errors.The container native load balancer only has one backend endpoint. During a rolling update, the old endpoint gets deprogrammed before the new endpoint gets programmed.
- Resolution
Configure containers to handle
SIGTERM
and continue responding to requests throughout the termination grace period (30 seconds by default). Configure Pods to begin failing health checks when they receiveSIGTERM
. This signals the load balancer to stop sending traffic to the Pod while endpoint deprograming is in progress.If your application does not gracefully shut down and stops responding to requests when receiving a
SIGTERM
, the preStop hook can be used to handleSIGTERM
and keep serving traffic while endpoint deprograming is in progress.lifecycle: preStop: exec: # if SIGTERM triggers a quick exit; keep serving traffic instead command: ["sleep","60"]
See the documentation on Pod termination and this post about Pod termination best practices for more information.
If your load balancer backend only has one instance, please configure the roll out strategy to avoid tearing down the only instance before the new instance is fully programmed. For application pods managed by
Deployment
workload, this can be achieved by configuring rollout strategy withmaxUnavailable
parameter equal to0
.strategy: rollingUpdate: maxSurge: 1 maxUnavailable: 0
To troubleshoot traffic not reaching the endpoints, verify that firewall rules allow incoming TCP traffic to your endpoints in the
130.211.0.0/22
and35.191.0.0/16
ranges. To learn more, refer to Adding Health Checks in the Cloud Load Balancing documentation.View the backend services in your project. The name string of the relevant backend service includes the name and namespace of the corresponding Google Kubernetes Engine Service:
gcloud compute backend-services list
Retrieve the backend health status from the backend service:
gcloud compute backend-services get-health backend-service-name
If all backends are unhealthy, your firewall, Ingress, or Service might be misconfigured.
If some backends are unhealthy for a short period of time, network programming latency might be the cause.
If some backends do not appear in the list of backend services, programming latency might be the cause. You can verify this by running the following command, where neg is the name of the backend service. (NEGs and backend services share the same name):
gcloud compute network-endpoint-groups list-network-endpoints neg
Check if all the expected endpoints are in the NEG.
Stalled rollout
- Symptoms
- Rolling out an updated Deployment stalls, and the number of up-to-date replicas does not match the desired number of replicas.
- Potential causes
The deployment's health checks are failing. The container image might be bad or the health check might be misconfigured. The rolling replacement of Pods waits until the newly started Pod passes its Pod readiness gate. This only occurs if the Pod is responding to load balancer health checks. If the Pod does not respond, or if the health check is misconfigured, the readiness gate conditions can't be met and the rollout can't continue.
If you're using
kubectl
1.13 or higher, you can check the status of a Pod's readiness gates with the following command:kubectl get my-pod -o wide
Check the
READINESS GATES
column.This column doesn't exist in
kubectl
1.12 and lower. A Pod that is marked as being in theREADY
state may have a failed readiness gate. To verify this, use the following command:kubectl get my-pod -o yaml
The readiness gates and their status are listed in the output.
- Resolution
Verify that the container image in your Deployment's Pod specification is functioning correctly and is able to respond to health checks. Verify that the health checks are correctly configured.
Known issues
Container-native load balancing on Google Kubernetes Engine has the following known issues:
Incomplete garbage collection
Google Kubernetes Engine garbage collects container-native load balancers every two minutes. If a cluster is deleted before load balancers are fully removed, you need to manually delete the load balancer's NEGs.
View the NEGs in your project by running the following command:
gcloud compute network-endpoint-groups list
In the command output, look for the relevant NEGs.
To delete a NEG, run the following command, where neg is the name of the NEG:
gcloud compute network-endpoint-groups delete neg
Aligning workload rollouts with endpoint propagation
When you deploy a workload to your cluster, or when you update an existing
workload, the container-native load balancer can take longer to propagate new
endpoints than it takes to finish the workload rollout. The sample Deployment
that you deploy in this guide uses two fields to align its rollout with the
propagation of endpoints: terminationGracePeriodSeconds
and minReadySeconds
.
terminationGracePeriodSeconds
allows the Pod to
shut down gracefully by waiting for connections to terminate after a Pod is
scheduled for deletion.
minReadySeconds
adds a latency period after a Pod is
created. You specify a minimum number of seconds for which a new Pod should be
in Ready
status, without any of its containers crashing,
for the Pod to be considered available.
You should configure your workloads' minReadySeconds
and
terminationGracePeriodSeconds
values to be 60 seconds or higher to ensure
that the service is not disrupted due to workload rollouts.
terminationGracePeriodSeconds
is available in all Pod specifications, and
minReadySeconds
is available for Deployments and DaemonSets.
To learn more about fine-tuning rollouts, refer to RollingUpdateStrategy.
initialDelaySeconds
in Pod readinessProbe
not respected
You might expect the initialDelaySeconds
configuration in the Pod's
readinessProbe
to be respected by the container-native load balancer; however, readinessProbe
is implemented by kubelet, and the initialDelaySeconds
configuration controls
the kubelet health check, not the container-native load balancer. Container-native
load balancing has its own load balancing health check.
What's next
- Learn more about NEGs.
- Learn more about VPC-native clusters.
- Learn more about HTTP(S) Load Balancing.
- Watch a KubeCon talk about Pod readiness gates.