This page shows you how to deploy an external LoadBalancer Service that builds a backend service-based external passthrough Network Load Balancer. Before reading this page, ensure that you're familiar with the following:
To learn more about external passthrough Network Load Balancers in general, see Backend service-based external passthrough Network Load Balancer.
Before you begin
Before you start, make sure you have performed the following tasks:
- Enable the Google Kubernetes Engine API. Enable Google Kubernetes Engine API
- If you want to use the Google Cloud CLI for this task,
install and then
initialize the
gcloud CLI. If you previously installed the gcloud CLI, get the latest
version by running
gcloud components update
.
Requirements
To create an external LoadBalancer Service, your GKE cluster must use version 1.25.5 or later. To use weighted load balancing, your cluster must use version 1.31.0-gke.1506000 or later.
The
HttpLoadBalancing
add-on must be enabled in your cluster. This add-on is enabled by default. It allows the cluster to manage load balancers which use backend services.
Choose a cluster
You can create a new cluster or choose an existing cluster that meets the requirements.
Create a new cluster
Autopilot
To create a new Autopilot cluster:
gcloud container clusters create-auto CLUSTER_NAME \
--release-channel=RELEASE_CHANNEL \
--cluster-version=VERSION \
--location=COMPUTE_LOCATION
Replace the following:
CLUSTER_NAME
: the name of the new cluster.RELEASE_CHANNEL
: the name of the GKE release channel for the cluster.VERSION
: the GKE version for the cluster.COMPUTE_LOCATION
: the Compute Engine region of the cluster.
Standard
To create a new Standard cluster:
gcloud container clusters create CLUSTER_NAME \
--release-channel=RELEASE_CHANNEL \
--cluster-version=VERSION \
--location=COMPUTE_LOCATION
Replace the following:
CLUSTER_NAME
: the name of the new cluster.RELEASE_CHANNEL
: the name of the GKE release channel for the cluster.VERSION
: the GKE version for the cluster.COMPUTE_LOCATION
: the Compute Engine region of the cluster.
Upgrade an existing cluster
Use the gcloud CLI to update an existing cluster:
gcloud container clusters upgrade CLUSTER_NAME \
--cluster-version=VERSION \
--master \
--location=COMPUTE_LOCATION
Replace the following:
CLUSTER_NAME
: the name of the existing cluster.VERSION
: the specific GKE version to which you want to upgrade your cluster. For more information, see Manually upgrading the control plane.COMPUTE_LOCATION
: the Compute Engine region of the cluster.
Deploy a sample workload
Deploy the following sample workload which provides the serving Pods for the external LoadBalancer Service.
Save the following sample Deployment as
store-deployment.yaml
:apiVersion: apps/v1 kind: Deployment metadata: name: store spec: replicas: 20 selector: matchLabels: app: store template: metadata: labels: app: store spec: containers: - image: gcr.io/google_containers/echoserver:1.10 imagePullPolicy: Always name: echoserver ports: - name: http containerPort: 8080 readinessProbe: httpGet: path: /healthz port: 8080 scheme: HTTP
Apply the manifest to the cluster:
kubectl apply -f store-deployment.yaml
Verify that there are 20 serving Pods for the Deployment:
kubectl get pods
The output is similar to the following:
NAME READY STATUS RESTARTS AGE store-cdb9bb4d6-s25vw 1/1 Running 0 10s store-cdb9bb4d6-vck6s 1/1 Running 0 10s ....
Create the external LoadBalancer Service
Expose the sample workload by creating an external LoadBalancer Service.
Save the following Service manifest as
store-v1-lb-svc.yaml
:apiVersion: v1 kind: Service metadata: name: store-v1-lb-svc annotations: cloud.google.com/l4-rbs: "enabled" spec: type: LoadBalancer selector: app: store ports: - name: tcp-port protocol: TCP port: 8080 targetPort: 8080
Apply the manifest to the cluster:
kubectl apply -f store-v1-lb-svc.yaml
Note the following points about this sample manifest:
The Service manifest must include the
cloud.google.com/l4-rbs: "enabled"
annotation at the time the manifest is first applied to the cluster. This instructs GKE to create a backend service-based external passthrough Network Load Balancer. Backend service-based external passthrough Network Load Balancers are required to support features like IPv6 and weighted load balancing.If you add the
cloud.google.com/l4-rbs: "enabled"
annotation to the manifest of an existing external LoadBalancer Service (that is, after the load balancer has been created), GKE ignores the annotation. External LoadBalancer Services created without this annotation in their manifests use target pool-based external passthrough Network Load Balancers. Using target pool-based external passthrough Network Load Balancers is discouraged.
Enable weighted load balancing
To distribute new connections proportionally to nodes based on how many serving,
ready, and non-terminating Pods are present on each node, enable weighted load
balancing by adding the networking.gke.io/weighted-load-balancing:
"pods-per-node"
annotation to the Service manifest.
Add the
networking.gke.io/weighted-load-balancing: "pods-per-node"
annotation to thestore-v1-lb-svc.yaml
Service manifest, and ensure that you also set theexternalTrafficPolicy: Local
so that it looks like this:apiVersion: v1 kind: Service metadata: name: store-v1-lb-svc annotations: cloud.google.com/l4-rbs: "enabled" networking.gke.io/weighted-load-balancing: "pods-per-node" spec: type: LoadBalancer externalTrafficPolicy: Local selector: app: store ports: - name: tcp-port protocol: TCP port: 8080 targetPort: 8080
Apply the manifest to the cluster:
kubectl apply -f store-v1-lb-svc.yaml
Note the following about this example on weighted load balancing:
The Service manifest uses
externalTrafficPolicy: Local
. If you don't need to enable weighted load balancing, you can also useexternalTrafficPolicy: Cluster
. For details about how theexternalTrafficPolicy
defines node grouping, which nodes pass their load balancer health checks, and how packets are processed, see LoadBalancer Service concepts.If you enable weighted load balancing, GKE doesn't prevent you from using
externalTrafficPolicy: Cluster
, butexternalTrafficPolicy: Cluster
effectively disables weighted load balancing because the packet might be routed, after the load balancer, to a different node.
You can also enable weighted load balancing on an existing external LoadBalancer
Service by using kubectl edit svc service-name
. The
kubectl edit
command opens the existing load balancer's Service manifest in
your configured text editor, where you can modify the manifest and save changes.
When you edit an existing external LoadBalancer Service,note the following
points:
The existing external LoadBalancer Service must have resulted in the creation of a backend service-based external passthrough Network Load Balancers. This means the existing external LoadBalancer Service must have included the
cloud.google.com/l4-rbs: "enabled"
annotation when the manifest was first applied to the cluster.Adding the
networking.gke.io/weighted-load-balancing: "pods-per-node"
annotation to an existing external LoadBalancer Service that uses a target pool-based external passthrough Network Load Balancer has no effect.When updating the existing external LoadBalancer Service manifest, make sure to set
externalTrafficPolicy: Local
. UsingexternalTrafficPolicy: Cluster
effectively disables weighted load balancing because the packet might be routed, after the load balancer, to a different node.
Disable weighted load balancing
To distribute new connections to nodes regardless of how many serving Pods are
present on each node, disable weighted load balancing by removing the
networking.gke.io/weighted-load-balancing: "pods-per-node"
annotation from the
Service manifest.
Verify the external LoadBalancer Service and its components
Verify that your Service is running:
kubectl get svc store-v1-lb-svc
The output is similar to the following:
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE store-v1-lb-svc LoadBalancer 10.44.196.160 35.193.28.231 8080:32466/TCP 11m
GKE assigned an
EXTERNAL_IP
for the external passthrough Network Load Balancer.Test connecting to the load balancer:
curl EXTERNAL_IP:PORT
Replace the following:
EXTERNAL_IP
: the allocated IP address for the external passthrough Network Load Balancer.PORT
: the allocated port number for the external passthrough Network Load Balancer.
The output is similar to the following:
Hostname: store-v1-lb-svc-cdb9bb4d6-hflxd Pod Information: -no pod information available- Server values: server_version=nginx: 1.13.3 - lua: 10008 Request Information: client_address=10.128.0.50 method=GET real path=/ query= request_version=1.1 request_scheme=http request_uri=EXTERNAL_IP Request Headers: accept=*/* host=EXTERNAL_IP user-agent=curl/7.81.0 Request Body: -no body in request-
Check your LoadBalancer Service and its set of annotations describing its Google Cloud resources:
kubectl describe svc store-v1-lb-svc
The output is similar to the following:
Name: my-service-external Namespace: default Labels: <none> Annotations: cloud.google.com/l4-rbs: enabled networking.gke.io/weighted-load-balancing: pods-per-node #This annotation appears in the output only if weighted load balancing is enabled. service.kubernetes.io/backend-service: k8s2-qvveq1d8-default-my-service-ext-5s55db85 service.kubernetes.io/firewall-rule: k8s2-qvveq1d8-default-my-service-ext-5s55db85 service.kubernetes.io/firewall-rule-for-hc: k8s2-qvveq1d8-default-my-service-ext-5s55db85-fw service.kubernetes.io/healthcheck: k8s2-qvveq1d8-default-my-service-ext-5s55db85 service.kubernetes.io/tcp-forwarding-rule: a808124abf8ce406ca51ab3d4e7d0b7d Selector: app=my-app Type: LoadBalancer IP Family Policy: SingleStack IP Families: IPv4 IP: 10.18.102.23 IPs: 10.18.102.23 LoadBalancer Ingress: 35.184.160.229 Port: tcp-port 8080/TCP TargetPort: 8080/TCP NodePort: tcp-port 31864/TCP Endpoints: 10.20.1.28:8080,10.20.1.29:8080 Session Affinity: None External Traffic Policy: Local HealthCheck NodePort: 30394 Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal ADD 4m55s loadbalancer-controller default/my-service-ext
There are several fields that indicate that a backend service-based external passthrough Network Load Balancer and its Google Cloud resources were successfully created:
Events
field. This field is empty when the LoadBalancer Service and its resources were created successfully. If an error has occurred, it is listed here.List of
Annotations
enabled: GKE adds the following list of read-only annotations to the Service manifest. Each annotation whose name begins withservice.kubernetes.io/
is used to indicate the name of a Google Cloud resource created as part of or to support the load balancer.- The
networking.gke.io/weighted-load-balancing: pods-per-node
annotation indicates that weighted load balancing was applied and the load balancer distributes traffic to backend Pods based on the number of Pods running on each node. - The
service.kubernetes.io/backend-service
annotation indicates the name of the load balancer's backend service. - The
service.kubernetes.io/healthcheck
annotation indicates the name of the load balancer health check used by the backend service. - The
service.kubernetes.io/tcp-forwarding-rule
orservice.kubernetes.io/udp-forwarding-rule
annotation indicates the name of the load balancer's forwarding rule. - The
service.kubernetes.io/firewall-rule
annotation indicates the name of the firewall rule created to permit traffic to the cluster nodes. Source ranges for this firewall rule are customizable usingspec.loadBalancerSourceRanges[]
. For additional detail about firewall rules for LoadBalancer Services, see Firewall rules and source IP address allowlist. - The
service.kubernetes.io/firewall-rule-for-hc
annotation indicates the name of the firewall rule required for load balancer health checks.
- The
Verify that load balancer resources and firewall rules have been created for the external LoadBalancer Service:
To see the forwarding rule, run the following command:
gcloud compute forwarding-rules describe FWD_RULE_NAME \ --region=REGION_NAME
Replace the following:
FWD_RULE_NAME
: the forwarding rule name provided by either theservice.kubernetes.io/tcp-forwarding-rule
orservice.kubernetes.io/udp-forwarding-rule
read-only annotations. To check these annotations, runkubectl describe svc SERVICE_NAME
.REGION_NAME
: the Google Cloud region containing the cluster. For zonal clusters, the region contains the zone used by the cluster.
To see the backend service, run the following command:
gcloud compute backend-services describe BACKEND_SERVICE_NAME \ --region=REGION_NAME
Replace the following:
BACKEND_SERVICE_NAME
: the name of the backend service provided by theservice.kubernetes.io/backend-service
read-only annotation. To check this read-only annotation, runkubectl describe svc SERVICE_NAME
.REGION_NAME
: the Google Cloud region containing the cluster. For zonal clusters, the region contains the zone used by the cluster.
To see the load balancer health check, run the following command:
gcloud compute health-checks describe HEALTH_CHECK_NAME \ --region=REGION_NAME
Replace the following:
HEALTH_CHECK_NAME
: the load balancer's health check name. The name of the health check is provided by theservice.kubernetes.io/healthcheck
read-only annotation. To check this read-only annotation, runkubectl describe svc SERVICE_NAME
.REGION_NAME
: the Google Cloud region containing the cluster. For zonal clusters, the region contains the zone used by the cluster.
To see the firewall rules, run the following commands:
gcloud compute firewall-rules describe FIREWALL_RULE_NAME \ gcloud compute firewall-rules describe HEALTH_CHECK_FIREWALL_RULE_NAME
Replace the following:
FIREWALL_RULE_NAME
: the name of the firewall rule that permits traffic to the load balancer. The name of this firewall rule is provided by theservice.kubernetes.io/firewall-rule
read-only annotation. To check this read-only annotation, runkubectl describe svc SERVICE_NAME
.HEALTH_CHECK_FIREWALL_RULE_NAME
: the name of the firewall rule that permits health checks of the load balancer's backends (the cluster's nodes). The name of this firewall rule is provided by theservice.kubernetes.io/firewall-rule-for-hc
read-only annotation. To check this read-only annotation, runkubectl describe svc SERVICE_NAME
.
Delete the external LoadBalancer Service
To delete the sample store-v1-lb-svc
external LoadBalancer Service, use the
following command:
kubectl delete service store-v1-lb-svc
GKE automatically removes all load balancer resources that it created for the external LoadBalancer Service.
Troubleshoot external LoadBalancer Service
If you don't set externalTrafficPolicy: Local
, you might get a warning event,
when describing the Service using the following command:
kubectl describe svc store-v1-lb-svc`
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning UnsupportedConfiguration 4m55s loadbalancer-controller Weighted load balancing by pods-per-node has no effect with External Traffic Policy: Cluster.
To effectively enable weighted load balancing, you must set externalTrafficPolicy: Local
.
What's next
- For a general overview of Load Balancer Services, see LoadBalancer Services.
- For a description of Load Balancer Services parameters, see LoadBalancer Service parameters.
- Troubleshoot load balancing in GKE.