Using an internal TCP/UDP load balancer

Autopilot Standard

---

This page explains how to create an internal TCP/UDP load balancer on Google Kubernetes Engine (GKE). To create an external network load balancer, learn how to Create a Service of type LoadBalancer.

• LoadBalancer Service.
• LoadBalancer Service parameters.
• Backend service-based network load balancer.

Using internal TCP/UDP load balancer subsetting

Internal TCP/UDP load balancers make your cluster's services accessible to applications outside of your cluster that use the same VPC network and are located in the same Google Cloud region. For example, suppose you have a cluster in the us-west1 region and you need to make one of its services accessible to Compute Engine virtual machine (VM) instances running in that region on the same VPC network.

Internal load balancer subsetting for GKE improves the scalability of internal TCP/UDP load balancers by partitioning backends into smaller, overlapping groups. With subsetting, you can configure internal TCP/UDP load balancers on clusters with more than 250 nodes.

Internal load balancer subsetting is enabled by default for Autopilot clusters, so you can skip to Deploy a workload. For Standard clusters, you can enable subsetting when you create a cluster and by editing an existing cluster.

Requirements and limitations

Subsetting for GKE has the following requirements and limitations:

• You can enable subsetting in new and existing Standard clusters in GKE versions 1.18.19-gke.1400 and later.
• The cluster must have the HttpLoadBalancing add-on enabled. This add-on is enabled by default and cannot be disabled in Autopilot clusters. A Standard cluster that has disabled this add-on is unable to use subsetting. To learn how to run a custom Ingress controller with the HttpLoadBalancing add-on enabled, see Use a custom Ingress controller.
• Google Cloud CLI version 345.0.0 and later.
• Quotas for Network Endpoint Groups apply. Google Cloud creates 1 NEG per internal TCP/UDP load balancer per zone.
• Quotas for forwarding rules, backend services and other network resources apply.
• Subsetting cannot be disabled once it is enabled in a Standard cluster.
• Subsetting cannot be used with the annotation to share backend services, alpha.cloud.google.com/load-balancer-backend-share.

Before you begin

Before you start, make sure you have performed the following tasks:

• Enable the Google Kubernetes Engine API.
Enable Google Kubernetes Engine API
• If you want to use the Google Cloud CLI for this task, install and then initialize the gcloud CLI.
  Note: For existing gcloud CLI installations, make sure to set the compute/region and compute/zone properties. By setting default locations, you can avoid errors in gcloud CLI like the following: One of [--zone, --region] must be supplied: Please specify location.

Enable internal load balancer subsetting in a new Standard cluster

You can create a Standard cluster with internal load balancer subsetting enabled using the Google Cloud CLI or the Google Cloud console:

Console

1. Go to the Google Kubernetes Engine page in the Google Cloud console.

   Go to Google Kubernetes Engine

2. Click Create.

3. Configure your cluster as desired.

4. From the navigation pane, under Cluster, click Networking.

5. Select the Enable subsetting for L4 internal load balancers checkbox.

6. Click Create.

gcloud

gcloud container clusters create CLUSTER_NAME \
    --cluster-version=VERSION \
    --enable-l4-ilb-subsetting \
    --region=COMPUTE_REGION

Replace the following:

• CLUSTER_NAME: the name of the new cluster.
• VERSION: the GKE version, which must be 1.18.19-gke.1400 or later. You can also use the --release-channel option to select a release channel. The release channel must have a default version 1.18.19-gke.1400 or later.
• COMPUTE_REGION: the compute region for the cluster.

Enable internal load balancer subsetting in an existing cluster

You can enable internal load balancer subsetting for an existing Standard cluster using the gcloud CLI or the Google Cloud console. You cannot disable internal load balancer subsetting after you have enabled it in a cluster.

Console

1. In the Google Cloud console, go to the Google Kubernetes Engine page.

   Go to Google Kubernetes Engine

2. In the cluster list, click the name of the cluster you want to modify.

3. Under Networking, next to the Subsetting for L4 Internal Load Balancers field, click Enable subsetting for L4 internal load balancers.

4. Select the Enable subsetting for L4 internal load balancers checkbox.

5. Click Save Changes.

gcloud

gcloud container clusters update CLUSTER_NAME \
    --enable-l4-ilb-subsetting

Replace the following:

• CLUSTER_NAME: the name of the cluster.

Note: After you enable subsetting for a GKE cluster, all new internal TCP/UDP load balancer services are created with subsetting enabled. This is not disruptive to existing instance group based Load Balancer services that continue to use instance groups. Instance group based services cannot be converted to NEG based services, but they can exist in the same cluster with NEG based services. To migrate to NEG based services, you must create new internal Load Balancer Services.

Deploy a workload

The following manifest describes a Deployment that runs a sample web application container image.

1. Save the manifest as ilb-deployment.yaml:

   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: ilb-deployment
   spec:
     replicas: 3
     selector:
       matchLabels:
         app: ilb-deployment
     template:
       metadata:
         labels:
           app: ilb-deployment
       spec:
         containers:
         - name: hello-app
           image: us-docker.pkg.dev/google-samples/containers/gke/hello-app:1.0
   

2. Apply the manifest to your cluster:

   kubectl apply -f ilb-deployment.yaml
   

Create an internal TCP load balancer

Create an internal TCP load balancer using a Service. This internal load balancer is assigned to TCP port 8080.

1. Save the manifest as ilb-svc.yaml:

   apiVersion: v1
   kind: Service
   metadata:
     name: ilb-svc
     annotations:
       networking.gke.io/load-balancer-type: "Internal"
   spec:
     type: LoadBalancer
     externalTrafficPolicy: Cluster
     selector:
       app: ilb-deployment
     ports:
     - name: tcp-port
       protocol: TCP
       port: 8080
       targetPort: 8080
   

   Your manifest must contain the following:

   • A name for the Service, in this case ilb-svc.
   • An annotation that specifies an internal TCP/UDP load balancer. The annotation depends on the version of your GKE cluster. For GKE versions 1.17 and later, use the annotation networking.gke.io/load-balancer-type: "Internal". For earlier versions, use the annotation cloud.google.com/load-balancer-type: "Internal".
   • The type: LoadBalancer.
   • A spec: selector field to specify the Pods the Service should target, for example, app: hello.
   • The port, the port over which the Service is exposed, and targetPort, the port on which the containers are listening.

2. Apply the manifest to your cluster:

   kubectl apply -f ilb-svc.yaml
   

3. Get detailed information about the Service:

   kubectl get service ilb-svc --output yaml
   

   The output is similar to the following:

   apiVersion: v1
   kind: Service
   metadata:
     annotations:
       cloud.google.com/neg: '{"ingress":true}'
       cloud.google.com/neg-status: '{"network_endpoint_groups":{"0":"k8s2-pn2h9n5f-default-ilb-svc-3bei4n1r"},"zones":["us-central1-a","us-central1-b","us-central1-c"]}'
       kubectl.kubernetes.io/last-applied-configuration: |
         {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"networking.gke.io/load-balancer-type":"Internal"},"name":"ilb-svc","namespace":"default"},"spec":{"externalTrafficPolicy":"Cluster","ports":[{"name":"tcp-port","port":8080,"protocol":"TCP","targetPort":8080}],"selector":{"app":"ilb-deployment"},"type":"LoadBalancer"}}
       networking.gke.io/load-balancer-type: Internal
       service.kubernetes.io/backend-service: k8s2-pn2h9n5f-default-ilb-svc-3bei4n1r
       service.kubernetes.io/firewall-rule: k8s2-pn2h9n5f-default-ilb-svc-3bei4n1r
       service.kubernetes.io/firewall-rule-for-hc: k8s2-pn2h9n5f-l4-shared-hc-fw
       service.kubernetes.io/healthcheck: k8s2-pn2h9n5f-l4-shared-hc
       service.kubernetes.io/tcp-forwarding-rule: k8s2-tcp-pn2h9n5f-default-ilb-svc-3bei4n1r
     creationTimestamp: "2022-07-22T17:26:04Z"
     finalizers:
     - gke.networking.io/l4-ilb-v2
     - service.kubernetes.io/load-balancer-cleanup
     name: ilb-svc
     namespace: default
     resourceVersion: "51666"
     uid: d7a1a865-7972-44e1-aa9e-db5be23d6567
   spec:
     allocateLoadBalancerNodePorts: true
     clusterIP: 10.88.2.141
     clusterIPs:
     - 10.88.2.141
     externalTrafficPolicy: Cluster
     internalTrafficPolicy: Cluster
     ipFamilies:
     - IPv4
     ipFamilyPolicy: SingleStack
     ports:
     - name: tcp-port
       nodePort: 30521
       port: 8080
       protocol: TCP
       targetPort: 8080
     selector:
       app: ilb-deployment
     sessionAffinity: None
     type: LoadBalancer
   status:
     loadBalancer:
       ingress:
       - ip: 10.128.15.245
   

   The output has the following attributes:

   • The internal load balancer's IP address under status.loadBalancer.ingress. This IP address is different from the value of clusterIP. In this example, the load balancer's IP address is 10.128.15.245:
   • Any Pod that has the label app: ilb-deployment is a member of this Service. These are the Pods that can be the final recipients of requests sent to your internal load balancer.

   • Clients call the Service by using the loadBalancer IP address and the TCP port specified in the port field of the Service manifest. The request is forwarded to one of the member Pods on the TCP port specified in the targetPort field. For the preceding example, a client calls the Service at 10.128.15.245 on TCP port 80. The request is forwarded to one of the member Pods on TCP port 8080. The member Pod must have a container listening on port 8080.

   • The nodePort value of 30521 is extraneous; it is not relevant to your internal load balancer.

4. Inspect the Service network endpoint group:

   kubectl get svc ilb-svc -o=jsonpath="{.metadata.annotations.cloud\.google\.com/neg-status}"
   

   The output is similar to the following:

   {"network_endpoint_groups":{"0":"k8s2-knlc4c77-default-ilb-svc-ua5ugas0"},"zones":["us-central1-c"]}
   

   The response indicates that GKE has created a network endpoint group named k8s2-knlc4c77-default-ilb-svc-ua5ugas0. This annotation is present in services of type LoadBalancer that use GKE subsetting and is not present in Services that do not use subsetting.

View the load balancer's forwarding rule

An internal load balancer is implemented as a forwarding rule. The forwarding rule has a backend service, which has an instance group.

The internal load balancer address, 10.128.15.245 in the preceding example, is the same as the forwarding rule address. To see the forwarding rule that implements your internal load balancer, start by listing all of the forwarding rules in your project:

gcloud compute forwarding-rules list --filter="loadBalancingScheme=INTERNAL"

The output includes the forwarding rule that has the same address as your internal load balancer, 10.128.15.245 in this example.

NAME                          ... IP_ADDRESS  ... TARGET
...
k8s2-tcp-pn2h9n5f-default-ilb-svc-3bei4n1r   10.128.15.245   us-central1/backendServices/k8s2-pn2h9n5f-default-ilb-svc-3bei4n1r

The output shows the associated backend service, k8s2-tcp-pn2h9n5f-default-ilb-svc-3bei4n1r in this example.

Describe the backend service:

gcloud compute backend-services describe k8s2-tcp-pn2h9n5f-default-ilb-svc-3bei4n1r --region=us-central1

The output shows the associated instance group, k8s2-tcp-pn2h9n5f-default-ilb-svc-3bei4n1r in this example:

backends:
- balancingMode: CONNECTION
  group: .../us-central1-b/networkEndpointGroups/k8s2-pn2h9n5f-default-ilb-svc-3bei4n1r
...
kind: compute#backendService
loadBalancingScheme: INTERNAL
name: aae3e263abe0911e9b32a42010a80008
...

Note: If you create a load balancer that is backed by instance groups and you set a high value for drainingTimeoutSec (for example, 3600 seconds), you might experience delays when you create or update other Services in the cluster that share the same instance group. If you need a high value for drainingTimeoutSec, consider using Internal load balancer subsetting.

Verify the internal TCP load balancer

SSH into a VM instance, and run the following command:

curl LOAD_BALANCER_IP

Replace LOAD_BALANCER_IP with your LoadBalancer ingress IP address.

The response shows the output of ilb-deployment:

Hello, world!
Version: 1.0.0
Hostname: ilb-deployment-77b45987f7-pw54n

Running the command from outside of the same VPC network or outside the same region results in a timed out error. If you configure global access, clients in any region in the same VPC network can access the load balancer.

Delete the internal TCP load balancer resources

You can delete the Deployment and Service using kubectl delete or the Google Cloud console.

kubectl

Delete the Deployment

To delete the Deployment, run the following command:

kubectl delete deployment ilb-deployment

Delete the Service

To delete the Service, run the following command:

kubectl delete service ilb-svc

Console

Delete the Deployment

To delete the Deployment, perform the following steps:

1. Go to the Workloads page in the Google Cloud console.

   Go to Workloads

2. Select the Deployment you want to delete, then click Delete.

3. When prompted to confirm, select the Delete Horizontal Pod Autoscaler associated with selected Deployment checkbox, then click Delete.

Delete the Service

To delete the Service, perform the following steps:

1. Go to the Services & Ingress page in the Google Cloud console.

   Go to Services & Ingress

2. Select the Service you want to delete, then click Delete.

3. When prompted to confirm, click Delete.

Service parameters

For more information about the load balancers parameters you can configure, see see Configuring TCP/UDP load balancing. In addition, internal LoadBalancer Services support the following additional parameters:

Feature	Summary	Service Field	GKE Version Support
Load Balancer Subnet	Specifies from which subnet the load balancer should automatically provision an IP	metadata:annotations: networking.gke.io/internal-load-balancer-subnet	Beta in GKE 1.17+ and 1.16.8-gke.10+ GA in GKE 1.17.9-gke.600+
Global Access	Allows the internal TCP/UDP load balancer virtual IP address to be accessible by clients across Google Cloud regions	metadata:annotations: networking.gke.io/internal-load-balancer-allow-global-access	Beta in GKE 1.16+ GA in GKE 1.17.9-gke.600+

Load balancer subnet

By default, GKE will deploy an internal TCP/UDP load balancer using the node subnet range. The subnet can be user-specified on a per-Service basis using the networking.gke.io/internal-load-balancer-subnet annotation. This is useful for separately firewalling the internal load balancer IPs from node IPs or for sharing the same Service subnet across multiple GKE clusters. This parameter is only relevant for the internal TCP/UDP load balancer Services.

The subnet must exist before it is referenced by the Service resource as GKE does not manage the lifecycle of the subnet itself. The subnet must also be in the same VPC and region as the GKE cluster. In this step it's created out of band from GKE:

gcloud compute networks subnets create gke-vip-subnet \
    --network=default \
    --range=10.23.0.0/24 \
    --region=us-central1

The following Service definition uses the internal-load-balancer-subnet to reference the subnet by name. By default an available IP from the subnet will automatically be chosen. You can also specify the loadBalancerIP but it must be part of the referenced subnet.

There are multiple ways to share this internal load balancer subnet to achieve different use cases:

• Multiple subnets for groups of Services in the same cluster
• A single subnet for all Services in a cluster
• A single subnet shared across multiple clusters and multiple Services

apiVersion: v1
kind: Service
metadata:
  name: ilb-svc
  annotations:
    networking.gke.io/load-balancer-type: "Internal"
    networking.gke.io/internal-load-balancer-subnet: "gke-vip-subnet"
  labels:
    app: hello
spec:
  type: LoadBalancer
  loadBalancerIP: 10.23.0.15
  selector:
    app: hello
  ports:
  - port: 80
    targetPort: 8080
    protocol: TCP

Note: The internal load balancer subnet is not related to the ClusterIP Service subnet range specified by the --services-ipv4-cidr flag.

Global access

Global access is an optional parameter for internal LoadBalancer Services that allows clients from any region in your VPC network to access the internal TCP/UDP load balancer. Without global access, traffic originating from clients in your VPC network must be in the same region as the load balancer. Global access allows clients in any region to access the load balancer. Backend instances must still be located in the same region as the load balancer.

Global access is enabled per-Service using the following annotation: networking.gke.io/internal-load-balancer-allow-global-access: "true".

Global access is not supported with legacy networks. Normal inter-region traffic costs apply when using global access across regions. Refer to Network pricing for information about network pricing for egress between regions. Global access is available in Beta on GKE clusters 1.16+ and GA on 1.17.9-gke.600+.

For on-premises clients, global access lets clients access the load balancer using Cloud VPN or Cloud Interconnect (VLAN) in any region. For more information, see Using Cloud VPN and Cloud Interconnect.

Note: Manually changing the Forwarding rule to enable GlobalAccess is not supported in 1.16+ versions. The Global Access annotation should be used instead.

Shared IP

The internal TCP/UDP load balancer allows the sharing of a Virtual IP address amongst multiple forwarding rules. This is useful for expanding the number of simultaneous ports on the same IP or for accepting UDP and TCP traffic on the same IP. It allows up to a maximum of 50 exposed ports per IP address. Shared IPs are supported natively on GKE clusters with internal LoadBalancer Services. When deploying, the Service's loadBalancerIP field is used to indicate which IP should be shared across Services.

Limitations

A shared IP for multiple load balancers has the following limitations and capabilities:

• Each Service (or forwarding rule) can have a maximum of five ports.
• A maximum of ten Services (forwarding rules) can share an IP address. This results in a maximum of 50 ports per shared IP.
• Protocol/port tuples cannot overlap between Services that share the same IP.
• A combination of TCP-only and UDP-only Services is supported on the same shared IP, however you cannot expose both TCP and UDP ports in the same Service.

Enabling Shared IP

To enable an internal LoadBalancer Services to share a common IP, follow these steps:

1. Create a static internal IP with --purpose SHARED_LOADBALANCER_VIP. An IP address must be created with this purpose to enable its ability to be shared. If you create the static internal IP address in a Shared VPC, you must create the IP address in the same service project as the instance that will use the IP address, even though the value of the IP address will come from the range of available IPs in a selected shared subnet of the Shared VPC network. Refer to reserving a static internal IP on the Provisioning Shared VPC page for more information.

2. Deploy up to ten internal LoadBalancer Services using this static IP in the loadBalancerIP field. The internal TCP/UDP load balancers are reconciled by the GKE service controller and deploy using the same frontend IP.

The following example demonstrates how this is done to support multiple TCP and UDP ports against the same internal load balancer IP.

1. Create a static IP in the same region as your GKE cluster. The subnet must be the same subnet that the load balancer uses, which by default is the same subnet that is used by the GKE cluster node IPs.

   If your cluster and the VPC network are in the same project:

   gcloud compute addresses create IP_ADDR_NAME \
       --project=PROJECT_ID \
       --subnet=SUBNET \
       --addresses=IP_ADDRESS \
       --region=COMPUTE_REGION \
       --purpose=SHARED_LOADBALANCER_VIP
   

   If your cluster is in a Shared VPC service project but uses a Shared VPC network in a host project:

   gcloud compute addresses create IP_ADDR_NAME \
       --project=SERVICE_PROJECT_ID \
       --subnet=projects/HOST_PROJECT_ID/regions/REGION/subnetworks/SUBNET \
       --addresses=IP_ADDRESS \
       --region=COMPUTE_REGION \
       --purpose=SHARED_LOADBALANCER_VIP
   

   Replace the following:

   • IP_ADDR_NAME: a name for the IP address object.
   • SERVICE_PROJECT_ID: the ID of the service project.
   • PROJECT_ID: the ID of your project (single project).
   • HOST_PROJECT_ID: the ID of the Shared VPC host project.
   • COMPUTE_REGION: the compute region containing the shared subnet.
   • IP_ADDRESS: an unused internal IP address from the selected subnet's primary IP address range. If you omit specifying an IP address, Google Cloud selects an unused internal IP address from the selected subnet's primary IP address range. To determine an automatically selected address, you'll need to run gcloud compute addresses describe.
   • SUBNET: the name of the shared subnet.

2. Save the following TCP Service configuration to a file named tcp-service.yaml and then deploy to your cluster. Replace IP_ADDRESS with the IP address you chose in the previous step.

   apiVersion: v1
   kind: Service
   metadata:
     name: tcp-service
     namespace: default
     annotations:
       networking.gke.io/load-balancer-type: "Internal"
   spec:
     type: LoadBalancer
     loadBalancerIP: IP_ADDRESS
     selector:
       app: myapp
     ports:
     - name: 8001-to-8001
       protocol: TCP
       port: 8001
       targetPort: 8001
     - name: 8002-to-8002
       protocol: TCP
       port: 8002
       targetPort: 8002
     - name: 8003-to-8003
       protocol: TCP
       port: 8003
       targetPort: 8003
     - name: 8004-to-8004
       protocol: TCP
       port: 8004
       targetPort: 8004
     - name: 8005-to-8005
       protocol: TCP
       port: 8005
       targetPort: 8005
   

3. Apply this Service definition against your cluster:

   kubectl apply -f tcp-service.yaml
   

4. Save the following UDP Service configuration to a file named udp-service.yaml and then deploy it. It also uses the IP_ADDRESS that you specified in the previous step.

   apiVersion: v1
   kind: Service
   metadata:
     name: udp-service
     namespace: default
     annotations:
       networking.gke.io/load-balancer-type: "Internal"
   spec:
     type: LoadBalancer
     loadBalancerIP: IP_ADDRESS
     selector:
       app: my-udp-app
     ports:
     - name: 9001-to-9001
       protocol: UDP
       port: 9001
       targetPort: 9001
     - name: 9002-to-9002
       protocol: UDP
       port: 9002
       targetPort: 9002
   

5. Apply this file against your cluster:

   kubectl apply -f udp-service.yaml
   

6. Validate that the VIP is shared amongst load balancer forwarding rules by listing them out and filtering for the static IP. This shows that there is a UDP and a TCP forwarding rule both listening across seven different ports on the shared IP_ADDRESS, which in this example is 10.128.2.98.

   gcloud compute forwarding-rules list | grep 10.128.2.98
   ab4d8205d655f4353a5cff5b224a0dde                         us-west1   10.128.2.98     UDP          us-west1/backendServices/ab4d8205d655f4353a5cff5b224a0dde
   acd6eeaa00a35419c9530caeb6540435                         us-west1   10.128.2.98     TCP          us-west1/backendServices/acd6eeaa00a35419c9530caeb6540435
   

All ports

Internal forwarding rules support up to five ports per forwarding rule or an optional parameter --ports=ALL that forwards all ports on the forwarding rule.

Requirements

All ports on GKE has the following requirements and limitations:

• Only supported when --enable-l4-ilb-subsetting is enabled.
• Only supported for internal load balancer services.
• Supports any number of ports across a maximum of 100 contiguous port ranges.

The GKE controller automatically enables all ports on the forwarding rule when a service has more than five ports. For example, the following service manifest has six ports configured across two contiguous ranges:

apiVersion: v1
kind: Service
metadata:
  name: all-ports
  annotations:
    networking.gke.io/load-balancer-type: "Internal"
spec:
  type: LoadBalancer
  selector:
    app: myapp
  ports:
  - port: 8081
    targetPort: 8081
    name: 8081-to-8081
    protocol: TCP
  - port: 8082
    targetPort: 8082
    name: 8082-to-8082
    protocol: TCP
  - port: 8083
    targetPort: 8083
    name: 8083-to-8083
    protocol: TCP
  - port: 9001
    targetPort: 9001
    name: 9001-to-9001
    protocol: TCP
  - port: 9002
    targetPort: 9002
    name: 9002-to-9002
    protocol: TCP
  - port: 9003
    targetPort: 9003
    name: 9003-to-9003
    protocol: TCP

The GKE controller enables all ports on the forwarding rule because the service has more than five ports. However, the GKE controller only creates firewall ports for the ports specified in the service. All other rules are blocked by VPC firewalls.

Restrictions and limits

Restrictions for internal TCP/UDP load balancers

• For clusters running Kubernetes version 1.7.4 and later, you can use internal load balancers with custom-mode subnets in addition to auto-mode subnets.
• Clusters running Kubernetes version 1.7.X and later support using a reserved IP address for the internal TCP/UDP load balancer if you create the reserved IP address with the --purpose flag set to SHARED_LOADBALANCER_VIP. Refer to Enabling Shared IP for step-by-step directions. GKE only preserves the IP address of an internal TCP/UDP load balancer if the Service references an internal IP address with that purpose. Otherwise, GKE might change the load balancer's IP address (spec.loadBalancerIP) if the Service is updated (for example, if ports are changed).
• Even if the load balancer's IP address changes (see previous point), the spec.clusterIP remains constant.

Restrictions for internal UDP load balancers

• Internal UDP load balancers do not support using sessionAffinity: ClientIP.

Limits

A Kubernetes service with type: LoadBalancer and the networking.gke.io/load-balancer-type: Internal annotation creates an internal load balancer that targets the Kubernetes service. The number of such services is limited by the number of internal forwarding rules that you can create in a VPC network. For details, see Per network limits.

The maximum number of nodes in a GKE cluster with an internal TCP/UDP load balancer depends on the value of externalTrafficPolicy:

• externalTrafficPolicy: Cluster: the internal TCP/UDP load balancer backend uses a maximum of 250 randomly selected nodes. If the cluster has more than 250 nodes, all load balancer traffic enters the cluster through the 250 nodes and is forwarded to a randomly selected matching Pod. Using this mode with more than 250 nodes is not recommended.

• externalTrafficPolicy: Local: the internal TCP/UDP load balancer backend uses a maximum of 250 randomly selected nodes. If none of the selected 250 nodes run the backend Pods for the internal TCP/UDP load balancer service, connections to the LoadBalancer IP fail. Using this mode with more than 250 nodes is not supported.

To remove this limitation, enable internal load balancer subsetting.

For more information about VPC limits, see Quotas and limits.

Known issues

Connection timeout every 10 minutes

Internal LoadBalancer Services created with Subsetting might observe traffic disruptions roughly every 10 minutes. This bug has been fixed in versions:

• 1.18.19-gke.1700 and later
• 1.19.10-gke.1000 and later
• 1.20.6-gke.1000 and later

Error creating load balancer in Standard tier

When you create an internal TCP/UDP load balancer in a project with the project default network tier set to Standard, the following error message appears:

Error syncing load balancer: failed to ensure load balancer: googleapi: Error 400: STANDARD network tier (the project's default network tier) is not supported: Network tier other than PREMIUM is not supported for loadBalancingScheme=INTERNAL., badRequest

To resolve this issue in GKE versions earlier than 1.23.3-gke.900, configure the project default network tier to Premium.

This issue is resolved in GKE versions 1.23.3-gke.900 and later. The GKE controller creates internal TCP/UDP load balancers in the Premium network tier even if the project default network tier is set to Standard.

Troubleshooting

To determine the list of nodes in a subset for a service, use the following command:

gcloud compute network-endpoint-groups list-network-endpoints NEG_NAME \
    --zone=COMPUTE_ZONE

Replace the following:

• NEG_NAME: the name of the network endpoint group created by the GKE controller.
• COMPUTE_ZONE: the compute zone of the network endpoint group to operate on.

To determine the list of healthy nodes for an internal TCP/UDP load balancer, use the following command:

gcloud compute backend-services get-health SERVICE_NAME \
    --region=COMPUTE_REGION

Replace the following:

• SERVICE_NAME: the name of the backend service. This value is the same as the name of the network endpoint group created by the GKE controller.
• COMPUTE_REGION: the compute region of the backend service to operate on.

What's next

• Read the GKE network overview.
• Learn more about Compute Engine load balancers.
• Learn how to create a VPC-native cluster.
• Learn about IP masquerade agent.
• Learn about configuring authorized networks.