Internal Load Balancing

This page explains how to create an Internal load balancer on Container Engine and Kubernetes.

The Internal load balancer creates a private (RFC1918) LoadBalancer Ingress IP address in the cluster for receiving traffic on the network within the same compute region.

You create an Internal load balancer by adding a LoadBalancer spec to your cluster's Service configuration file. If your cluster does not have a Service configuration file, you'll need to create one. Then, you can create the Internal load balancer using the Kubernetes kubectl command-line interface.

Before you begin

Writing the Service configuration file

The following is an example of a Service resource that includes an Internal load balancer:

config.yaml

apiVersion: v1
kind: Service
metadata:
  name: [SERVICE-NAME]
  annotations:
    cloud.google.com/load-balancer-type: "internal"
  labels:
    app: echo
spec:
  type: LoadBalancer
  loadBalancerIP: [IP-ADDRESS]
  ports:
  - port: 9000
    protocol: TCP
  selector:
    [KEY]: [VALUE]

Your Service configuration file must contain the following:

  • The type LoadBalancer and port fields
  • An annotation, cloud.google.com/load-balancer-type: "internal", which specifies that an Internal load balancer is enabled
  • [SERVICE-NAME] is the name you choose for your service

The optional spec: protocol field defines the network protocol the Internal load balancer should use. If the field is omitted, the protocol is set to TCP.

The optional spec: loadBalancerIP field enables you to choose a specific IP address if the cloud provider allows it. The IP address must not be in use by another Internal load balancer or service. If the cloud provider does not allow this, the field is ignored. If the loadBalancerIP field is not specified, an ephemeral IP is assigned to the LoadBalancer.

The Service should also have a spec: selector field to specify the Pods it should target. For example, the selector might target Pods labelled with app: echo.

Create the Internal load balancer

To create the Internal load balancer, run the following command in your shell or terminal window:

kubectl apply -f config.yaml

This command will update the existing Service to add an Internal load balancer, or create a new Service with an Internal load balancer.

Inspect the Service

To verify that the Service was created or updated as expected, run the following command:

kubectl describe service [SERVICE-NAME]

The command's output is similar to the following:

Name:           [SERVICE-NAME]
Namespace:      default
Labels:         app=echo
Annotations:        cloud.google.com/load-balancer-type=internal
Selector:       app=echo
Type:           LoadBalancer
IP:         10.0.146.226
LoadBalancer Ingress:   10.128.0.6
Port:               9000/TCP
NodePort:       30387/TCP
Endpoints:      10.100.1.10:80,10.100.2.10:80,10.100.3.8:80
Session Affinity:   ClientIP

You can also view details about the Internal load balancer in the Cloud Platform Console.

Using the Internal load balancer

Internal load balancers support Kubernetes Services, such as externalTrafficPolicy and sessionAffinity.

To access the Service from within the cluster, use the clusterIP (in the example above, IP) address. To access the Service from outside the cluster, use the LoadBalancer Ingress IP address.

Considerations for existing ingresses

If your cluster has an existing Ingress resource, that resource must use the balancing mode RATE. UTILIZATION balancing mode is not compatible.

Earlier BackendService resources created by Kubernetes Ingress Resources objects were created with no balancing mode specified. By default, the API used balancing mode UTILIZATION for HTTP load balancers. However, Internal load balancers cannot be pointed to instance groups with other load balancers using UTILIZATION.

To ensure compatibility in a Kubernetes cluster with an Internal load balancer and ingress resources, you may need to perform some manual steps.

Determine if your Ingress is compatible

To determine if your Ingress is compatible, run the following commands from your shell or terminal window:

GROUPNAME=`kubectl get configmaps ingress-uid -o jsonpath='k8s-ig--{.data.uid}' --namespace=kube-system`
gcloud compute backend-services list --format="table(name,backends[].balancingMode,backends[].group)" | grep $GROUPNAME

These commands export a shell variable, GROUPNAME, which fetches your cluster's instance group name. Then, your project's Compute BackendService resources are polled and the results are narrowed down based on the contents of $GROUPNAME.

The output may be similar to the following:

k8s-be-31210--...  [u'RATE']       us-central1-b/instanceGroups/k8s-ig--...
k8s-be-32125--...  [u'RATE']       us-central1-b/instanceGroups/k8s-ig--...

If the output displays RATE or zero entries are returned, then internal load balancers are compatible and no additional work is needed.

If the output displays entries marked with UTILIZATION, your ingresses are not compatible.

Updating your existing ingresses

The balancing mode type can only change when there are no existing HTTP(S) load balancers pointing to the cluster.

To update your ingress resources to be compatible with an Internal load balancer, you must do one of the following:

  1. If your master is running Kubernetes version 1.5.8 or higher, or 1.6.4 or higher, you can delete all ingresses and recreate them. Then, upgrade your cluster to Kubernetes version 1.7.2 or higher to be compatible with the Internal load balancer.
  2. You can create a new cluster running Kubernetes version 1.7.2 or higher, then migrate your services to that cluster. Migrating to the new cluster ensures that no ingresses can exist with the incompatible balancingMode value.

Restrictions for Internal load balancers

  • Your master and nodes must be running Kubernetes version 1.7.2 or higher.
  • Internal load balancers can only serve traffic on one type of protocol (TCP or UDP), and it uses the protocol of the first port specified in the Service definition.
  • Internal load balancers are not accessible from within the Kubernetes cluster (nodes and Pods).
  • Internal load balancers are only accessible from within the same network and region.
  • On clusters running Kubernetes version 1.7.4 or later, you can use internal load balancers with custom-mode subnets in addition to auto-mode subnets (available in version 1.7.2 or later).
  • While the clusterIP remains unchanged, Internal load balancer IP addresses cannot be reserved, so changes made to ports, protocols, or session affinity may cause this IP addresses change.

Limits

Each node of the cluster is a backend instance and will count against the backend instance limitation. For example, if a cluster contains 300 nodes and there is a limitation of 250 backend instances, only 250 instances will receive traffic. This may adversely affect services with externalTrafficPolicy set to Local.

For information on the limitations of Internal load balancers, see the Limits section of the Compute Engine internal load balancing topic.

Monitor your resources on the go

Get the Google Cloud Console app to help you manage your projects.

Send feedback about...

Container Engine