Internal Load Balancing

This page explains how to create an internal load balancer on Kubernetes Engine and Kubernetes.

Overview

Internal load balancing makes your cluster's services accessible to applications running on the same network but outside of the cluster. For example, if you run a cluster alongside some Compute Engine VM instances in the same network and you would like your cluster-internal services to be available to the cluster-external instances, you need to configure one of your cluster's Service resources to add an internal load balancer.

Without internal load balancing, you would need to set up external load balancers, create firewall rules to limit the access, and set up network routes to make the IP address of the application accessible outside of the cluster.

Internal load balancing creates a private (RFC1918) LoadBalancer Ingress IP address in the cluster for receiving traffic on the network within the same compute region from an IP range in the user’s subnet.

You create an internal load balancer by adding a LoadBalancer specification to a Service resource. If your cluster does not have a Service, you'll need to create one. Then, you can create the internal load balancer using the Kubernetes kubectl command-line interface.

Before you begin

Writing the Service configuration file

The following is an example of a Service resource that configures an internal load balancer:

config.yaml

apiVersion: v1
kind: Service
metadata:
  name: [SERVICE-NAME]
  annotations:
    cloud.google.com/load-balancer-type: "Internal"
  labels:
    app: echo
spec:
  type: LoadBalancer
  loadBalancerIP: [IP-ADDRESS]
  ports:
  - port: 9000
    protocol: TCP
  selector:
    [KEY]: [VALUE]

Your Service configuration file must contain the following:

  • The type LoadBalancer and port fields
  • An annotation, cloud.google.com/load-balancer-type: "Internal", which specifies that an internal load balancer is to be configured
  • [SERVICE-NAME] is the name you choose for your service

The optional spec: protocol field defines the network protocol the internal load balancer should use. If the field is omitted, the protocol is set to TCP.

The optional spec: loadBalancerIP field enables you to choose a specific IP address. The IP address must not be in use by another internal load balancer or service. If the loadBalancerIP field is not specified, an ephemeral IP is assigned to the LoadBalancer.

The Service should also have a spec: selector field to specify the Pods it should target. For example, the selector might target Pods labelled with app: echo.

Creating the internal load balancer

To create the internal load balancer, run the following command in your shell or terminal window:

kubectl apply -f config.yaml

This command will update the existing Service to add an internal load balancer, or create a new Service with an internal load balancer.

Inspecting the Service

Console

To view details about the internal load balancer, perform the following steps:

  1. Visit the Kubernetes Engine Discovery & load balancing menu in the Google Cloud Platform Console.

    Visit the Discovery & load balancing menu

  2. Select the desired Service.

gcloud

To verify that the Service was created or updated as expected, run the following command:

kubectl describe service [SERVICE-NAME]

The command's output is similar to the following:

Name:     [SERVICE-NAME]
Namespace:    default
Labels:     app=echo
Annotations:    cloud.google.com/load-balancer-type=Internal
Selector:   app=echo
Type:     LoadBalancer
IP:     10.0.146.226
LoadBalancer Ingress: 10.128.0.6
Port:       9000/TCP
NodePort:   30387/TCP
Endpoints:    10.100.1.10:80,10.100.2.10:80,10.100.3.8:80
Session Affinity: ClientIP

Using the internal load balancer

Internal load balancers support Kubernetes Services parameters, such as externalTrafficPolicy, sessionAffinity, and loadBalancerSourceRanges.

You can continue to access the Service from within the cluster using the clusterIP (in the example above, IP) address. To access the Service from outside the cluster, use the LoadBalancer Ingress IP address.

For more information about configuring Service parameters, refer to Configure Your Cloud Provider's Firewalls

Considerations for existing ingresses

If your cluster has an existing Ingress resource, that resource must use the balancing mode RATE. UTILIZATION balancing mode is not compatible.

Earlier BackendService resources created by Kubernetes Ingress Resources objects were created with no balancing mode specified. By default, the API used balancing mode UTILIZATION for HTTP load balancers. However, internal load balancers cannot be pointed to instance groups with other load balancers using UTILIZATION.

To ensure compatibility in a Kubernetes cluster with an internal load balancer and ingress resources, you may need to perform some manual steps.

Determining if your Ingress is compatible

To determine if your Ingress is compatible, run the following commands from your shell or terminal window:

GROUPNAME=`kubectl get configmaps ingress-uid -o jsonpath='k8s-ig--{.data.uid}' --namespace=kube-system`
gcloud compute backend-services list --format="table(name,backends[].balancingMode,backends[].group)" | grep $GROUPNAME

These commands export a shell variable, GROUPNAME, which fetches your cluster's instance group name. Then, your project's Compute BackendService resources are polled and the results are narrowed down based on the contents of $GROUPNAME.

The output may be similar to the following:

k8s-be-31210--...  [u'RATE']       us-central1-b/instanceGroups/k8s-ig--...
k8s-be-32125--...  [u'RATE']       us-central1-b/instanceGroups/k8s-ig--...

If the output displays RATE or zero entries are returned, then internal load balancers are compatible and no additional work is needed.

If the output displays entries marked with UTILIZATION, your ingresses are not compatible.

Updating your existing ingresses

The balancing mode type can only change when there are no existing HTTP(S) load balancers pointing to the cluster.

To update your ingress resources to be compatible with an internal load balancer, you must do one of the following:

  1. If your master is running Kubernetes version 1.5.8 or higher, or 1.6.4 or higher, you can delete all ingresses and recreate them. Then, upgrade your cluster to Kubernetes version 1.7.2 or higher to be compatible with the internal load balancer.
  2. You can create a new cluster running Kubernetes version 1.7.2 or higher, then migrate your services to that cluster. Migrating to the new cluster ensures that no ingresses can exist with the incompatible balancingMode value.

Restrictions for internal load balancers

  • Your master and nodes must be running Kubernetes version 1.7.2 or higher.
  • Internal load balancers can only serve traffic on one type of protocol (TCP or UDP), and it uses the protocol of the first port specified in the Service definition.
  • Internal load balancers are only accessible from within the same network and region.
  • On clusters running Kubernetes version 1.7.4 or later, you can use internal load balancers with custom-mode subnets in addition to auto-mode subnets (available in version 1.7.2 or later).
  • While the clusterIP remains unchanged, internal load balancer IP addresses cannot be reserved, so changes made to ports, protocols, or session affinity may cause this IP addresses change.

Limits

Each node of the cluster is a backend instance and will count against the backend instance limitation. For example, if a cluster contains 300 nodes and there is a limitation of 250 backend instances, only 250 instances will receive traffic. This may adversely affect services with externalTrafficPolicy set to Local.

For information on the limitations of internal load balancers, see the Limits section of the Compute Engine internal load balancing topic.

What's next

Monitor your resources on the go

Get the Google Cloud Console app to help you manage your projects.

Send feedback about...

Kubernetes Engine