Internal TCP/UDP Load Balancing

Beta

Este produto ou recurso está em um estado de pré-lançamento e pode ser alterado ou ter suporte limitado. Para mais informações, consulte Estágios de lançamento do produto.

This page explains how to create a Compute Engine internal TCP/UDP load balancer on Google Kubernetes Engine.

Overview

Internal TCP/UDP Load Balancing makes your cluster's services accessible to applications outside of your cluster that use the same VPC network and are located in the same GCP region. For example, suppose you have a cluster in the us-west1 region and you need to make one of its services accessible to Compute Engine VM instances running in that region on the same VPC network.

You can create an internal TCP/UDP load balancer by creating a Service resource with the cloud.google.com/load-balancer-type: "Internal" annotation and a type: LoadBalancer specification. The instructions and example below highlight how to do this.

Without Internal TCP/UDP Load Balancing, you would need to set up an external load balancer and firewall rules to make the application accessible outside of the cluster.

Internal TCP/UDP Load Balancing creates a private (RFC 1918) IP address for the cluster that receives traffic on the network within the same compute region.

Note: An internal TCP/UDP load balancer is not assigned a fully-qualified domain name (FQDN), and it is not possible to configure one using GKE. You can assign a static IP address to your internal TCP/UDP load balancer and assign that IP to a record in Cloud DNS, but Cloud DNS is not tightly integrated with GKE.

Pricing

You are charged per Compute Engine's pricing model. For more information, refer to the Internal TCP/UDP Load Balancing pricing page.

Before you begin

To prepare for this task, perform the following steps:

• Ensure that you have enabled the Google Kubernetes Engine API.
Enable Google Kubernetes Engine API
• Ensure that you have installed the Cloud SDK.
• Set your default project ID:

  gcloud config set project [PROJECT_ID]

• If you are working with zonal clusters, set your default compute zone:

  gcloud config set compute/zone [COMPUTE_ZONE]

• If you are working with regional clusters, set your default compute region:

  gcloud config set compute/region [COMPUTE_REGION]

• Update gcloud to the latest version:

  gcloud components update

Note: You can override these default settings in gcloud commands using the --project, --zone, and --region operational flags.

Create a Deployment

The following manifest describes a Deployment that runs 3 replicas of a Hello World app.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: hello-app
spec:
  selector:
    matchLabels:
      app: hello
  replicas: 3
  template:
    metadata:
      labels:
        app: hello
    spec:
      containers:
      - name: hello
        image: "gcr.io/google-samples/hello-app:2.0"

The source code and Dockerfile for this sample app is available on GitHub. Since no PORT environment variable is specified, the containers listen on the default port: 8080.

To create the Deployment, create the file my-deployment.yaml from the manifest, and then run the following command in your shell or terminal window:

kubectl apply -f my-deployment.yaml

Create an internal TCP load balancer

The following sections explain how to create an internal TCP load balancer using a Service.

Writing the Service configuration file

The following is an example of a Service that creates an internal TCP load balancer:

apiVersion: v1
kind: Service
metadata:
  name: ilb-service
  annotations:
    cloud.google.com/load-balancer-type: "Internal"
  labels:
    app: hello
spec:
  type: LoadBalancer
  selector:
    app: hello
  ports:
  - port: 80
    targetPort: 8080
    protocol: TCP

Note: protocol: defines the network protocol the internal load balancer's port should use. If omitted, the protocol defaults to TCP.

Minimum Service requirements

Your manifest must contain the following:

• A name for the Service, in this case ilb-service.
• The annotation: cloud.google.com/load-balancer-type: "Internal", which specifies that an internal TCP/UDP load balancer is to be configured.
• The type: LoadBalancer.
• A spec: selector field to specify the Pods the Service should target, for example, app: hello.
• The port, the port over which the Service is exposed, and targetPort, the port on which the containers are listening.

Deploying the Service

To create the internal TCP load balancer, create the file my-service.yaml from the manifest, and then run the following command in your shell or terminal window:

kubectl apply -f my-service.yaml

Inspecting the Service

After deployment, inspect the Service to verify that it has been configured successfully.

gcloud

To inspect the internal TCP/UDP load balancer, run the following command:

kubectl describe service ilb-service

The command's output is similar to the following:

Name:                     ilb-service
Namespace:                default
Labels:                   app=hello
Annotations:              cloud.google.com/load-balancer-type=Internal
Selector:                 app=hello
Type:                     LoadBalancer
IP:                       10.23.250.184
LoadBalancer Ingress:     10.138.0.13
Port:                     80/TCP
TargetPort:               8080/TCP
NodePort:                 31979/TCP
Endpoints:                10.20.0.10:8080,10.20.1.12:8080,10.20.1.13:8080
Session Affinity:         None

LoadBalancer Ingress is the IP address of the internal TCP load balancer.

Console

To inspect the internal TCP/UDP load balancer, perform the following steps:

1. Visit the Google Kubernetes Engine Services menu in GCP Console.

   Visit the Services menu

2. Select the desired Service.

The Service details menu includes the following:

• External endpoints
• Cluster IP
• Load balancer IP
• A list of Pods served by the Service

Verifying the internal TCP load balancer

SSH into a VM instance, and run the following command:

curl [LOAD_BALANCER_IP]

Where [LOAD_BALANCER_IP] is your LoadBalancer Ingress IP address.

The response shows the output of hello-app:

Hello, world!
Version: 2.0.0
Hostname: hello-app-77b45987f7-pw54n

Running the command from outside of the same VPC network or outside the same region results in a timed out error.

Cleaning up

You can delete the Deployment and Service using kubectl delete or GCP Console.

GCLOUD

Delete the Deployment

To delete the Deployment, run the following command:

kubectl delete deployment hello-app

Delete the Service

To delete the Service, run the following command:

kubectl delete service ilb-service

Console

Delete the Deployment

To delete the Deployment, perform the following steps:

1. Visit the Google Kubernetes Engine Workloads menu in GCP Console.

   Visit the Workloads menu

2. From the menu, select the desired workload.

3. Click Delete.

4. From the confirmation dialog menu, click Delete.

Delete the Service

To delete the Service, perform the following steps:

1. Visit the Google Kubernetes Engine Services menu in GCP Console.

   Visit the Services menu

2. From the menu, select the desired Service.

3. Click Delete.

4. From the confirmation dialog menu, click Delete.

Considerations for existing Ingresses

You cannot have both an internal TCP/UDP load balancer and an Ingress that uses balancing mode UTILIZATION. To use both an Ingress and internal TCP/UDP load balancing, the Ingress must use the balancing mode RATE.

If your cluster has an existing Ingress resource created with Kubernetes version 1.7.1 or lower, it is not compatible with internal TCP/UDP load balancers. Earlier BackendService resources created by Kubernetes Ingress Resource objects were created with no balancing mode specified. By default, the API used balancing mode UTILIZATION for HTTP load balancers. However, internal TCP/UDP load balancers cannot be pointed to instance groups with other load balancers using UTILIZATION.

Determining your Ingress balancing mode

To determine what your Ingress balancing mode is, run the following commands from your shell or terminal window:

GROUPNAME=`kubectl get configmaps ingress-uid -o jsonpath='k8s-ig--{.data.uid}' --namespace=kube-system`
gcloud compute backend-services list --format="table(name,backends[].balancingMode,backends[].group)" | grep $GROUPNAME

These commands export a shell variable, GROUPNAME, which fetches your cluster's instance group name. Then, your project's Compute Engine backend service resources are polled and the results are narrowed down based on the contents of $GROUPNAME.

The output is similar to the following:

k8s-be-31210--...  [u'RATE']       us-central1-b/instanceGroups/k8s-ig--...
k8s-be-32125--...  [u'RATE']       us-central1-b/instanceGroups/k8s-ig--...

If the output returns RATE entries or zero entries are returned, then internal load balancers are compatible and no additional work is needed.

If the output returns entries marked with UTILIZATION, your Ingresses are not compatible.

To update your Ingress resources to be compatible with an internal TCP/UDP load balancer, you can create a new cluster running Kubernetes version 1.7.2 or higher, then migrate your services to that cluster.

Note: Ingress balancing mode type can only change when there are no existing HTTP(S) load balancers pointing to the cluster.

Additional Service parameters

Internal TCP/UDP load balancers support Service parameters, such as loadBalancerSourceRanges.

• A spec: loadBalancerSourceRanges array to specify one or more RFC 1918 ranges used by your VPC Networks, Subnetworks, or VPN Gateways. loadBalancerSourceRanges restricts traffic through the load balancer to the IPs specified in this field. If you do not set this field manually, the field defaults to 0.0.0.0, which allows all IPv4 traffic to reach the nodes.

• The spec: loadBalancerIP enables you to choose a specific IP address for the load balancer. The IP address must not be in use by another internal TCP/UDP load balancer or Service. If omitted, an ephemeral IP is assigned. For more information about reservering private IP addresses within subnets, see Reserving a Static Internal IP Address.

For more information about configuring loadBalancerSourceRanges to restrict access to your internal TCP/UDP load balancer, refer to Configure Your Cloud Provider's Firewalls. For more information about the Service specification, see the Service API reference.

Warning: When you create any load balancer Service, unless you set the loadBalancerSourceRanges field yourself, you create a firewall rule based on the field's default IP range, 0.0.0.0/0. Be aware that this may expose other node processes using the same port.

If other processes are bound to the port on a node, the firewall also applies to those processes and the processes are exposed through the node's external IP. GCP does not currently support matching destination IP addresses for incoming traffic. Kubernetes services are still protected by kube-proxy.

Using all ports

If you create an internal TCP/UDP load balancer by using an annotated Service, there is no way to set up a forwarding rule that uses all ports. However, if you create an internal TCP/UDP load balancer manually, you can choose your Google Kubernetes Engine nodes' instance group as the backend. Kubernetes Services of type: NodePort are available through the ILB.

Restrictions for internal TCP/UDP load balancers

• For clusters running Kubernetes version 1.7.3 and earlier, you could only use internal TCP/UDP load balancers with auto-mode subnets, but with Kubernetes version 1.7.4 and later, you can use interal load balancers with custom-mode subnets in addition to auto-mode subnets.
• For clusters running Kubernetes 1.7.X, while the clusterIP remains unchanged, internal TCP/UDP load balancer IP addresses cannot be reserved. Changes made to ports, protocols, or session affinity may cause these IP addresses to change.

Restrictions for internal UDP load balancers

• Internal UDP load balancers do not support using sessionAffinity: ClientIP.

Limits

A Kubernetes Service with type: Loadbalancer and the cloud.google.com/load-balancer-type: Internal annotation creates an ILB that targets the Kubernetes Service. The number of such Services is limited by the number of internal forwarding rules that you can create in a VPC network. For details, see Per network limits.

In a GKE cluster, an internal forwarding rule points to all the nodes in the cluster. Each node in the cluster is a backend VM for the ILB. The maximum number of backend VMs for an ILB is 250, regardless of how the VMs are associated with instance groups. So the maximum number of nodes in a GKE cluster with an ILB is 250. If you have autoscaling enabled for your cluster, you must ensure that autoscaling does not scale your cluster beyond 250 nodes.

For more information about these limits, see VPC Resource Quotas.

What's next

• Read the GKE network overview.
• Learn more about Compute Engine load balancers.
• Learn about Alias IPs.
• Learn about IP masquerade agent.
• Learn about configuring authorized networks.