Controlling scheduling with node taints

Standard

This page provides an overview of node taints on Google Kubernetes Engine (GKE). When you schedule workloads to be deployed on your cluster, node taints help you control which nodes they are allowed to run on.

Overview

When you submit a workload to run in a cluster, the scheduler determines where to place the Pods associated with the workload. The scheduler is free to place a Pod on any node that satisfies the Pod's CPU, memory, and custom resource requirements.

If your cluster runs a variety of workloads, you might want to exercise some control over which workloads can run on a particular pool of nodes.

A node taint lets you mark a node so that the scheduler avoids or prevents using it for certain Pods. A complementary feature, tolerations, lets you designate Pods that can be used on "tainted" nodes.

Node taints are key-value pairs associated with an effect. Here are the available effects:

NoSchedule: Pods that do not tolerate this taint are not scheduled on the node; existing Pods are not evicted from the node.
PreferNoSchedule: Kubernetes avoids scheduling Pods that do not tolerate this taint onto the node.
NoExecute: Pod is evicted from the node if it is already running on the node, and is not scheduled onto the node if it is not yet running on the node.

Note that some system Pods (for example, kube-proxy and fluent-bit) tolerate all NoExecute and NoSchedule taints, and will not be evicted.

Before you begin

Before you start, make sure you have performed the following tasks:

Ensure that you have enabled the Google Kubernetes Engine API.

Enable Google Kubernetes Engine API

Ensure that you have installed the Cloud SDK.

Set up default gcloud settings using one of the following methods:

Using gcloud init, if you want to be walked through setting defaults.
Using gcloud config, to individually set your project ID, zone, and region.

Using gcloud init

If you receive the error One of [--zone, --region] must be supplied: Please specify location, complete this section.

Run gcloud init and follow the directions:
```
gcloud init
```
If you are using SSH on a remote server, use the --console-only flag to prevent the command from launching a browser:
```
gcloud init --console-only
```
Follow the instructions to authorize gcloud to use your Google Cloud account.
Create a new configuration or select an existing one.
Choose a Google Cloud project.
Choose a default Compute Engine zone for zonal clusters or a region for regional or Autopilot clusters.

Note: You can override these default settings in gcloud commands using the --project, --zone, and --region flags.

Using gcloud config

Set your default project ID:
```
gcloud config set project PROJECT_ID
```
If you are working with zonal clusters, set your default compute zone:
```
gcloud config set compute/zone COMPUTE_ZONE
```
If you are working with Autopilot or regional clusters, set your default compute region:
```
gcloud config set compute/region COMPUTE_REGION
```
Update gcloud to the latest version:
```
gcloud components update
```

Note: You can override these default settings in gcloud commands using the --project, --zone, and --region flags.

Creating a cluster with node taints

When you create a cluster, you can assign node taints to the cluster. This assigns the taints to all nodes created with the cluster.

gcloud

To create a cluster with node taints, run the following command:

gcloud container clusters create cluster-name \
  --node-taints key=value:effect

Replace the following:

cluster-name is the name of the cluster to be created.
effect is PreferNoSchedule, NoSchedule, or NoExecute.
key and value form a key-value pair associated with the effect.

For example, the following command applies a taint that has key dedicated, value experimental, and effect PreferNoSchedule:

gcloud container clusters create example-cluster \
  --node-taints dedicated=experimental:PreferNoSchedule

Console

To create a cluster with node taints:

Go to the Google Kubernetes Engine page in Cloud Console.

Go to Google Kubernetes Engine
Click Create.
Configure your cluster as desired.
From the navigation pane, under Node Pools, expand the node pool you want to modify, and then click Metadata.
In the Node taints section, click Add Taint.
In the Effect drop-down list, select the desired effect.
Enter the desired key-value pair in the Key and Value fields.
Click Create.

API

When you use the API to create a cluster, include the nodeTaints field under nodeConfig. Here's an example:

POST https://container.googleapis.com/v1/projects/project-id/zones/compute-zone/clusters

{
  'cluster': {
    'name': 'example-cluster',
    'nodeConfig': {
      'nodeTaints': [
        {
          'key': 'special',
          'Value': 'gpu',
          'effect': 'PreferNoSchedule'
        }
      ]
      ...
    }
    ...
  }
}

Creating a node pool with node taints

When you apply a taint to a node, only Pods that tolerate the taint are allowed to run on the node. In a GKE cluster, you can apply a taint to a node pool, which applies the taint to all nodes in the pool.

To create a node pool with node taints, you can use the gcloud command-line tool, Cloud Console, or the GKE API.

gcloud

To create a node pool with node taints, run the following command:

gcloud container node-pools create pool-name \
  --cluster cluster-name \
  --node-taints key=value:effect

Replace the following:

pool-name is the name of the node pool to be created.
cluster-name is the name of the cluster in which the node pool is created.
effect is PreferNoSchedule, NoSchedule, or NoExecute.
key and value form a key-value pair associated with the effect.

For example, the following command applies a taint that has key dedicated, value experimental, and effect NoSchedule:

gcloud container node-pools create example-pool --cluster example-cluster \
  --node-taints dedicated=experimental:NoSchedule

This command applies a taint that has key special, value gpu, and effect NoExecute:

gcloud container node-pools create example-pool-2 --cluster example-cluster \
  --node-taints special=gpu:NoExecute

Console

To create a node pool with node taints, perform the following steps:

Visit the Google Kubernetes Engine menu in Cloud Console.

Visit the Google Kubernetes Engine menu
In the cluster list, click the name of the cluster you want to modify.
On the Cluster details page, click Add Node Pool.
From the navigation pane, click Metadata.
Under Node taints, click Add Taint.
Select the desired effect in the Effect drop-down list.
Enter the desired key-value pair in the Key and Value fields.
Click Create.

API

When you use the API to create a node pool, include the nodeTaints field under nodeConfig. Here's an example:

POST https://container.googleapis.com/v1/projects/project-id/zones/compute-zone/clusters/cluster-id/nodePools

{
  'nodePool': {
    'name': 'example-pool',
    'nodeConfig': {
      'nodeTaints': [
        {
          'key': 'dedicated',
          'Value': 'experimental',
          'effect': 'NoSchedule'
        }
      ]
      ...
    }
    ...
  }
}

Configuring Pods to tolerate a taint

You can configure Pods to tolerate a taint by including the tolerations field in the Pods' specification. Here's a portion of a Pod specification.

This Pod can be scheduled on a node that has the taint dedicated=experimental:NoSchedule:

tolerations:
- key: dedicated
  operator: Equal
  value: experimental
  effect: NoSchedule

Adding a taint to an existing node

You can add taints to an existing node using the kubectl taint command:

kubectl taint nodes node-name key=value:effect

For example:

kubectl taint nodes node1 key=value:NoSchedule

You can also add taints to nodes with a specific label:

kubectl taint node -l myLabel=X dedicated=foo:PreferNoSchedule

For more information, refer to Taints and tolerations in the Kubernetes documentation.

Inspecting a node's taints

To see a node's taints, you can use the kubectl command-line tool.

Get a list of all nodes in your cluster by running the following command:
```
kubectl get nodes
```
Inspect a node by running the following command:
```
kubectl describe node node-name
```
In the returned node description, look for the Taints field:
```
Taints: key=value:effect
```

Removing a taint from a node

You can use kubectl taint to remove taints. You can remove taints by key, key-value, or key-effect.

For example, the following command removes from node foo all the taints with key dedicated:

kubectl taint nodes foo dedicated-

Advantages of using the GKE node taints feature

Using the GKE node taints feature with clusters and node pools instead of setting taints manually by using kubectl has several advantages:

Taints are preserved when a node is restarted or replaced.
Taints are created automatically when a node is added to a pool or cluster.
Taints are created automatically during cluster autoscaling.

Controlling scheduling with node taints

Overview

Before you begin

Using gcloud init

Using gcloud config

Creating a cluster with node taints

gcloud

Console

API

Creating a node pool with node taints

gcloud

Console

API

Configuring Pods to tolerate a taint

Adding a taint to an existing node

Inspecting a node's taints

Removing a taint from a node

Advantages of using the GKE node taints feature

What's next