Control scheduling with node taints

Standard

This page provides an overview of node taints on Google Kubernetes Engine (GKE). When you schedule workloads to be deployed on your cluster, node taints help you control which nodes they are allowed to run on.

Overview

When you submit a workload to run in a cluster, the scheduler determines where to place the Pods associated with the workload. The scheduler is free to place a Pod on any node that satisfies the Pod's CPU, memory, and custom resource requirements.

If your cluster runs a variety of workloads, you might want to exercise some control over which workloads can run on a particular pool of nodes.

A node taint lets you mark a node so that the scheduler avoids or prevents using it for certain Pods. A complementary feature, tolerations, lets you designate Pods that can be used on "tainted" nodes.

Taints and tolerations work together to ensure that Pods are not scheduled onto inappropriate nodes.

Taints are key-value pairs associated with an effect. The following table lists the available effects:

Effect	Description
`NoSchedule`	Pods that do not tolerate this taint are not scheduled on the node; existing Pods are not evicted from the node.
`PreferNoSchedule`	Kubernetes avoids scheduling Pods that do not tolerate this taint onto the node.
`NoExecute`	The Pod is evicted from the node if it is already running on the node, and is not scheduled onto the node if it is not yet running on the node.

Advantages of setting node taints in GKE

You can add node taints to clusters and nodes in GKE or by using the kubectl taint command. Specifying node taints in GKE has several advantages over kubectl:

Taints are preserved when a node is restarted or replaced.
Taints are created automatically when a node is added to a node pool or cluster.
Taints are created automatically during cluster autoscaling.

Before you begin

Before you start, make sure you have performed the following tasks:

Ensure that you have enabled the Google Kubernetes Engine API.

Enable Google Kubernetes Engine API

Ensure that you have installed the Google Cloud CLI.
Set up default Google Cloud CLI settings for your project by using one of the following methods:

Use gcloud init, if you want to be walked through setting project defaults.
Use gcloud config, to individually set your project ID, zone, and region.

gcloud init

Run gcloud init and follow the directions:
```
gcloud init
```
If you are using SSH on a remote server, use the --console-only flag to prevent the command from launching a browser:
```
gcloud init --console-only
```
Follow the instructions to authorize the gcloud CLI to use your Google Cloud account.
Create a new configuration or select an existing one.
Choose a Google Cloud project.
Choose a default Compute Engine zone.
Choose a default Compute Engine region.

Note: You can override these default settings in the gcloud CLI by using the --project, --zone, and --region flags.

gcloud config

Set your default project ID:
```
gcloud config set project PROJECT_ID
```
Set your default Compute Engine region (for example, us-central1):
```
gcloud config set compute/region COMPUTE_REGION
```
Set your default Compute Engine zone (for example, us-central1-c):
```
gcloud config set compute/zone COMPUTE_ZONE
```
Update gcloud to the latest version:
```
gcloud components update
```

Note: You can override these default settings in the gcloud CLI by using the --project, --zone, and --region flags.

By setting default locations, you can avoid errors in gcloud CLI like the following: One of [--zone, --region] must be supplied: Please specify location.

Create a cluster with node taints

When you create a cluster in GKE, you can assign node taints to the cluster. This assigns the taints to all nodes created with the cluster.

gcloud

To create a cluster with node taints, run the following command:

gcloud container clusters create CLUSTER_NAME \
    --node-taints KEY=VALUE:EFFECT

Replace the following:

CLUSTER_NAME: the name of the new cluster.
EFFECT: one of the following effect values: PreferNoSchedule, NoSchedule, or NoExecute.
KEY=VALUE: a key-value pair associated with the EFFECT.

For example, the following command applies a taint that has a key-value of dedicated=experimental with an effect of PreferNoSchedule:

gcloud container clusters create example-cluster \
    --node-taints dedicated=experimental:PreferNoSchedule

Console

To create a cluster with node taints:

Go to the Google Kubernetes Engine page in the Google Cloud console.

Go to Google Kubernetes Engine
Click Create.
Configure your cluster as desired.
From the navigation pane, under Node Pools, expand the node pool you want to modify, and then click Metadata.
In the Node taints section, click Add Taint.
In the Effect drop-down list, select the desired effect.
Enter the desired key-value pair in the Key and Value fields.
Click Create.

API

When you use the API to create a cluster, include the nodeTaints field under nodeConfig. Here's an example:

POST https://container.googleapis.com/v1/projects/PROJECT_ID/zones/COMPUTE_ZONE/clusters

{
  'cluster': {
    'name': 'example-cluster',
    'nodeConfig': {
      'nodeTaints': [
        {
          'key': 'special',
          'Value': 'gpu',
          'effect': 'PreferNoSchedule'
        }
      ]
      ...
    }
    ...
  }
}

Create a node pool with node taints

When you apply a taint to a node, only Pods that tolerate the taint are allowed to run on the node. In a GKE cluster, you can apply a taint to a node pool, which applies the taint to all nodes in the pool.

To create a node pool with node taints, you can use the Google Cloud CLI, Google Cloud console, or the GKE API.

gcloud

To create a node pool with node taints, run the following command:

gcloud container node-pools create POOL_NAME \
    --cluster CLUSTER_NAME \
    --node-taints KEY=VALUE:EFFECT

Replace the following:

POOL_NAME: the name of the node pool to create.
CLUSTER_NAME: the name of the cluster in which the node pool is created.
EFFECT: one of the following effect values: PreferNoSchedule, NoSchedule, or NoExecute.
KEY=VALUE: a key-value pair associated with the EFFECT.

For example, the following command creates a node pool on an existing cluster and applies a taint that has a key-value of dedicated=experimental with a NoSchedule effect:

gcloud container node-pools create example-pool --cluster example-cluster \
    --node-taints dedicated=experimental:NoSchedule

This command creates a node pool and applies a taint that has key-value of special=gpu with a NoExecute effect:

gcloud container node-pools create example-pool-2 --cluster example-cluster \
    --node-taints special=gpu:NoExecute

Console

To create a node pool with node taints, perform the following steps:

Go to the Google Kubernetes Engine page in Google Cloud console.

Go to Google Kubernetes Engine
In the cluster list, click the name of the cluster you want to modify.
On the Cluster details page, click Add Node Pool.
From the navigation pane, click Metadata.
Under Node taints, click Add Taint.
Select the desired effect in the Effect drop-down list.
Enter the desired key-value pair in the Key and Value fields.
Click Create.

API

When you use the API to create a node pool, include the nodeTaints field under nodeConfig. Here's an example:

POST https://container.googleapis.com/v1/projects/PROJECT_ID/zones/COMPUTE_ZONE/clusters/CLUSTER_ID/nodePools

{
  'nodePool': {
    'name': 'example-pool',
    'nodeConfig': {
      'nodeTaints': [
        {
          'key': 'dedicated',
          'Value': 'experimental',
          'effect': 'NoSchedule'
        }
      ]
      ...
    }
    ...
  }
}

Configure Pods to tolerate a taint

You can configure Pods to tolerate a taint by including the tolerations field in the Pods' specification. Here's a portion of a Pod specification.

This Pod can be scheduled on a node that has the dedicated=experimental:NoSchedule taint:

tolerations:
- key: dedicated
  operator: Equal
  value: experimental
  effect: NoSchedule

Add a taint to an existing node

You can add taints to an existing node by using the kubectl taint command:

kubectl taint nodes NODE_NAME KEY=VALUE:EFFECT

For example, the following command applies a taint that has a key-value of dedicated=experimental with a NoSchedule effect to the mynode node:

kubectl taint nodes mynode dedicated=experimental:NoSchedule

You can also add taints to nodes that have a specific label by using the -l selector along with the specified label and value:

kubectl taint nodes -l LABEL=LABEL_VALUE KEY=VALUE:EFFECT

Note: In GKE, all node pools have a label in the following format (where POOL_NAME is the name of the node pool):

cloud.google.com/gke-nodepool:POOL_NAME

For example, the following command adds a taint with key dedicated-pool to GKE nodes in the my_pool node pool:

kubectl taint nodes -l cloud.google.com/gke-nodepool=my_pool dedicated-pool=my_pool:NoSchedule

Inspect taints for a node

To see the taints for a node, use the kubectl command-line tool.

Get a list of all nodes in your cluster by running the following command:
```
kubectl get nodes
```
Inspect a node by running the following command:
```
kubectl describe node NODE_NAME
```
In the returned output, look for the Taints field. The output is similar to the following:
```
Taints:  dedicated-pool=mypool:NoSchedule
```

Remove a taint from a node

You can use kubectl taint to remove taints. You can remove taints by key, key-value, or key-effect.

For example, the following command removes all the taints with the dedicated key from the mynode node:

kubectl taint nodes mynode dedicated-

Remove all taints from a node pool

To remove all taints from a node pool, run the following command:

gcloud beta container node-pools update POOL_NAME \
--node-taints="" \
--cluster=CLUSTER_NAME

Replace the following:

POOL_NAME: the name of the node pool to change.
CLUSTER_NAME: the name of the cluster in which the node pool was created.

Control scheduling with node taints

Overview

Advantages of setting node taints in GKE

Before you begin

gcloud init

gcloud config

Create a cluster with node taints

gcloud

Console

API

Create a node pool with node taints

gcloud

Console

API

Configure Pods to tolerate a taint

Add a taint to an existing node

Inspect taints for a node

Remove a taint from a node

Remove all taints from a node pool

What's next