Controlling Scheduling with Node Taints

This page provides an overview of node taints on Kubernetes Engine. When you schedule workloads to be deployed on your cluster, node taints help you control which nodes they are allowed to run on.

Overview

When you submit a workload to run in a cluster, the scheduler determines where to place the Pods associated with the workload. The scheduler is free to place a Pod on any node that satisfies the Pod's CPU, memory, and custom resource requirements.

If your cluster runs a variety of workloads, you might want to exercise some control over which workloads can run on a particular pool of nodes.

A node taint lets you mark a node so that the scheduler avoids or prevents using it for certain Pods. A complementary feature, tolerations, lets you designate Pods that can be used on "tainted" nodes.

Node taints are key-value pairs associated with an effect. Here are the available effects:

  • NoSchedule: Pods that do not tolerate this taint are not scheduled on the node.
  • PreferNoSchedule: Kubernetes avoids scheduling Pods that do not tolerate this taint onto the node.
  • NoExecute: Pod is evicted from the node if it is already running on the node, and is not scheduled onto the node if it is not yet running on the node.

Creating a cluster with node taints

When you create a cluster, you can assign node taints to the cluster. This assigns the taints to all nodes created with the cluster.

gcloud

The gcloud command for creating a cluster with node taints has this form:

gcloud beta container clusters create [CLUSTER_NAME] --node-taints=[KEY]=[VALUE]:[EFFECT]

where:

  • [CLUSTER_NAME] is the name of the cluster to be created.
  • [EFFECT] is PreferNoSchedule, NoSchedule, or NoExecute.
  • [KEY] and [VALUE] form a key-value pair associated with the [EFFECT].

For example, this command applies a taint that has key dedicated, value experimental, and effect PreferNoSchedule:

gcloud beta container clusters create example-cluster --node-taints=dedicated=experimental:PreferNoSchedule

API

When you use the API to create a cluster, include the nodeTaints field under nodeConfig. Here's an example:

POST https://container.googleapis.com/v1/projects/[PROJECT-ID]/zones/[ZONE]/clusters

{
  'cluster': {
    'name': 'example-cluster',
    'nodeConfig': {
      'nodeTaints': [
        {
          'key': 'special',
          'Value': 'gpu',
          'effect': 'PreferNoSchedule'
        }
      ]
      ...
    }
    ...
  }
}

Creating a node pool with node taints

When you apply a taint to a node, only Pods that tolerate the taint are allowed to run on the node. In a Kubernetes Engine cluster, you can apply a taint to a node pool, which applies the taint to all nodes in the pool.

To create a node pool with node taints, you can use the gcloud command-line tool or the Kubernetes Engine API.

gcloud

The gcloud command for applying taints to a node pool has this format:

gcloud beta container node-pools create [POOL_NAME] --cluster=[CLUSTER_NAME] --node-taints=[KEY]=[VALUE]:[EFFECT]

where:

  • [CLUSTER_NAME] is the name of your cluster.
  • [POOL_NAME] is the name of the node pool to be created.
  • [EFFECT] is PreferNoSchedule, NoSchedule, or NoExecute.
  • [KEY] and [VALUE] form a key-value pair associated with the [EFFECT].

For example, this command applies a taint that has key dedicated, value experimental, and effect NoSchedule:

gcloud beta container node-pools create example-pool --cluster=example-cluster --node-taints=dedicated=experimental:NoSchedule

This command applies a taint that has key special, value gpu, and effect NoExecute:

gcloud beta container node-pools create example-pool-2 --cluster=example-cluster --node-taints=special=gpu:NoExecute

API

When you use the API to create a node pool, include the nodeTaints field under nodeConfig. Here's an example:

POST https://container.googleapis.com/v1/projects/[PROJECT-ID]/zones/[ZONE]/clusters/[CLUSTER-ID]/nodePools

{
  'nodePool': {
    'name': 'example-pool',
    'nodeConfig': {
      'nodeTaints': [
        {
          'key': 'dedicated',
          'Value': 'experimental',
          'effect': 'NoSchedule'
        }
      ]
      ...
    }
    ...
  }
}

Configuring Pods to tolerate a taint

You can configure Pods to tolerate a taint by including the tolerations field in the Pods' specification. Here's a portion of a Pod specification.

This Pod can be scheduled on a node that has the taint dedicated=experimental:NoSchedule:

tolerations:
- key: dedicated
  operator: Equal
  value: experimental
  effect: NoSchedule

Adding a taint to an existing node

You can add taints to an existing node using the kubectl taint command:

kubectl taint nodes [NODE-NAME] [KEY]=[VALUE]:[EFFECT]

For example:

kubectl taint nodes node1 key=value:NoSchedule

You can also add taints to nodes with a specific label:

kubectl taint node -l myLabel=X dedicated=foo:PreferNoSchedule

For more information, refer to Taints and tolerations in the Kubernetes documentation.

Inspecting a node's taints

To see a node pool's taints, you can use the kubectl command-line tool.

Run the following command:

kubectl describe node [POOL-NAME]

In the returned node description, look for the Taints field:

Taints: [KEY]=[VALUE]:[EFFECT]

Removing a taint from a node

You can use kubectl taint to remove taints. You can remove taints by key, key-value, or key-effect.

For example, the following command removes from node foo all the taints with key dedicated:

kubectl taint nodes foo dedicated-

Advantages of using node taints

Using node taints over setting taints manually using kubectl has several advantages:

  • Taints are preserved when a node is restarted or replaced.
  • Taints are created automatically when a node is added to a pool or cluster.
  • Taints are created automatically during cluster autoscaling.

What's next

Send feedback about...

Kubernetes Engine