Controlling Scheduling with Node Taints

This page provides an overview of node taints on Kubernetes Engine. When you schedule workloads to be deployed on your cluster, node taints help you control which nodes they are allowed to run on.

Overview

When you submit a workload to run in a cluster, the scheduler determines where to place the Pods associated with the workload. The scheduler is free to place a Pod on any node that satisfies the Pod's CPU, memory, and custom resource requirements.

If your cluster runs a variety of workloads, you might want to exercise some control over which workloads can run on a particular pool of nodes.

A node taint lets you mark a node so that the scheduler avoids or prevents using it for certain Pods. A complementary feature, tolerations, lets you designate Pods that can be used on "tainted" nodes.

Node taints are key-value pairs associated with an effect. Here are the available effects:

  • NoSchedule: Pods that do not tolerate this taint are not scheduled on the node.
  • PreferNoSchedule: Kubernetes avoids scheduling Pods that do not tolerate this taint onto the node.
  • NoExecute: Pod is evicted from the node if it is already running on the node, and is not scheduled onto the node if it is not yet running on the node.

Before you begin

To prepare for this task, perform the following steps:

  • Ensure that you have enabled the Kubernetes Engine API.
  • Enable Kubernetes Engine API
  • Ensure that you have installed the Cloud SDK.
  • Set your default project ID:
    gcloud config set project [PROJECT_ID]
  • Set your default compute zone:
    gcloud config set compute/zone [COMPUTE_ZONE]
  • Update all gcloud commands to the latest version:
    gcloud components update

Creating a cluster with node taints

When you create a cluster, you can assign node taints to the cluster. This assigns the taints to all nodes created with the cluster.

gcloud

To create a cluster with node taints, run the following command:

gcloud container clusters create [CLUSTER_NAME] --node-taints [KEY]=[VALUE]:[EFFECT]

where:

  • [CLUSTER_NAME] is the name of the cluster to be created.
  • [EFFECT] is PreferNoSchedule, NoSchedule, or NoExecute.
  • [KEY] and [VALUE] form a key-value pair associated with the [EFFECT].

For example, this command applies a taint that has key dedicated, value experimental, and effect PreferNoSchedule:

gcloud container clusters create example-cluster --node-taints dedicated=experimental:PreferNoSchedule

Console

To create a cluster with node taints, perform the following steps:

  1. Visit the Kubernetes Engine menu in GCP Console.

    Visit the Kubernetes Engine menu

  2. Click Create cluster.

  3. Configure your cluster as desired. Then, click More.
  4. From Node taints, click Add taint.
  5. From Effect, select the desired effect.
  6. Fill Key and Value with the desired key-value pair.
  7. Click Create.

API

When you use the API to create a cluster, include the nodeTaints field under nodeConfig. Here's an example:

POST https://container.googleapis.com/v1/projects/[PROJECT_ID]/zones/[ZONE]/clusters

{
  'cluster': {
    'name': 'example-cluster',
    'nodeConfig': {
      'nodeTaints': [
        {
          'key': 'special',
          'Value': 'gpu',
          'effect': 'PreferNoSchedule'
        }
      ]
      ...
    }
    ...
  }
}

Creating a node pool with node taints

When you apply a taint to a node, only Pods that tolerate the taint are allowed to run on the node. In a Kubernetes Engine cluster, you can apply a taint to a node pool, which applies the taint to all nodes in the pool.

To create a node pool with node taints, you can use the gcloud command-line tool or the Kubernetes Engine API.

gcloud

To create a node pool with node taints, run the following command:

gcloud container node-pools create [POOL_NAME] --cluster [CLUSTER_NAME] \
--node-taints [KEY]=[VALUE]:[EFFECT]

where:

  • [POOL_NAME] is the name of the node pool to be created.
  • [CLUSTER_NAME] is the name of the cluster in which the node pool is created.
  • [EFFECT] is PreferNoSchedule, NoSchedule, or NoExecute.
  • [KEY] and [VALUE] form a key-value pair associated with the [EFFECT].

For example, this command applies a taint that has key dedicated, value experimental, and effect NoSchedule:

gcloud container node-pools create example-pool --cluster example-cluster \
--node-taints dedicated=experimental:NoSchedule

This command applies a taint that has key special, value gpu, and effect NoExecute:

gcloud container node-pools create example-pool-2 --cluster example-cluster \
--node-taints special=gpu:NoExecute

Console

To create a node pool with node taints, perform the following steps:

  1. Visit the Kubernetes Engine menu in GCP Console.

    Visit the Kubernetes Engine menu

  2. Select the desired cluster.

  3. From Node pools click Add node pool.
  4. From Node taints, click Add taint.
  5. From Effect, select the desired effect.
  6. Fill Key and Value with the desired key-value pair.
  7. Click Create.

API

When you use the API to create a node pool, include the nodeTaints field under nodeConfig. Here's an example:

POST https://container.googleapis.com/v1/projects/[PROJECT_ID]/zones/[ZONE]/clusters/[CLUSTER_ID]/nodePools

{
  'nodePool': {
    'name': 'example-pool',
    'nodeConfig': {
      'nodeTaints': [
        {
          'key': 'dedicated',
          'Value': 'experimental',
          'effect': 'NoSchedule'
        }
      ]
      ...
    }
    ...
  }
}

Configuring Pods to tolerate a taint

You can configure Pods to tolerate a taint by including the tolerations field in the Pods' specification. Here's a portion of a Pod specification.

This Pod can be scheduled on a node that has the taint dedicated=experimental:NoSchedule:

tolerations:
- key: dedicated
  operator: Equal
  value: experimental
  effect: NoSchedule

Adding a taint to an existing node

You can add taints to an existing node using the kubectl taint command:

kubectl taint nodes [NODE_NAME] [KEY]=[VALUE]:[EFFECT]

For example:

kubectl taint nodes node1 key=value:NoSchedule

You can also add taints to nodes with a specific label:

kubectl taint node -l myLabel=X dedicated=foo:PreferNoSchedule

For more information, refer to Taints and tolerations in the Kubernetes documentation.

Inspecting a node's taints

To see a node's taints, you can use the kubectl command-line tool.

First, to get a list of all nodes in your cluster, run the following command:

kubectl get nodes

Then, to inspect a node:

kubectl describe node [NODE_NAME]

In the returned node description, look for the Taints field:

Taints: [KEY]=[VALUE]:[EFFECT]

Removing a taint from a node

You can use kubectl taint to remove taints. You can remove taints by key, key-value, or key-effect.

For example, the following command removes from node foo all the taints with key dedicated:

kubectl taint nodes foo dedicated-

Advantages of using node taints

Using node taints over setting taints manually using kubectl has several advantages:

  • Taints are preserved when a node is restarted or replaced.
  • Taints are created automatically when a node is added to a pool or cluster.
  • Taints are created automatically during cluster autoscaling.

What's next

Was this page helpful? Let us know how we did:

Send feedback about...

Kubernetes Engine