Controlling scheduling with node taints

This page provides an overview of node taints on Google Kubernetes Engine. When you schedule workloads to be deployed on your cluster, node taints help you control which nodes they are allowed to run on.

Overview

When you submit a workload to run in a cluster, the scheduler determines where to place the Pods associated with the workload. The scheduler is free to place a Pod on any node that satisfies the Pod's CPU, memory, and custom resource requirements.

If your cluster runs a variety of workloads, you might want to exercise some control over which workloads can run on a particular pool of nodes.

A node taint lets you mark a node so that the scheduler avoids or prevents using it for certain Pods. A complementary feature, tolerations, lets you designate Pods that can be used on "tainted" nodes.

Node taints are key-value pairs associated with an effect. Here are the available effects:

• NoSchedule: Pods that do not tolerate this taint are not scheduled on the node.
• PreferNoSchedule: Kubernetes avoids scheduling Pods that do not tolerate this taint onto the node.
• NoExecute: Pod is evicted from the node if it is already running on the node, and is not scheduled onto the node if it is not yet running on the node.

Before you begin

Before you start, make sure you have performed the following tasks:

• Ensure that you have enabled the Google Kubernetes Engine API.
Enable Google Kubernetes Engine API
• Ensure that you have installed the Cloud SDK.

Set up default gcloud settings using one of the following methods:

• Using gcloud init, if you want to be walked through setting defaults.
• Using gcloud config, to individually set your project ID, zone, and region.

Using gcloud init

1. Run gcloud init and follow the directions:

   gcloud init

   If you are using SSH on a remote server, use the --console-only flag to prevent the command from launching a browser:

   gcloud init --console-only

2. Follow the instructions to authorize gcloud to use your Google Cloud account.
3. Create a new configuration or select an existing one.
4. Choose a Google Cloud project.
5. Choose a default Compute Engine zone.

Note: You can override these default settings in gcloud commands using the --project, --zone, and --region flags.

Using gcloud config

• Set your default project ID:

  gcloud config set project [PROJECT_ID]

• If you are working with zonal clusters, set your default compute zone:

  gcloud config set compute/zone [COMPUTE_ZONE]

• If you are working with regional clusters, set your default compute region:

  gcloud config set compute/region [COMPUTE_REGION]

• Update gcloud to the latest version:

  gcloud components update

Note: You can override these default settings in gcloud commands using the --project, --zone, and --region flags.

Creating a cluster with node taints

When you create a cluster, you can assign node taints to the cluster. This assigns the taints to all nodes created with the cluster.

Note: If you create a node pool, the pool does not inherit taints from the cluster. If you want taints on the pool, you must use the --node-taints flag when you create the node pool.

gcloud

To create a cluster with node taints, run the following command:

gcloud container clusters create [CLUSTER_NAME] --node-taints [KEY]=[VALUE]:[EFFECT]

where:

• [CLUSTER_NAME] is the name of the cluster to be created.
• [EFFECT] is PreferNoSchedule, NoSchedule, or NoExecute.
• [KEY] and [VALUE] form a key-value pair associated with the [EFFECT].

For example, this command applies a taint that has key dedicated, value experimental, and effect PreferNoSchedule:

gcloud container clusters create example-cluster --node-taints dedicated=experimental:PreferNoSchedule

Console

To create a cluster with node taints, perform the following steps:

1. Visit the Google Kubernetes Engine menu in Cloud Console.

   Visit the Google Kubernetes Engine menu

2. Click Create cluster.

3. Configure your cluster as desired. Then, click Advanced edit for the node pool you want to modify.

4. In the Node taints section, click Add taint.

5. From Effect, select the desired effect.

6. Fill Key and Value with the desired key-value pair.

7. Add additional node taints if desired.

8. Click Save to close the Advanced edit overlay.

9. Click Create.

API

When you use the API to create a cluster, include the nodeTaints field under nodeConfig. Here's an example:

POST https://container.googleapis.com/v1/projects/[PROJECT_ID]/zones/[ZONE]/clusters

{
  'cluster': {
    'name': 'example-cluster',
    'nodeConfig': {
      'nodeTaints': [
        {
          'key': 'special',
          'Value': 'gpu',
          'effect': 'PreferNoSchedule'
        }
      ]
      ...
    }
    ...
  }
}

Creating a node pool with node taints

When you apply a taint to a node, only Pods that tolerate the taint are allowed to run on the node. In a GKE cluster, you can apply a taint to a node pool, which applies the taint to all nodes in the pool.

To create a node pool with node taints, you can use the gcloud command-line tool or the GKE API.

gcloud

To create a node pool with node taints, run the following command:

gcloud container node-pools create [POOL_NAME] --cluster [CLUSTER_NAME] \
--node-taints [KEY]=[VALUE]:[EFFECT]

where:

• [POOL_NAME] is the name of the node pool to be created.
• [CLUSTER_NAME] is the name of the cluster in which the node pool is created.
• [EFFECT] is PreferNoSchedule, NoSchedule, or NoExecute.
• [KEY] and [VALUE] form a key-value pair associated with the [EFFECT].

For example, this command applies a taint that has key dedicated, value experimental, and effect NoSchedule:

gcloud container node-pools create example-pool --cluster example-cluster \
--node-taints dedicated=experimental:NoSchedule

This command applies a taint that has key special, value gpu, and effect NoExecute:

gcloud container node-pools create example-pool-2 --cluster example-cluster \
--node-taints special=gpu:NoExecute

Console

To create a node pool with node taints, perform the following steps:

1. Visit the Google Kubernetes Engine menu in Cloud Console.

   Visit the Google Kubernetes Engine menu

2. Click the cluster's Edit button, which looks like a pencil.

3. From Node pools click Add node pool.

4. In the Node taints section, click Add taint.

5. From Effect, select the desired effect.

6. Fill Key and Value with the desired key-value pair.

7. Add more node taints if desired.

8. Click Save to exit the node pool modification overlay.

9. Click Create.

API

When you use the API to create a node pool, include the nodeTaints field under nodeConfig. Here's an example:

POST https://container.googleapis.com/v1/projects/[PROJECT_ID]/zones/[ZONE]/clusters/[CLUSTER_ID]/nodePools

{
  'nodePool': {
    'name': 'example-pool',
    'nodeConfig': {
      'nodeTaints': [
        {
          'key': 'dedicated',
          'Value': 'experimental',
          'effect': 'NoSchedule'
        }
      ]
      ...
    }
    ...
  }
}

Configuring Pods to tolerate a taint

You can configure Pods to tolerate a taint by including the tolerations field in the Pods' specification. Here's a portion of a Pod specification.

This Pod can be scheduled on a node that has the taint dedicated=experimental:NoSchedule:

tolerations:
- key: dedicated
  operator: Equal
  value: experimental
  effect: NoSchedule

Adding a taint to an existing node

You can add taints to an existing node using the kubectl taint command:

kubectl taint nodes [NODE_NAME] [KEY]=[VALUE]:[EFFECT]

For example:

kubectl taint nodes node1 key=value:NoSchedule

You can also add taints to nodes with a specific label:

kubectl taint node -l myLabel=X dedicated=foo:PreferNoSchedule

For more information, refer to Taints and tolerations in the Kubernetes documentation.

Inspecting a node's taints

To see a node's taints, you can use the kubectl command-line tool.

First, to get a list of all nodes in your cluster, run the following command:

kubectl get nodes

Then, to inspect a node:

kubectl describe node [NODE_NAME]

In the returned node description, look for the Taints field:

Taints: [KEY]=[VALUE]:[EFFECT]

Removing a taint from a node

You can use kubectl taint to remove taints. You can remove taints by key, key-value, or key-effect.

For example, the following command removes from node foo all the taints with key dedicated:

kubectl taint nodes foo dedicated-

Advantages of using node taints

Using node taints over setting taints manually using kubectl has several advantages:

• Taints are preserved when a node is restarted or replaced.
• Taints are created automatically when a node is added to a pool or cluster.
• Taints are created automatically during cluster autoscaling.

What's next

• Learn more about node pools.
• Read the Kubernetes documentation for taints and tolerations.
• Read the kubectl taint documentation.