Control scheduling with node taints


This page provides an overview of node taints on Google Kubernetes Engine (GKE). Node taints help you specify the nodes where workloads should run.

Autopilot clusters support using node taints for workload separation only. For instructions, see Configure workload separation in GKE.

Overview

When you submit a workload to run in a cluster, the scheduler determines where to place the Pods associated with the workload. The scheduler is free to place a Pod on any node that satisfies the Pod's CPU, memory, and custom resource requirements.

If your cluster runs a variety of workloads, you might want to exercise some control over which workloads can run on a particular pool of nodes.

A node taint lets you mark a node so that the scheduler avoids or prevents using it for certain Pods. A complementary feature, tolerations, lets you designate Pods that can be used on "tainted" nodes.

Taints and tolerations work together to ensure that Pods are not scheduled onto inappropriate nodes.

Taints are key-value pairs associated with an effect. The following table lists the available effects:

Effect Description
NoSchedule Pods that do not tolerate this taint are not scheduled on the node; existing Pods are not evicted from the node.
PreferNoSchedule Kubernetes avoids scheduling Pods that do not tolerate this taint onto the node.
NoExecute The Pod is evicted from the node if it is already running on the node, and is not scheduled onto the node if it is not yet running on the node.

Advantages of setting node taints in GKE

You can add node taints to clusters and nodes in GKE or by using the kubectl taint command. Specifying node taints in GKE has several advantages over kubectl:

  • Taints are preserved when a node is restarted or replaced.
  • Taints are created automatically when a node is added to a node pool or cluster.
  • Taints are created automatically during cluster autoscaling.

Before you begin

Before you start, make sure you have performed the following tasks:

  • Enable the Google Kubernetes Engine API.
  • Enable Google Kubernetes Engine API
  • If you want to use the Google Cloud CLI for this task, install and then initialize the gcloud CLI. If you previously installed the gcloud CLI, get the latest version by running gcloud components update.

Create a cluster with node taints

When you create a cluster in GKE, you can assign node taints to the cluster. This assigns the taints to all nodes created with the cluster.

If you create a Standard cluster with node taints that have the NoSchedule effect or the NoExecute effect, GKE can't schedule some GKE managed components, such as kube-dns or metrics-server on the default node pool that GKE creates when you create the cluster. GKE can't schedule these components because they don't have the corresponding tolerations for your node taints. You must add a new node pool that satisfies one of the following conditions:

  • No taints
  • A taint that has the PreferNoSchedule effect
  • The components.gke.io/gke-managed-components=true:NoSchedule taint

Any of these conditions allow GKE to schedule GKE managed components in the new node pool.

For instructions, refer to Isolate workloads on dedicated nodes.

gcloud

To create a cluster with node taints, run the following command:

gcloud container clusters create CLUSTER_NAME \
    --node-taints KEY=VALUE:EFFECT

Replace the following:

  • CLUSTER_NAME: the name of the new cluster.
  • EFFECT: one of the following effect values: PreferNoSchedule, NoSchedule, or NoExecute.
  • KEY=VALUE: a key-value pair associated with the EFFECT.

For example, the following command applies a taint that has a key-value of dedicated=experimental with an effect of PreferNoSchedule:

gcloud container clusters create example-cluster \
    --node-taints dedicated=experimental:PreferNoSchedule

Console

To create a cluster with node taints:

  1. Go to the Google Kubernetes Engine page in the Google Cloud console.

    Go to Google Kubernetes Engine

  2. Click Create.

  3. Configure your cluster as desired.

  4. From the navigation pane, under Node Pools, expand the node pool you want to modify, and then click Metadata.

  5. In the Node taints section, click Add Taint.

  6. In the Effect drop-down list, select the desired effect.

  7. Enter the desired key-value pair in the Key and Value fields.

  8. Click Create.

API

When you use the API to create a cluster, include the nodeTaints field under nodeConfig. Here's an example:

POST https://container.googleapis.com/v1/projects/PROJECT_ID/zones/COMPUTE_ZONE/clusters

{
  'cluster': {
    'name': 'example-cluster',
    'nodeConfig': {
      'nodeTaints': [
        {
          'key': 'special',
          'Value': 'gpu',
          'effect': 'PreferNoSchedule'
        }
      ]
      ...
    }
    ...
  }
}

Create a node pool with node taints

When you apply a taint to a node, only Pods that tolerate the taint are allowed to run on the node. In a GKE cluster, you can apply a taint to a node pool, which applies the taint to all nodes in the pool.

To create a node pool with node taints, you can use the Google Cloud CLI, the Google Cloud console, or the GKE API.

gcloud

To create a node pool with node taints, run the following command:

gcloud container node-pools create POOL_NAME \
    --cluster CLUSTER_NAME \
    --node-taints KEY=VALUE:EFFECT

Replace the following:

  • POOL_NAME: the name of the node pool to create.
  • CLUSTER_NAME: the name of the cluster in which the node pool is created.
  • EFFECT: one of the following effect values: PreferNoSchedule, NoSchedule, or NoExecute.
  • KEY=VALUE: a key-value pair associated with the EFFECT.

For example, the following command creates a node pool on an existing cluster and applies a taint that has a key-value of dedicated=experimental with a NoSchedule effect:

gcloud container node-pools create example-pool --cluster example-cluster \
    --node-taints dedicated=experimental:NoSchedule

This command creates a node pool and applies a taint that has key-value of special=gpu with a NoExecute effect:

gcloud container node-pools create example-pool-2 --cluster example-cluster \
    --node-taints special=gpu:NoExecute

Console

To create a node pool with node taints, perform the following steps:

  1. Go to the Google Kubernetes Engine page in the Google Cloud console.

    Go to Google Kubernetes Engine

  2. In the cluster list, click the name of the cluster you want to modify.

  3. On the Cluster details page, click Add Node Pool.

  4. From the navigation pane, click Metadata.

  5. Under Node taints, click Add Taint.

  6. Select the desired effect in the Effect drop-down list.

  7. Enter the desired key-value pair in the Key and Value fields.

  8. Click Create.

API

When you use the API to create a node pool, include the nodeTaints field under nodeConfig. Here's an example:

POST https://container.googleapis.com/v1/projects/PROJECT_ID/zones/COMPUTE_ZONE/clusters/CLUSTER_ID/nodePools

{
  'nodePool': {
    'name': 'example-pool',
    'nodeConfig': {
      'nodeTaints': [
        {
          'key': 'dedicated',
          'Value': 'experimental',
          'effect': 'NoSchedule'
        }
      ]
      ...
    }
    ...
  }
}

Configure Pods to tolerate a taint

You can configure Pods to tolerate a taint by including the tolerations field in the Pods' specification. Here's a portion of a Pod specification.

This Pod can be scheduled on a node that has the dedicated=experimental:NoSchedule taint:

tolerations:
- key: dedicated
  operator: Equal
  value: experimental
  effect: NoSchedule

Add a taint to an existing node

You can add taints to an existing node by using the kubectl taint command:

kubectl taint nodes NODE_NAME KEY=VALUE:EFFECT

For example, the following command applies a taint that has a key-value of dedicated=experimental with a NoSchedule effect to the mynode node:

kubectl taint nodes mynode dedicated=experimental:NoSchedule

You can also add taints to nodes that have a specific label by using the -l selector along with the specified label and value:

kubectl taint nodes -l LABEL=LABEL_VALUE KEY=VALUE:EFFECT

For example, the following command adds a taint with key dedicated-pool to GKE nodes in the my_pool node pool:

kubectl taint nodes -l cloud.google.com/gke-nodepool=my_pool dedicated-pool=my_pool:NoSchedule

Inspect taints for a node

To see the taints for a node, use the kubectl command-line tool.

  1. Get a list of all nodes in your cluster by running the following command:

    kubectl get nodes
    
  2. Inspect a node by running the following command:

    kubectl describe node NODE_NAME
    
  3. In the returned output, look for the Taints field. The output is similar to the following:

    Taints:  dedicated-pool=mypool:NoSchedule
    

Remove a taint from a node

You can use kubectl taint to remove taints. You can remove taints by key, key-value, or key-effect.

For example, the following command removes all the taints with the dedicated key from the mynode node:

kubectl taint nodes mynode dedicated-

Remove all taints from a node pool

To remove all taints from a node pool, run the following command:

gcloud beta container node-pools update POOL_NAME \
--node-taints="" \
--cluster=CLUSTER_NAME

Replace the following:

  • POOL_NAME: the name of the node pool to change.
  • CLUSTER_NAME: the name of the cluster in which the node pool was created.

What's next