Using Node Auto-Provisioning

This page explains how to use Google Kubernetes Engine's node auto-provisioning feature.

Overview

Node auto-provisioning is a mechanism of cluster autoscaler which automatically manages a list of node pools on the user's behalf. Without node auto-provisioning, the cluster autoscaler considers new nodes only from the set of user chosen node pools. With node auto-provisioning, new node pools can be created and deleted automatically.

Before you begin

To prepare for this task, perform the following steps:

  • Ensure that you have enabled the Google Kubernetes Engine API.
  • Enable Google Kubernetes Engine API
  • Ensure that you have installed the Cloud SDK.
  • Set your default project ID:
    gcloud config set project [PROJECT_ID]
  • If you are working with zonal clusters, set your default compute zone:
    gcloud config set compute/zone [COMPUTE_ZONE]
  • If you are working with regional clusters, set your default compute region:
    gcloud config set compute/region [COMPUTE_REGION]
  • Update gcloud to the latest version:
    gcloud components update

Operation

Cluster autoscaler works on per-node pool basis. With node auto-provisioning, the set of autoscaled node pools can be extended automatically based on the specifications of unschedulable Pods.

Node auto-provisioning creates node pools based on the following information:

Resource limits

Node auto-provisioning requires upper limits for CPU and memory. Setting limits for GPUs is optional. Limits are checked against all the nodes in the cluster. By default, auto-provisioned node pools are limited to 1000 nodes.

Cluster autoscaler does not create new nodes if doing so would exceed one of the defined limits. However, if limits are already exceeded, no nodes are automatically deleted.

Workload separation

If there are pending Pods with node affinities and tolerations, indicating workload separation, node auto-provisioner can provision nodes with matching labels and taints.

Currently, node auto-provisioner considers creating node pools of nodes with labels and taints if all of the following conditions are met:

  • A pending Pod requires a node labeled with specific key and value.
  • The Pod has a toleration for a taint with the same key.
  • The toleration is for the NoSchedule effect, NoExecute effect, or all effects.

The Pod's specification can express that it requires nodes with specific labels in two ways:

  • Using a nodeSelector field.
  • Using a nodeAffinity field with an In operator and exactly one value.

The following example is an excerpt of a Pod specification that is interpreted as a workload separation request. The Pod has a toleration for nodes with labels with a key of foo and a value of bar. This example uses nodeAffinity for node selection:

spec:
  tolerations:
  - key: foo
    operator: Equal
    value: bar
    effect: NoSchedule
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: foo
            operator: In
            values:
            - bar

If this Pod exists, nodes with taint key=value:NoSchedule and label key=value are considered for creation.

The example below uses nodeSelector and has the same effect:

spec:
  tolerations:
  - key: foo
    operator: Equal
    value: bar
    effect: NoSchedule
  nodeSelector:
    foo: bar

Deletion of auto-provisioned node pools

When there are no nodes in an auto-provisioned node pool, GKE deletes the node pool. Node pools that are not marked as auto-provisioned are not deleted.

Supported machine types

Currently, node auto-provisioning will only consider creating node pools with machines with up to 16 vCPUs. This limitation will be lifted in the future.

Limit on number of auto-provisioned node pools

There can be no more than 15 auto-provisioned node pools in a cluster. If the limit is reached, no new auto-provisioned node pools are created.

Enabling node auto-provisioning

gcloud

To enable node auto-provisioning, run the following command:

gcloud beta container clusters update [CLUSTER_NAME] --enable-autoprovisioning \
        --max-cpu 10 --max-memory 64

In this command:

  • --enable-autoprovisioning indicates that node auto-provisioning is enabled.
  • --max-cpu specifies the maximum number of cores in the cluster.
  • --max-memory specifies the maximum number of gigabytes of memory in the cluster.

Setting up GPU limits

gcloud

When using node auto-provisioning with GPUs, it's recommended to set the limit for total number of each GPU type in the cluster. To do so, run the following command:

gcloud beta container clusters update [CLUSTER_NAME] --enable-autoprovisioning \
        --max-cpu 10 --max-memory 64 --max-accelerator type=nvidia-tesla-k80,count=4

In this command:

  • --enable-autoprovisioning indicates that node auto-provisioning is enabled.
  • --max-cpu specifies the maximum number of cores in the cluster.
  • --max-memory specifies the maximum number of gigabytes of memory in the cluster.
  • --max-accelerator specify maximum number of nvidia-tesla-k80 GPU accelerators.

Alternatively, you can load limits from a configuration file:

gcloud beta container clusters update [CLUSTER_NAME] --enable-autoprovisioning \
         --autoprovisioning-config-file [FILE_NAME]

In this command: --enable-autoprovisioning indicates that node auto-provisioning is enabled. --autoprovisioning-config-file specifies the file with resource limits. Example configuration:

-resourceType: 'cpu' minimum: 4 maximum: 10 -resourceType: 'memory' maximum: 64 -resourceType: 'nvidia-tesla-k80' maximum: 4 -resourceType: 'nvidia-tesla-v100' maximum: 2

Disabling node auto-provisioning

When you disable node auto-provisioning for a cluster, node pools are no longer auto-provisioned.

gcloud

To disable node auto-provisioning, run the following command:

gcloud beta container clusters update [CLUSTER_NAME] --no-enable-autoprovisioning

In this command:

  • --no-enable-autoprovisioning indicates that node auto-provisioning is disabled.

Marking node pool as auto-provisioned

You can mark any node pool in the cluster as auto-provisioned. Doing so might cause the node pool to be automatically deleted when it is no longer in use.

gcloud

To mark node pool as auto-provisioned, run the following command:

gcloud beta container node-pool update [NODE_POOL_NAME] --enable-autoprovisioning

In this command:

  • --enable-autoprovisioning indicates that node pool is marked as auto-provisioned.

Marking node pool as not auto-provisioned

You can mark a node pool as not auto-provisioned.

gcloud

To mark node pool as not auto-provisioned run following gcloud command:

gcloud beta container node-pool update [NODE_POOL_NAME] --no-enable-autoprovisioning

In this command:

  • --no-enable-autoprovisioning indicates that node pool is marked as not auto-provisioned.

What's next

Was this page helpful? Let us know how we did:

Send feedback about...

Kubernetes Engine