Using node auto-provisioning

This page explains how to use Google Kubernetes Engine (GKE)'s Node auto-provisioning feature.

Overview

Node auto-provisioning automatically manages a set of node pools on the user's behalf. Without Node auto-provisioning, GKE considers starting new nodes only from the set of user created node pools. With node auto-provisioning, new node pools can be created and deleted automatically.

Before you begin

Before you start, make sure you have performed the following tasks:

Set up default gcloud settings using one of the following methods:

  • Using gcloud init, if you want to be walked through setting defaults.
  • Using gcloud config, to individually set your project ID, zone, and region.

Using gcloud init

If you receive the error One of [--zone, --region] must be supplied: Please specify location, complete this section.

  1. Run gcloud init and follow the directions:

    gcloud init

    If you are using SSH on a remote server, use the --console-only flag to prevent the command from launching a browser:

    gcloud init --console-only
  2. Follow the instructions to authorize gcloud to use your Google Cloud account.
  3. Create a new configuration or select an existing one.
  4. Choose a Google Cloud project.
  5. Choose a default Compute Engine zone.

Using gcloud config

  • Set your default project ID:
    gcloud config set project project-id
  • If you are working with zonal clusters, set your default compute zone:
    gcloud config set compute/zone compute-zone
  • If you are working with regional clusters, set your default compute region:
    gcloud config set compute/region compute-region
  • Update gcloud to the latest version:
    gcloud components update

Requirements

Node auto-provisioning is available in GKE Release:

  • v1.11.2-gke.25 and higher for zonal clusters
  • v1.12.x and higher for regional clusters

Unsupported features

The following features are not supported by Node auto-provisioning, meaning that Node auto-provisioning will not provision node pools with these features, but will autoscale existing node pools:

  • Local SSDs consumed through by hostPath volumes. Autoscaling local PersistentVolumes is not supported.
  • Compute Engine instance families other than n1 (n2, e2, c2, ...).
  • Operating systems other than COS.
  • gvisor
  • Custom shielded VM settings. Auto-provisioned node-pools will use the default settings for shielded VMs.
  • Controlling reservation affinity.
  • Auto-provisioning node pools for Pods using ephemeral storage.
  • Auto-provisioning through custom scheduling that uses altered Filters.

Operation

Node auto-provisioning is a mechanism of the cluster autoscaler, which scales on a per-node pool basis. With Node auto-provisioning enabled, the cluster autoscaler can extend node pools automatically based on the specifications of unschedulable Pods.

Node auto-provisioning creates node pools based on the following information:

Resource limits

Node auto-provisioning and the cluster autoscaler have limits at two levels:

  • Node pool level
  • Cluster level

Limits for node pools

Node pools created by Node auto-provisioning are limited to 1000 nodes.

Limits for clusters

The limits you define are enforced based on the total CPU and memory resources used across your cluster, not just auto-provisioned pools.

Cluster autoscaler does not create new nodes if doing so would exceed one of the defined limits. If limits are already exceeded, nodes are not automatically deleted.

Workload separation

If pending Pods have node affinities and tolerations, node auto-provisioning can provision nodes with matching labels and taints.

Node auto-provisioning might create node pools with labels and taints if all of the following conditions are met:

  • A pending Pod requires a node with a specific label key and value.
  • The Pod has a toleration for a taint with the same key.
  • The toleration is for the NoSchedule effect, NoExecute effect, or all effects.

The Pod's specification can express that it requires nodes with specific labels in two ways:

  • Using a nodeSelector field.
  • Using a nodeAffinity field with an In operator and exactly one value.

The following example is an excerpt of a Pod specification that is interpreted as a workload separation request. In this example, the cluster administrator has chosen dedicated as the key that will be used for workload isolation, and the UI team has determined that they need dedicated nodes for their workloads.

The Pod has a toleration for nodes labeled with dedicated=ui-team and uses nodeAffinity for node selection:

spec:
  tolerations:
  - key: dedicated
    operator: Equal
    value: ui-team
    effect: NoSchedule
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: dedicated
            operator: In
            values:
            - ui-team

If this Pod exists, nodes with taint dedicated=ui-team:NoSchedule and label dedicated=ui-team are considered for creation by Node auto-provisioning .

The following example uses nodeSelector and has the same effect:

spec:
  tolerations:
  - key: dedicated
    operator: Equal
    value: ui-team
    effect: NoSchedule
  nodeSelector:
    dedicated: ui-team

Deletion of auto-provisioned node pools

When there are no nodes in an auto-provisioned node pool, GKE deletes the node pool. Node pools that are not marked as auto-provisioned are not deleted.

Supported machine types

Currently, Node auto-provisioning will only consider creating node pools with machines from N1 machine types with up to 64 vCPUs.

Supported node images

Currently Node auto-provisioning will only consider creating node pools with Container-Optimized OS node image.

Support for preemptible VMs

Node auto-provisioning supports creating node pools based on preemptible virtual machine (VM) instances.

Creating preemptible VM-based node pools is only considered if unschedulable pods with toleration for cloud.google.com/gke-preemptible="true":NoSchedule taint exist.

The nodes that are based on preemptible VMs and created by auto-provisioned node pools have the taint applied.

Scalability limitations

Node auto-provisioning has the same limitations as the cluster autoscaler, as well as the following additional limitations:

Limit on number of separated workloads
Node auto-provisioning supports a maximum of 100 distinct separated workloads.
Limit on number of node pools
Node auto-provisioning de-prioritizes creating new node pools when the number of pools approaches 100. Creating over 100 node pools is possible but taken when creating a node pool is the only option to schedule a pending pod.

Enabling Node auto-provisioning

You enable Node auto-provisioning on a cluster with gcloud or the Google Cloud Console.

gcloud

To enable Node auto-provisioning, run the following command:

gcloud container clusters update cluster-name \
    --enable-autoprovisioning \
    --min-cpu minimum-cpu \
    --min-memory minimum-memory \
    --max-cpu maximum-cpu \
    --max-memory maximum-memory

where:

  • cluster-name is the name of the cluster to enable Node auto-provisioning.
  • minimum-cpu is the minimum number of cores in the cluster.
  • minimum-memory is the minimum number of gigabytes of memory in the cluster.
  • maximum-cpu is the maximum number of cores in the cluster.
  • maximum-memory is the maximum number of gigabytes of memory in the cluster.

The following example enables Node auto-provisioning on the dev-cluster and allows scaling between a total cluster size of 1 CPU and 1 gigabyte of memory to a maximum of 10 CPU and 64 gigabytes of memory:

gcloud container clusters update dev-cluster \
    --enable-autoprovisioning \
    --min-cpu 1 \
    --min-memory 1 \
    --max-cpu 10 \
    --max-memory 64

Console

To enable Node auto-provisioning, perform the following steps:

  1. Visit the Google Kubernetes Engine menu in Cloud Console.

Visit the Google Kubernetes Engine menu

  1. Select the desired cluster.
  2. Click the icon.
  3. Scroll down to Node auto-provisioning and select Enabled.
  4. Set your desired minimum and maximum CPU and memory usage for the cluster.
  5. Click Save. GKE updates your cluster.

Setting identity defaults for auto-provisioned node pools

Permissions for Google Cloud resources are provided by identities.

You can specify the default identity (either a service account or one or more scopes) used by new auto-provisioned node pools. Changing identity defaults does not affect any existing node pools.

To specify the default IAM service account used by node auto-provisioning, run the following command:

gcloud container clusters update cluster-name \
    --enable-autoprovisioning --autoprovisioning-service-account=service-account

where:

  • cluster-name is the name of the cluster to enable Node auto-provisioning.
  • service-account is the email address for the default service account.

The following example sets test-service-account@google.com as the default service account on the dev-cluster cluster:

gcloud container clusters update dev-cluster \
    --enable-autoprovisioning --autoprovisioning-service-account=test-service-account@google.com

To specify the default scopes used by Node auto-provisioning, run the following command:

gcloud container clusters update cluster-name \
    --enable-autoprovisioning --autoprovisioning-scopes=scope

where:

  • cluster-name is the name of the cluster to enable Node auto-provisioning.
  • scope is the Google Cloud scopes used by auto-provisioned node pools. To specify multiple scopes, separate the scopes by a comma (for example, scope, scope,...).

The following example sets the default scope on the dev-cluster cluster to devstorage.read_only:

gcloud container clusters update dev-cluster \
    --enable-autoprovisioning \
    --autoprovisioning-scopes=https://www.googleapis.com/auth/pubsub,https://www.googleapis.com/auth/devstorage.read_only

Configuring GPU limits

When using Node auto-provisioning with GPUs, you can set the maximum limit for each GPU type in the cluster by using the gcloud tool or the Google Cloud Console. To configure multiple types of GPU, you must use a configuration file.

To list the available resourceTypes, run gcloud compute accelerator-types list.

gcloud

gcloud container clusters update cluster-name \
    --enable-autoprovisioning \
    --max-cpu maximum-cpu \
    --max-memory maximum-memory \
    --min-accelerator type=gpu-type,count=minimum-accelerator \
    --max-accelerator type=gpu-type,count=maximum-accelerator

where:

  • cluster-name is the name of the cluster to enable Node auto-provisioning.
  • maximum-cpu is the maximum number of cores in the cluster.
  • maximum-memory is the maximum number of gigabytes of memory in the cluster.
  • gpu-type is the type of GPU accelerator.
  • minimum-accelerator is the minimum number of gpu-type GPU accelerators in the cluster.
  • maximum-accelerator is the maximum number of gpu-type GPU accelerators in the cluster.

The following example sets the GPU limits for the nvidia-tesla-k80 GPU accelerator type in the dev-cluster cluster:

gcloud container clusters update dev-cluster \
    --enable-autoprovisioning \
    --max-cpu 10 \
    --max-memory 64 \
    --min-accelerator type=nvidia-tesla-k80,count=1 \
    --max-accelerator type=nvidia-tesla-k80,count=4

File

You can load limits for multiple types of GPU by using a configuration file. The following YAML configuration configures two different types of GPUs:

  resourceLimits:
    -resourceType: 'cpu'
     minimum: 4
     maximum: 10
    -resourceType: 'memory'
     maximum: 64
    -resourceType: 'nvidia-tesla-k80'
     maximum: 4
    -resourceType: 'nvidia-tesla-v100'
     maximum: 2

To use an auto-provisioning configuration file:

  1. Copy the configuration above to a file in a location where gcloud can access it. Edit the values for cpu and memory. Add as many values for resourceType as you need. Save the file.

  2. Use gcloud to apply the configuration to your cluster:

gcloud container clusters update cluster-name \
    --enable-autoprovisioning \
    --autoprovisioning-config-file file-name

Where:

  • cluster-name is the name of the cluster to enable Node auto-provisioning.
  • file-name is the name of the file containing the resource limits.

For more information, see the gcloud container clusters update documentation.

Console

To enable Node auto-provisioning with GPU resources, perform the following steps:

  1. Visit the Google Kubernetes Engine menu in Cloud Console.

Visit the Google Kubernetes Engine menu

  1. Select the desired cluster.
  2. Click the Edit icon.
  3. Scroll down to Node auto-provisioning and select Enabled.
  4. Set your desired minimum and maximum CPU and memory usage for the cluster.
  5. Click Add resource.
  6. Select the type of GPU (for example, NVIDIA TESLA K80) you wish to add. Set your desired minimum and maximum number of GPUs to add to the cluster.
  7. Accept the limitations of GPUs in GKE.
  8. Click Save. GKE updates your cluster.

Node auto-provisioning locations

You set the zones where Node auto-provisioning can create new node pools. Regional locations are not supported. Zones must belong to the same region as the cluster but are not limited to node locations defined on the cluster level. Changing Node auto-provisioning locations doesn't affect any existing node pools.

To set locations where Node auto-provisioning can create new node pools, run the following command:

gcloud container clusters update cluster-name \
    --enable-autoprovisioning --autoprovisioning-locations=zone

where:

  • cluster-name is the name of cluster to enable Node auto-provisioning.
  • zone is the zone where Node auto-provisioning can create new node pools. To specify multiple zones, separate the zones by a comma (for example, zone, zone,...).

Support for node autorepair and autoupgrade

Node auto-provisioning supports creating node pools with node autorepair and node autoupgrade enabled.

gcloud

To enable autorepair and autoupgrade for all new autoprovisioned node pools, run the following command:

gcloud container clusters update cluster-name \
    --enable-autoprovisioning --enable-autoprovisioning-autorepair \
    --enable-autoprovisioning-autoupgrade

where:

  • cluster-name is the name of the cluster to enable Node auto-provisioning.

To disable autorepair and autorepair for all new autoprovisioned node pools, run the following command:

gcloud container clusters update cluster-name \
    --enable-autoprovisioning --no-enable-autoprovisioning-autorepair \
    --no-enable-autoprovisioning-autoupgrade

where:

  • cluster-name is the name of the cluster to enable Node auto-provisioning.

File

You can enable node autorepair and autoupgrade by using a configuration file. The following YAML configuration enables autorepair and disables autoupgrade:

  management:
    autoRepair: true
    autoUpgrade: false

To use an auto-provisioning configuration file:

  1. Copy the configuration above to a file in a location where gcloud can access it. Edit the values for autoUpgrade and autoRepair. Save the file.

  2. Use gcloud to apply the configuration to your cluster:

gcloud container clusters update cluster-name \
    --enable-autoprovisioning \
    --autoprovisioning-config-file file-name

Where:

  • cluster-name is the name of the cluster to enable Node auto-provisioning.
  • file-name is the name of the file containing the resource limits.

For more information, see the gcloud container clusters update documentation.

Specifying for node surge upgrades settings

Node auto-provisioning allows creating node pools with specified surge upgrade settings.

gcloud

To specify surge upgrade settings for all new autoprovisioned node pools, run the following command:

gcloud container clusters update cluster-name \
    --autoprovisioning-max-surge-upgrade max-surge-upgrade \
    --autoprovisioning-max-unavailable-upgrade max-unavailable-upgrade

where:

  • cluster-name is the name of the cluster to enable Node auto-provisioning.
  • max-surge-upgrade is the value of max-surge-upgrade set on autoprovisioned node pools.
  • max-unavailable-upgrade is the value of max-unavailable-upgrade set on autoprovisioned node pools.

File

You can specify surge upgrade settings for all new autoprovisioned node pools by using a configuration file. The following YAML configuration configures max-surge-upgrade of 1 and max-unavailable-upgrade of 2:

  upgradeSettings:
    maxSurgeUpgrade: 1
    maxUnavailableUpgrade: 2

To use an auto-provisioning configuration file:

  1. Copy the configuration above to a file in a location where gcloud can access it. Edit the values for maxSurgeUpgrade and maxUnavailableUpgrade. Add as many values for resourceType as you need. Save the file.

  2. Use gcloud to apply the configuration to your cluster:

gcloud container clusters update cluster-name \
    --enable-autoprovisioning \
    --autoprovisioning-config-file file-name

Where:

  • cluster-name is the name of the cluster to enable Node auto-provisioning.
  • file-name is the name of the file containing the resource limits.

For more information, see the gcloud container clusters update documentation.

Disabling Node auto-provisioning

When you disable Node auto-provisioning for a cluster, node pools are no longer auto-provisioned.

gcloud

To disable Node auto-provisioning for a cluster, run the following command:

gcloud container clusters update cluster-name --no-enable-autoprovisioning

Console

To disable Node auto-provisioning using the Google Cloud Console:

  1. Visit the Google Kubernetes Engine menu in Cloud Console.

Visit the Google Kubernetes Engine menu

  1. Select the desired cluster.
  2. Click the Edit icon.
  3. Scroll down to Node auto-provisioning and select Disabled.

Marking node pool as auto-provisioned

After enabling Node auto-provisioning on the cluster, you can specify which node pools are auto-provisioned. An auto-provisioned node pool is automatically deleted when no workloads are using it.

To mark a node pool as auto-provisioned, run the following command:

gcloud container node-pools update node-pool-name --enable-autoprovisioning

where node-pool-name is the name of the node pool.

Marking node pool as not auto-provisioned

To mark a node pool as not auto-provisioned, run the following command:

gcloud container node-pools update node-pool-name --no-enable-autoprovisioning

where node-pool-name is the name of the node pool.

What's next