Using node auto-provisioning

This page explains how to use Google Kubernetes Engine's node auto-provisioning feature.

Overview

Node auto-provisioning automatically manages a set of node pools on the user's behalf. Without node auto-provisioning, GKE considers starting new nodes only from the set of user chosen node pools. With node auto-provisioning, new node pools can be created and deleted automatically.

Before you begin

To prepare for this task, perform the following steps:

  • Ensure that you have enabled the Google Kubernetes Engine API.
  • Enable Google Kubernetes Engine API
  • Ensure that you have installed the Cloud SDK.
  • Set your default project ID:
    gcloud config set project [PROJECT_ID]
  • If you are working with zonal clusters, set your default compute zone:
    gcloud config set compute/zone [COMPUTE_ZONE]
  • If you are working with regional clusters, set your default compute region:
    gcloud config set compute/region [COMPUTE_REGION]
  • Update gcloud to the latest version:
    gcloud components update

Requirements

Node auto-provisioning is available in GKE Release:

  • v1.11.2-gke.25 and higher for zonal clusters
  • v1.12.x and higher for regional clusters

Operation

Node auto-provisioning is a mechanism of the cluster autoscaler, which scales on a per-node pool basis. With node auto-provisioning enabled, the cluster autoscaler can extend node pools automatically based on the specifications of unschedulable Pods.

Node auto-provisioning creates node pools based on the following information:

Resource limits

Node auto-provisioning and the cluster autoscaler have limits at two levels:

  • Node pool level
  • Cluster level

Limits for node pools

Node pools created by NAP are limited to 1000 nodes.

Limits for Clusters

The limits you define are enforced based on the amount of CPU and memory used across your cluster, not just auto-provisioned pools.

Cluster autoscaler does not create new nodes if doing so would exceed one of the defined limits. If limits are already exceeded, no nodes are automatically deleted.

Workload separation

If there are pending Pods with node affinities and tolerations, node auto-provisioning can provision nodes with matching labels and taints.

Currently, node auto-provisioning considers creating node pools of nodes with labels and taints if all of the following conditions are met:

  • A pending Pod requires a node labeled with specific key and value.
  • The Pod has a toleration for a taint with the same key.
  • The toleration is for the NoSchedule effect, NoExecute effect, or all effects.

The Pod's specification can express that it requires nodes with specific labels in two ways:

  • Using a nodeSelector field.
  • Using a nodeAffinity field with an In operator and exactly one value.

The following example is an excerpt of a Pod specification that is interpreted as a workload separation request. In this example, the cluster administrator has chosen dedicated as the key that will be used for workload isolation, and the UI team has determined that they need dedicated nodes for their workloads.

The Pod has a toleration for nodes labeled with dedicated=ui-team and uses nodeAffinity for node selection:

spec:
  tolerations:
  - key: dedicated
    operator: Equal
    value: ui-team
    effect: NoSchedule
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: dedicated
            operator: In
            values:
            - ui-team

If this Pod exists, nodes with taint dedicated=ui-team:NoSchedule and label dedicated=ui-team are considered for creation by node auto-provisioning.

The example below uses nodeSelector and has the same effect:

spec:
  tolerations:
  - key: dedicated
    operator: Equal
    value: ui-team
    effect: NoSchedule
  nodeSelector:
    dedicated: ui-team

Deletion of auto-provisioned node pools

When there are no nodes in an auto-provisioned node pool, GKE deletes the node pool. Node pools that are not marked as auto-provisioned are not deleted.

Supported machine types

Currently, node auto-provisioning will only consider creating node pools with machines with up to 64 vCPUs. This limitation will be lifted in the future.

Scalability limitations

Node auto-provisioning has the same limitations as the cluster autoscaler, as well as additional limitations described in the following sections.

Limit on number of separated workloads
Node auto-provisioning supports a maximum of 100 distinct separated workloads.
Limit on number of node pools
Node auto-provisioning deprioritizes creating new node pools when the number of pools approaches 100. Creating over 100 node pools is possible but taken when creating a node pool is the only option to schedule a pending pod.

Enabling node auto-provisioning

gcloud

To enable node auto-provisioning, run the following command:

gcloud container clusters update [CLUSTER_NAME] \
  --enable-autoprovisioning \
  --max-cpu 10 \
  --max-memory 64

In this command:

  • --enable-autoprovisioning indicates that node auto-provisioning is enabled.
  • --max-cpu specifies the maximum number of cores in the cluster.
  • --max-memory specifies the maximum number of gigabytes of memory in the cluster.

Setting identity defaults for auto-provisioned node pools

You can specify the default Cloud Identity and Access Management used by auto-provisioned node pools. You can specify either a service account or one or more scopes for node auto-provisioning to use. Changing identity defaults does not affect any existing node pools.

To specify the default Cloud IAM service account used by node auto-provisioning run the following gcloud command:

gcloud container clusters update \
[CLUSTER_NAME] --enable-autoprovisioning --autoprovisioning-service-account=SERVICE_ACCOUNT

For example, to set test-service-account@google.com as your auto-provisioning default service account:

gcloud container clusters update dev-cluster \
--enable-autoprovisioning --autoprovisioning-service-account=test-service-account@google.com

To specify the default scopes to be used by node auto-provisioning run the following gcloud command:

gcloud container clusters update \
[CLUSTER_NAME] --enable-autoprovisioning --autoprovisioning-scopes=SCOPE,[SCOPE,...]

Example command for specifying auto-provisioning default scopes:

gcloud container clusters update dev-cluster \
--enable-autoprovisioning --autoprovisioning-scopes=https://www.googleapis.com/auth/pubsub,https://www.googleapis.com/auth/devstorage.read_only

In the above commands:

  • --enable-autoprovisioning indicates that node auto-provisioning is enabled.
  • --autoprovisioning-service-account specifies the GCP service account used by autoprovisioned node pools.
  • --autoprovisioning-scopes specifies the GCP scopes used by autoprovisioned node pools.

Configuring GPU limits

gcloud

When using node auto-provisioning with GPUs, it's recommended to set the limit for each GPU type in the cluster, using the --max-accelerator flag.

To configure multiple types of GPU, you must use a configuration file instead of the flag. A configuration file example follows this example.

gcloud container clusters update [CLUSTER_NAME] \
  --enable-autoprovisioning \
  --max-cpu 10 \
  --max-memory 64 \
  --max-accelerator type=nvidia-tesla-k80,count=4

In this command:

  • --enable-autoprovisioning indicates that node auto-provisioning is enabled.
  • --max-cpu specifies the maximum number of cores in the cluster.
  • --max-memory specifies the maximum number of gigabytes of memory in the cluster.
  • --max-accelerator specify maximum number of nvidia-tesla-k80 GPU accelerators.

You can load limits from a configuration file. The following configuration file configures two different types of GPU:

  resourceLimits:
    -resourceType: 'cpu'
     minimum: 4
     maximum: 10
    -resourceType: 'memory'
     maximum: 64
    -resourceType: 'nvidia-tesla-k80'
     maximum: 4
    -resourceType: 'nvidia-tesla-v100'
     maximum: 2

The following command shows how to apply a configuration file saved locally:

gcloud container clusters update [CLUSTER_NAME] \
  --enable-autoprovisioning \
  --autoprovisioning-config-file [FILE_NAME]

In this command:

  • --enable-autoprovisioning indicates that node auto-provisioning is enabled.
  • --autoprovisioning-config-file specifies the file with resource limits.

Node Auto-provisioning locations

You set the zones where node auto-provisioning can create new node pools. Regional locations are not supported. Zones all have to belong to the same region as the cluster but are not limited to node locations defined on the cluster level. Changing node auto-provisioning locations doesn't affect any existing node pools.

gcloud

To set locations where node auto-provisioning can create new node pools run following gcloud command:

gcloud container clusters update [CUSTER_NAME] \
  --enable-autoprovisioning --autoprovisioning-locations=ZONE,[ZONE,...]

In this command:

  • --enable-autoprovisioning indicates that node auto-provisioning is enabled.
  • --autoprovisioning-locations specifies locations where node auto-provisioning can create new node pools.

Disabling node auto-provisioning

When you disable node auto-provisioning for a cluster, node pools are no longer auto-provisioned.

gcloud

To disable node auto-provisioning, update the cluster with the --no-enable-autoprovisioning flag:

gcloud container clusters update [CLUSTER_NAME] --no-enable-autoprovisioning

In this command:

  • --no-enable-autoprovisioning indicates that node auto-provisioning is disabled.

Marking node pool as auto-provisioned

After enabling node auto-provisioning on the cluster, you can specify which node pools are auto-provisioned. An auto-provisioned node pool is automatically deleted when no workloads are using it.

gcloud

To mark node pool as auto-provisioned, run the following command:

gcloud container node-pools update [NODE_POOL_NAME] --enable-autoprovisioning

In this command:

  • --enable-autoprovisioning indicates that node pool is marked as auto-provisioned.

Marking node pool as not auto-provisioned

You can remove auto-provisioning from a node pool by using the --no-enable-autoprovisioning flag.

gcloud

To mark node pool as not auto-provisioned run following gcloud command:

gcloud container node-pools update [NODE_POOL_NAME] --no-enable-autoprovisioning

In this command:

  • --no-enable-autoprovisioning indicates that node pool is marked as not auto-provisioned.

What's next

Apakah halaman ini membantu? Beri tahu kami pendapat Anda:

Kirim masukan tentang...

Kubernetes Engine Documentation