Using node auto-provisioning

This page explains how to use Google Kubernetes Engine (GKE)'s node auto-provisioning feature.

Overview

Node auto-provisioning automatically manages a set of node pools on the user's behalf. Without node auto-provisioning, GKE considers starting new nodes only from the set of user created node pools. With node auto-provisioning, new node pools can be created and deleted automatically.

Before you begin

To prepare for this task, perform the following steps:

  • Ensure that you have enabled the Google Kubernetes Engine API.
  • Enable Google Kubernetes Engine API
  • Ensure that you have installed the Cloud SDK.
  • Set your default project ID:
    gcloud config set project [PROJECT_ID]
  • If you are working with zonal clusters, set your default compute zone:
    gcloud config set compute/zone [COMPUTE_ZONE]
  • If you are working with regional clusters, set your default compute region:
    gcloud config set compute/region [COMPUTE_REGION]
  • Update gcloud to the latest version:
    gcloud components update

Requirements

Node auto-provisioning is available in GKE Release:

  • v1.11.2-gke.25 and higher for zonal clusters
  • v1.12.x and higher for regional clusters

Operation

Node auto-provisioning is a mechanism of the cluster autoscaler, which scales on a per-node pool basis. With node auto-provisioning enabled, the cluster autoscaler can extend node pools automatically based on the specifications of unschedulable Pods.

Node auto-provisioning creates node pools based on the following information:

Resource limits

Node auto-provisioning and the cluster autoscaler have limits at two levels:

  • Node pool level
  • Cluster level

Limits for node pools

Node pools created by NAP are limited to 1000 nodes.

Limits for Clusters

The limits you define are enforced based on the total CPU and memory resources used across your cluster, not just auto-provisioned pools.

Cluster autoscaler does not create new nodes if doing so would exceed one of the defined limits. If limits are already exceeded, nodes are not automatically deleted.

Workload separation

If pending Pods have node affinities and tolerations, node auto-provisioning can provision nodes with matching labels and taints.

Node auto-provisioning might create node pools with labels and taints if all of the following conditions are met:

  • A pending Pod requires a node with a specific label key and value.
  • The Pod has a toleration for a taint with the same key.
  • The toleration is for the NoSchedule effect, NoExecute effect, or all effects.

The Pod's specification can express that it requires nodes with specific labels in two ways:

  • Using a nodeSelector field.
  • Using a nodeAffinity field with an In operator and exactly one value.

The following example is an excerpt of a Pod specification that is interpreted as a workload separation request. In this example, the cluster administrator has chosen dedicated as the key that will be used for workload isolation, and the UI team has determined that they need dedicated nodes for their workloads.

The Pod has a toleration for nodes labeled with dedicated=ui-team and uses nodeAffinity for node selection:

spec:
  tolerations:
  - key: dedicated
    operator: Equal
    value: ui-team
    effect: NoSchedule
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: dedicated
            operator: In
            values:
            - ui-team

If this Pod exists, nodes with taint dedicated=ui-team:NoSchedule and label dedicated=ui-team are considered for creation by node auto-provisioning.

The example below uses nodeSelector and has the same effect:

spec:
  tolerations:
  - key: dedicated
    operator: Equal
    value: ui-team
    effect: NoSchedule
  nodeSelector:
    dedicated: ui-team

Deletion of auto-provisioned node pools

When there are no nodes in an auto-provisioned node pool, GKE deletes the node pool. Node pools that are not marked as auto-provisioned are not deleted.

Supported machine types

Currently, node auto-provisioning will only consider creating node pools with machines with up to 64 vCPUs. This limitation will be lifted in the future.

Scalability limitations

Node auto-provisioning has the same limitations as the cluster autoscaler, as well as additional limitations described in the following sections.

Limit on number of separated workloads
Node auto-provisioning supports a maximum of 100 distinct separated workloads.
Limit on number of node pools
Node auto-provisioning de-prioritizes creating new node pools when the number of pools approaches 100. Creating over 100 node pools is possible but taken when creating a node pool is the only option to schedule a pending pod.

Enabling node auto-provisioning

You enable node auto-provisioning on a cluster with gcloud or the Google Cloud Console.

gcloud

To enable node auto-provisioning, run the following command:

gcloud container clusters update [CLUSTER_NAME] \
  --enable-autoprovisioning \
  --min-cpu [MINIMUM_CPU] \
  --min-memory [MINIMUM_MEMORY] \
  --max-cpu [MAXIMUM_CPU] \
  --max-memory [MAXIMUM_MEMORY]

In this command:

  • --enable-autoprovisioning indicates that node auto-provisioning is enabled.
  • --min-cpu specifies the minimum number of cores in the cluster.
  • --min-memory specifies the minimum number of gigabytes of memory in the cluster.
  • --max-cpu specifies the maximum number of cores in the cluster.
  • --max-memory specifies the maximum number of gigabytes of memory in the cluster.

For example, to enable node auto-provisioning on a cluster and allowing scaling between total cluster size of 1 CPU and 1 gigabyte of memory to a maximum of 10 CPU and 64 gigabytes of memory, run:

gcloud container clusters update [CLUSTER_NAME] \
  --enable-autoprovisioning \
  --min-cpu 1 \
  --min-memory 1 \
  --max-cpu 10 \
  --max-memory 64

Console

To enable node auto-provisioning, perform the following steps:

  1. Visit the Google Kubernetes Engine menu in Cloud Console.

Visit the Google Kubernetes Engine menu

  1. Select the desired cluster.
  2. Click the edit icon.
  3. Scroll down to Node auto-provisioning and select Enabled.
  4. Set your desired minimum and maximum CPU and memory usage for the cluster.
  5. Click Save. GKE updates your cluster.

Setting identity defaults for auto-provisioned node pools

Permissions for Google Cloud resources are provided by identities.

You can specify the default identity (either a service account or one or more scopes) used by new auto-provisioned node pools. Changing identity defaults does not affect any existing node pools.

To specify the default Cloud IAM service account used by node auto-provisioning run the following command:

gcloud container clusters update \
[CLUSTER_NAME] --enable-autoprovisioning --autoprovisioning-service-account=SERVICE_ACCOUNT

For example, to set test-service-account@google.com as the default service account on the cluster dev-cluster:

gcloud container clusters update dev-cluster \
--enable-autoprovisioning --autoprovisioning-service-account=test-service-account@google.com

To specify the default scopes used by node auto-provisioning run the following gcloud command:

gcloud container clusters update \
[CLUSTER_NAME] --enable-autoprovisioning --autoprovisioning-scopes=SCOPE,[SCOPE,...]

For example, to set the default scope on the cluster dev-cluster as devstorage.read_only:

gcloud container clusters update dev-cluster \
--enable-autoprovisioning --autoprovisioning-scopes=https://www.googleapis.com/auth/pubsub,https://www.googleapis.com/auth/devstorage.read_only

Where:

  • --enable-autoprovisioning enables node auto-provisioning.
  • --autoprovisioning-service-account specifies the Google Cloud service account used by auto-provisioned node pools.
  • --autoprovisioning-scopes specifies the Google Cloud scopes used by auto-provisioned node pools.

Configuring GPU limits

When using node auto-provisioning with GPUs, set the maximum limit for each GPU type in the cluster using the --max-accelerator flag or from the Google Cloud Console.

A list of available resourceTypes is in the output of the command gcloud compute accelerator-types list.

To configure multiple types of GPU, you must use a configuration file.

gcloud

gcloud container clusters update [CLUSTER_NAME] \
  --enable-autoprovisioning \
  --max-cpu 10 \
  --max-memory 64 \
  --min-accelerator type=nvidia-tesla-k80,count=1
  --max-accelerator type=nvidia-tesla-k80,count=4

In this command:

  • --enable-autoprovisioning indicates that node auto-provisioning is enabled.
  • --max-cpu specifies the maximum number of cores in the cluster.
  • --max-memory specifies the maximum number of gigabytes of memory in the cluster.
  • --min-accelerator specifies the minimum number of nvidia-tesla-k80 GPU accelerators.
  • --max-accelerator specifies the maximum number of nvidia-tesla-k80 GPU accelerators.

File

You can load limits from a configuration file. The following YAML configuration configures two different types of GPU:

  resourceLimits:
    -resourceType: 'cpu'
     minimum: 4
     maximum: 10
    -resourceType: 'memory'
     maximum: 64
    -resourceType: 'nvidia-tesla-k80'
     maximum: 4
    -resourceType: 'nvidia-tesla-v100'
     maximum: 2

To use an auto-provisioning configuration file:

  1. Copy the configuration above to a file in a location where gcloud can access it. Edit the values for cpu and memory. Add as many values for resourceType as you need. Save the file.

  2. Use gcloud to apply the configuration to your cluster:

gcloud container clusters update [CLUSTER_NAME] \
  --enable-autoprovisioning \
  --autoprovisioning-config-file [FILE_NAME]

Where:

  • --enable-autoprovisioning indicates that node auto-provisioning is enabled.
  • --autoprovisioning-config-file specifies the file with resource limits.

For more information, see the gcloud container clusters update documentation.

Console

To enable node auto-provisioning with GPU resources, perform the following steps:

  1. Visit the Google Kubernetes Engine menu in Cloud Console.

Visit the Google Kubernetes Engine menu

  1. Select the desired cluster.
  2. Click the edit Edit icon.
  3. Scroll down to Node auto-provisioning and select Enabled.
  4. Set your desired minimum and maximum CPU and memory usage for the cluster.
  5. Click add Add resource.
  6. Select the type of GPU (for example, NVIDIA TESLA K80) you wish to add. Set your desired minimum and maximum number of GPUs to add to the cluster.
  7. Accept the limitations of GPUs in GKE.
  8. Click Save. GKE updates your cluster.

Node Auto-provisioning locations

You set the zones where node auto-provisioning can create new node pools. Regional locations are not supported. Zones all have to belong to the same region as the cluster but are not limited to node locations defined on the cluster level. Changing node auto-provisioning locations doesn't affect any existing node pools.

gcloud

To set locations where node auto-provisioning can create new node pools run following gcloud command:

gcloud container clusters update [CUSTER_NAME] \
  --enable-autoprovisioning --autoprovisioning-locations=ZONE,[ZONE,...]

In this command:

  • --enable-autoprovisioning indicates that node auto-provisioning is enabled.
  • --autoprovisioning-locations specifies locations where node auto-provisioning can create new node pools.

Disabling node auto-provisioning

When you disable node auto-provisioning for a cluster, node pools are no longer auto-provisioned.

gcloud

To disable node auto-provisioning, update the cluster with the --no-enable-autoprovisioning flag:

gcloud container clusters update [CLUSTER_NAME] --no-enable-autoprovisioning

In this command:

  • --no-enable-autoprovisioning indicates that node auto-provisioning is disabled.

Console

To disable node auto-provisioning, use the Google Cloud Console.

  1. Visit the Google Kubernetes Engine menu in Cloud Console.

Visit the Google Kubernetes Engine menu

  1. Select the desired cluster.
  2. Click the edit Edit icon.
  3. Scroll down to Node auto-provisioning and select Disabled.

Marking node pool as auto-provisioned

After enabling node auto-provisioning on the cluster, you can specify which node pools are auto-provisioned. An auto-provisioned node pool is automatically deleted when no workloads are using it.

gcloud

To mark node pool as auto-provisioned, run the following command:

gcloud container node-pools update [NODE_POOL_NAME] --enable-autoprovisioning

In this command:

  • --enable-autoprovisioning indicates that node pool is marked as auto-provisioned.

Marking node pool as not auto-provisioned

You can remove auto-provisioning from a node pool by using the --no-enable-autoprovisioning flag.

gcloud

To mark node pool as not auto-provisioned run following gcloud command:

gcloud container node-pools update [NODE_POOL_NAME] --no-enable-autoprovisioning

In this command:

  • --no-enable-autoprovisioning indicates that node pool is marked as not auto-provisioned.

What's next

หน้านี้มีประโยชน์ไหม โปรดแสดงความคิดเห็น

ส่งความคิดเห็นเกี่ยวกับ...