Using node auto-provisioning

は、

ベータ版 です。

これは、 のベータ版リリースです。 は下位互換性のない方法で変更される可能性があり、と は、SLA または非推奨ポリシーの対象ではありません。

This page explains how to use Google Kubernetes Engine's node auto-provisioning feature.

Overview

Node auto-provisioning is a mechanism of cluster autoscaler which automatically manages a list of node pools on the user's behalf. Without node auto-provisioning, the cluster autoscaler considers new nodes only from the set of user chosen node pools. With node auto-provisioning, new node pools can be created and deleted automatically.

Before you begin

To prepare for this task, perform the following steps:

• Ensure that you have enabled the Google Kubernetes Engine API.
Enable Google Kubernetes Engine API
• Ensure that you have installed the Cloud SDK.
• Set your default project ID:

  gcloud config set project [PROJECT_ID]

• If you are working with zonal clusters, set your default compute zone:

  gcloud config set compute/zone [COMPUTE_ZONE]

• If you are working with regional clusters, set your default compute region:

  gcloud config set compute/region [COMPUTE_REGION]

• Update gcloud to the latest version:

  gcloud components update

Note: You can override these default settings in gcloud commands using the --project, --zone, and --region operational flags.

Requirements

Node auto-provisioning is available in GKE Release:

• v1.11.2-gke.25 and higher for zonal clusters
• v1.12.x and higher for regional clusters

Operation

Cluster autoscaler works on per-node pool basis. With node auto-provisioning, the set of autoscaled node pools can be extended automatically based on the specifications of unschedulable Pods.

Node auto-provisioning creates node pools based on the following information:

• CPU and memory resource requests.
• GPU requests
• Pending Pods' node affinitities and label selectors.
• Pending Pods' node taints and tolerations.

Note: For node auto-provisioning to work as expected, Pod resource requests need to be set up correctly. If resource requests are set to values smaller than what Pods actually require for operation, auto-provisioned nodes might be too small, which can cause Pods to fail due to lack of resources on the provisioned node.

Resource limits

Node auto-provisioning requires upper limits for CPU and memory. Setting limits for GPUs is optional. Setting lower limits on resources is also optional. Limits are checked against all the nodes in the cluster. By default, auto-provisioned node pools are limited to 1000 nodes.

Cluster autoscaler does not create new nodes if doing so would exceed one of the defined limits. However, if limits are already exceeded, no nodes are automatically deleted.

Note: The limits you define are enforced for all autoscaled node pools, not just auto-provisioned pools.

Workload separation

If there are pending Pods with node affinities and tolerations, indicating workload separation, node auto-provisioner can provision nodes with matching labels and taints.

Currently, node auto-provisioner considers creating node pools of nodes with labels and taints if all of the following conditions are met:

• A pending Pod requires a node labeled with specific key and value.
• The Pod has a toleration for a taint with the same key.
• The toleration is for the NoSchedule effect, NoExecute effect, or all effects.

The Pod's specification can express that it requires nodes with specific labels in two ways:

• Using a nodeSelector field.
• Using a nodeAffinity field with an In operator and exactly one value.

The following example is an excerpt of a Pod specification that is interpreted as a workload separation request. The Pod has a toleration for nodes with labels with a key of foo and a value of bar. This example uses nodeAffinity for node selection:

spec:
  tolerations:
  - key: foo
    operator: Equal
    value: bar
    effect: NoSchedule
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: foo
            operator: In
            values:
            - bar

If this Pod exists, nodes with taint key=value:NoSchedule and label key=value are considered for creation.

The example below uses nodeSelector and has the same effect:

spec:
  tolerations:
  - key: foo
    operator: Equal
    value: bar
    effect: NoSchedule
  nodeSelector:
    foo: bar

Deletion of auto-provisioned node pools

When there are no nodes in an auto-provisioned node pool, GKE deletes the node pool. Node pools that are not marked as auto-provisioned are not deleted.

Supported machine types

Currently, node auto-provisioning will only consider creating node pools with machines with up to 16 vCPUs. This limitation will be lifted in the future.

Scalability limitations

We currently test node auto-provisioning in clusters with up to 100 nodes. We do not recommend using the feature on larger clusters. Apart from that, cluster autoscaler scalability limits apply.

Limit on number of auto-provisioned node pools

There can be no more than 15 auto-provisioned node pools in a cluster. If the limit is reached, no new auto-provisioned node pools are created.

Enabling node auto-provisioning

gcloud

To enable node auto-provisioning, run the following command:

gcloud beta container clusters update [CLUSTER_NAME] --enable-autoprovisioning \
        --max-cpu 10 --max-memory 64

In this command:

• --enable-autoprovisioning indicates that node auto-provisioning is enabled.
• --max-cpu specifies the maximum number of cores in the cluster.
• --max-memory specifies the maximum number of gigabytes of memory in the cluster.

Setting identity defaults for auto-provisioned node pools

You can specify the default Cloud Identity and Access Management (service accounts and scopes) used by auto-provisioned node pools. You can specify either scopes or service account for node auto-provisioning to use. Changing identity defaults does not affect any existing node pools.

To specify the default service account to be used by node auto-provisioning run the following gcloud command:

gcloud beta container clusters update \
[CLUSTER_NAME] --enable-autoprovisioning --autoprovisioning-service-account=SERVICE_ACCOUNT

Example command for specifying auto-provisioning default service account:

gcloud beta container clusters update dev-cluster \
--enable-autoprovisioning --autoprovisioning-service-account=[e-mail]

To specify the default scopes to be used by node auto-provisioning run the following gcloud command:

gcloud beta container clusters update \
[CLUSTER_NAME] --enable-autoprovisioning --autoprovisioning-scopes=SCOPE,[SCOPE,...]

Example command for specifying auto-provisioning default scopes:

gcloud beta container clusters update dev-cluster \
--enable-autoprovisioning --autoprovisioning-scopes=https://www.googleapis.com/auth/pubsub,https://www.googleapis.com/auth/devstorage.read_only

In the above commands:

• --enable-autoprovisioning indicates that node auto-provisioning is enabled.
• --autoprovisioning-service-account specifies the GCP service account used by autoprovisioned node pools.
• --autoprovisioning-scopes specifies the GCP scopes used by autoprovisioned node pools.

Setting up GPU limits

gcloud

When using node auto-provisioning with GPUs, it's recommended to set the limit for total number of each GPU type in the cluster. To do so, run the following command:

Note: CPU and memory limits must also be passed in the command. You can only set a limit for one type of GPU this way. To set limits for multiple GPU types in the same cluster, use a configuration file instead.

gcloud beta container clusters update [CLUSTER_NAME] --enable-autoprovisioning \
        --max-cpu 10 --max-memory 64 --max-accelerator type=nvidia-tesla-k80,count=4

In this command:

• --enable-autoprovisioning indicates that node auto-provisioning is enabled.
• --max-cpu specifies the maximum number of cores in the cluster.
• --max-memory specifies the maximum number of gigabytes of memory in the cluster.
• --max-accelerator specify maximum number of nvidia-tesla-k80 GPU accelerators.

Alternatively, you can load limits from a configuration file:

gcloud beta container clusters update [CLUSTER_NAME] --enable-autoprovisioning \
         --autoprovisioning-config-file [FILE_NAME]

In this command: * --enable-autoprovisioning indicates that node auto-provisioning is enabled. * --autoprovisioning-config-file specifies the file with resource limits. Example configuration:

  resourceLimits:
    -resourceType: 'cpu'
     minimum: 4
     maximum: 10
    -resourceType: 'memory'
     maximum: 64
    -resourceType: 'nvidia-tesla-k80'
     maximum: 4
    -resourceType: 'nvidia-tesla-v100'
     maximum: 2

Disabling node auto-provisioning

When you disable node auto-provisioning for a cluster, node pools are no longer auto-provisioned.

Note: All the node pools are marked as not auto-provisioned when you disable auto-provisioning. Reenabling auto-provisioning does not automatically mark node pools as auto-provisioned.

gcloud

To disable node auto-provisioning, run the following command:

gcloud beta container clusters update [CLUSTER_NAME] --no-enable-autoprovisioning

In this command:

• --no-enable-autoprovisioning indicates that node auto-provisioning is disabled.

Marking node pool as auto-provisioned

You can mark any node pool in the cluster as auto-provisioned. Doing so might cause the node pool to be automatically deleted when it is no longer in use.

Note: You must enable node auto-provisioning before you can mark a node as auto-provisioned.

gcloud

To mark node pool as auto-provisioned, run the following command:

gcloud beta container node-pools update [NODE_POOL_NAME] --enable-autoprovisioning

In this command:

• --enable-autoprovisioning indicates that node pool is marked as auto-provisioned.

Marking node pool as not auto-provisioned

You can mark a node pool as not auto-provisioned.

gcloud

To mark node pool as not auto-provisioned run following gcloud command:

gcloud beta container node-pools update [NODE_POOL_NAME] --no-enable-autoprovisioning

In this command:

• --no-enable-autoprovisioning indicates that node pool is marked as not auto-provisioned.

What's next

• Learn more about cluster autoscaler.