Use node auto-provisioning

Standard

---

This page explains how to use node auto-provisioning in Standard Google Kubernetes Engine (GKE) clusters. Before reading this page, ensure that you're familiar with node auto-provisioning.

Before you begin

Before you start, make sure you have performed the following tasks:

• Enable the Google Kubernetes Engine API.
Enable Google Kubernetes Engine API
• If you want to use the Google Cloud CLI for this task, install and then initialize the gcloud CLI. If you previously installed the gcloud CLI, get the latest version by running gcloud components update.
  Note: For existing gcloud CLI installations, make sure to set the compute/region and compute/zone properties. By setting default locations, you can avoid errors in gcloud CLI like the following: One of [--zone, --region] must be supplied: Please specify location.

Requirements

Node auto-provisioning is available in the following GKE releases:

• Version 1.27.6 and later or 1.28 and later, for Cloud TPU v4 and v5e.
• Version 1.28.7-gke.1020000 or later and 1.29.2-gke.1035000 or later for Cloud TPU v5p.
• Version 1.31.1-gke.1146000 or later for Cloud TPU v6e in Preview.

Enable node auto-provisioning

You can enable node auto-provisioning on a cluster with the gcloud CLI or the Google Cloud console.

Node auto-provisioning has the following resource limitations:

You must plan the node IP address range carefully. You can expand the node IP address range after you create a cluster. However, we recommend not to expand the node IP address range after you create the cluster as you must update the firewall rules to include the new range as a source. You can expand the Pod IP address range by using discontiguous multi-Pod CIDR with node auto-provisioning.

Note: If you disable then re-enable auto-provisioning on your cluster, existing node pools won't have auto-provisioning enabled. To re-enable auto-provisioning for these node pools, you need to mark individual node pools as auto-provisioned.

gcloud

To enable node auto-provisioning, run the following command:

gcloud container clusters update CLUSTER_NAME \
    --enable-autoprovisioning \
    --min-cpu MINIMUM_CPU \
    --min-memory MIMIMUM_MEMORY \
    --max-cpu MAXIMUM_CPU \
    --max-memory MAXIMUM_MEMORY \
    --autoprovisioning-scopes=https://www.googleapis.com/auth/logging.write,https://www.googleapis.com/auth/monitoring,https://www.googleapis.com/auth/devstorage.read_only

Replace the following:

• CLUSTER_NAME: the name of the cluster to enable node auto-provisioning.
• MINIMUM_CPU: the minimum number of cores in the cluster.
• MINIMUM_MEMORY: the minimum number of gigabytes of memory in the cluster.
• MAXIMUM_CPU: the maximum number of cores in the cluster.
• MAXIMUM_MEMORY: the maximum number of gigabytes of memory in the cluster.

The following example enables node auto-provisioning on the dev-cluster and allows scaling between a total cluster size of 1 CPU and 1 gigabyte of memory to a maximum of 10 CPU and 64 gigabytes of memory:

gcloud container clusters update dev-cluster \
    --enable-autoprovisioning \
    --min-cpu 1 \
    --min-memory 1 \
    --max-cpu 10 \
    --max-memory 64

Console

To enable node auto-provisioning, perform the following steps:

1. Go to the Google Kubernetes Engine page in Google Cloud console.

   Go to Google Kubernetes Engine

2. Click the name of the cluster.

3. In the Automation section, for Node auto-provisioning, click Edit.

4. Select the Enable node auto-provisioning checkbox.

5. Set your desired minimum and maximum CPU and memory usage for the cluster.

6. Click Save changes.

Using an auto-provisioning config file

Node auto-provisioning can be configured by using a YAML configuration file. The configuration file can contain just a single line if it's used to change a single setting. Multiple settings can be specified in a single config file. In this case, all those setting will be changed when the config file is applied.

Some advanced configurations can only be specified by using a configuration file.

Example 1: Applying the following configuration file enables node auto-repair and auto-upgrade for any new node pools created by node auto-provisioning:

management:
  autoRepair: true
  autoUpgrade: true

Example 2: Applying the following configuration file would change the following settings:

• Sets resource limits for CPU, memory and GPU. Node auto-provisioning will not create a node if the total size of the cluster exceeds the specified resource limits.
• Enables node auto-repair and auto-upgrade for any new node pools created by node auto-provisioning.
• Enables Secure boot and integrity monitoring for any new node pools created by node auto-provisioning.
• Sets boot disk size to 100 GB for any new node pools created by node auto-provisioning.

resourceLimits:
  - resourceType: 'cpu'
    minimum: 4
    maximum: 10
  - resourceType: 'memory'
    maximum: 64
  - resourceType: 'nvidia-tesla-t4'
    maximum: 4
management:
  autoRepair: true
  autoUpgrade: true
shieldedInstanceConfig:
  enableSecureBoot: true
  enableIntegrityMonitoring: true
diskSizeGb: 100

To use an auto-provisioning configuration file:

1. Create a file with the desired configuration in a location where the gcloud CLI can access it.

2. Apply the configuration to your cluster by running the following command:

   gcloud container clusters update CLUSTER_NAME \
      --enable-autoprovisioning \
      --autoprovisioning-config-file FILE_NAME

   Replace the following:

   • CLUSTER_NAME: the name of the cluster.
   • FILE_NAME: the name of the configuration file.

   For more information, see the gcloud container clusters update documentation.

Auto-provisioning defaults

Node auto-provisioning looks at Pod requirements in your cluster to determine what type of nodes would best fit those Pods. However, some node pool settings are not directly specified by Pods (for example settings related to node upgrades). You can set default values for those settings, which will be applied to all newly created node pools.

Note: Changing auto-provisioning defaults does not affect any existing node pools, including node pools previously created by node auto-provisioning.

Setting the default node image type

You can specify the node image type to use for all new auto-provisioned node pools using the gcloud CLI or a configuration file. This setting is only available for GKE cluster version 1.20.6-gke.1800 and later.

gcloud

To set the default node image type, run the following command:

gcloud container clusters update CLUSTER_NAME \
  --enable-autoprovisioning \
  --autoprovisioning-image-type IMAGE_TYPE

Replace the following:

• CLUSTER_NAME: the name of the cluster.

• IMAGE_TYPE: the node image type, which can be one of the following:

  • cos_containerd: Container-Optimized OS with containerd.
  • ubuntu_containerd: Ubuntu with containerd.

Warning: In GKE version 1.24 and later, Docker-based node image types are not supported. In GKE version 1.23, you also cannot create new node pools with Docker node image types. You must migrate to a containerd node image type. To learn more about this change, see About the Docker node image deprecation.

Note: For ubuntu and ubuntu_containerd image types, specify --autoprovisioning-scopes https://www.googleapis.com/auth/devstorage.read_only in the command.

File

For all new auto-provisioned node pools, you can specify the node image type to use by using a configuration file. The following YAML configuration specifies that for new auto-provisioned node pools, the image type is cos_containerd, and has associated resource limits for CPU and memory. You must specify maximum values for CPU and memory to enable auto-provisioning.

Note: For ubuntu and ubuntu_containerd image types, add scopes: https://www.googleapis.com/auth/devstorage.read_only to the configuration file.

Warning: In GKE version 1.24 and later, Docker-based node image types are not supported. In GKE version 1.23, you also cannot create new node pools with Docker node image types. You must migrate to a containerd node image type. To learn more about this change, see About the Docker node image deprecation.

1. Save the YAML configuration:

   resourceLimits:
     - resourceType: 'cpu'
       minimum: 4
       maximum: 10
     - resourceType: 'memory'
       maximum: 64
   imageType: 'cos_containerd'
   

2. Apply the configuration:

   gcloud container clusters update CLUSTER_NAME \
     --enable-autoprovisioning \
     --autoprovisioning-config-file FILE_NAME
   

   Replace the following:

   • CLUSTER_NAME: the name of the cluster.
   • FILE_NAME: the name of the configuration file.

Setting identity defaults for auto-provisioned node pools

Permissions for Google Cloud resources are provided by identities.

You can specify the default identity (either a service account or one or more scopes) for new auto-provisioned node pools using the gcloud CLI or through a configuration file.

Note: New node pools do not inherit identities from other node pools. If you do not specify a default identity, workloads running in auto-provisioned node pools cannot access any Google Cloud APIs. For example, if you pull container images from a private Artifact Registry repository, you must add the https://www.googleapis.com/auth/devstorage.read_only scope.

gcloud

To specify the default IAM service account used by node auto-provisioning, run the following command:

gcloud container clusters update CLUSTER_NAME \
    --enable-autoprovisioning --autoprovisioning-service-account=SERVICE_ACCOUNT

Replace the following:

• CLUSTER_NAME: the name of the cluster.
• SERVICE_ACCOUNT: the name of the default service account.

The following example sets [e-mail] as the default service account on the dev-cluster cluster:

gcloud container clusters update dev-cluster \
    --enable-autoprovisioning --autoprovisioning-service-account=[e-mail]

To specify the default scopes used by node auto-provisioning, run the following command:

gcloud container clusters update CLUSTER_NAME \
    --enable-autoprovisioning --autoprovisioning-scopes=SCOPE

Replace the following:

• CLUSTER_NAME: the name of the cluster.
• SCOPE: the Google Cloud scopes used by auto-provisioned node pools. To specify multiple scopes, separate the scopes by a comma (for example, SCOPE1, SCOPE2,...).

The following example sets the default scope on the dev-cluster cluster to devstorage.read_only:

gcloud container clusters update dev-cluster \
    --enable-autoprovisioning \
    --autoprovisioning-scopes=https://www.googleapis.com/auth/pubsub,https://www.googleapis.com/auth/devstorage.read_only

File

You can specify identity default used by node auto-provisioning by using a configuration file. The following YAML configuration sets IAM service account:

  serviceAccount: SERVICE_ACCOUNT

Replace SERVICE_ACCOUNT with the name of the default service account.

Alternatively, you can use the following YAML configuration to specify default scopes used by node auto-provisioning:

  scopes: SCOPE

Replace SCOPE with the Google Cloud scope used by auto-provisioned node pools. To specify multiple scopes, separate the scopes by a comma (for example, SCOPE1, SCOPE2,...).

To use an auto-provisioning configuration file:

1. Create a configuration file specifying identity defaults in a location where the gcloud CLI can access it.

2. Apply the configuration to your cluster by running the following command:

   gcloud container clusters update CLUSTER_NAME \
      --enable-autoprovisioning \
      --autoprovisioning-config-file FILE_NAME

Replace the following:

• CLUSTER_NAME: the name of the cluster.
• FILE_NAME: the name of the configuration file.

Customer-managed encryption keys (CMEK)

You can specify Customer Managed Encryption Keys (CMEK) used by new auto-provisioned node pools.

Note: You need to create a key before you can use it with node auto-provisioning.

You can enable customer-managed encryption for boot drives by using a configuration file. The following YAML configuration sets the CMEK key:

  bootDiskKmsKey: projects/KEY_PROJECT_ID/locations/LOCATION/keyRings/KEY_RING/cryptoKeys/KEY_NAME

Replace the following:

• KEY_PROJECT_ID: your key project ID.
• LOCATION: the location of your key ring.
• KEY_RING: the name of your key ring.
• KEY_NAME: the name of your key.

To use an auto-provisioning configuration file:

1. Create a configuration file specifying a CMEK key in a location where the gcloud CLI can access it.

2. Apply the configuration to your cluster by running the following command:

   gcloud container clusters update CLUSTER_NAME \
      --enable-autoprovisioning \
      --autoprovisioning-config-file FILE_NAME

   Replace the following:

   • CLUSTER_NAME: the name of the cluster.
   • FILE_NAME: the name of the configuration file.

Node integrity

Node auto-provisioning supports creating node pools with Secure Boot and Integrity Monitoring enabled.

You can enable Secure Boot and Integrity Monitority by using a configuration file. The following YAML configuration enables Secure Boot and disables Integrity Monitoring:

  shieldedInstanceConfig:
    enableSecureBoot: true
    enableIntegrityMonitoring: false

To use an auto-provisioning configuration file:

1. Copy the configuration above to a file in a location where the gcloud CLI can access it. Edit the values for enableSecureBoot and enableIntegrityMonitoring. Save the file.

2. Apply the configuration to your cluster by running the following command:

   gcloud container clusters update CLUSTER_NAME \
      --enable-autoprovisioning \
      --autoprovisioning-config-file FILE_NAME

   Replace the following:

   • CLUSTER_NAME: the name of the cluster.
   • FILE_NAME: the name of the configuration file.

Node auto-repair and auto-upgrade

Node auto-provisioning supports creating node pools with node auto-repair and node auto-upgrade enabled.

Warning: Disabling auto-upgrade when the underlying cluster is enrolled in a release channel can cause a failure in creating auto-provisioned node pools.

gcloud

To enable auto-repair and auto-upgrade for all new auto-provisioned node pools, run the following command:

gcloud container clusters update CLUSTER_NAME \
    --enable-autoprovisioning --enable-autoprovisioning-autorepair \
    --enable-autoprovisioning-autoupgrade

Replace CLUSTER_NAME with the name of the cluster.

To disable auto-repair and auto-upgrade for all new auto-provisioned node pools, run the following command:

gcloud container clusters update CLUSTER_NAME \
    --enable-autoprovisioning --no-enable-autoprovisioning-autorepair \
    --no-enable-autoprovisioning-autoupgrade

Replace CLUSTER_NAME with the name of the cluster.

File

You can enable or disable node auto-repair and auto-upgrade by using a configuration file. The following YAML configuration enables auto-repair and disables auto-upgrade:

  management:
    autoRepair: true
    autoUpgrade: false

To use an auto-provisioning configuration file:

1. Copy the configuration above to a file in a location where the gcloud CLI can access it. Edit the values for autoUpgrade and autoRepair. Save the file.

2. Apply the configuration to your cluster by running the following command:

gcloud container clusters update CLUSTER_NAME \
    --enable-autoprovisioning \
    --autoprovisioning-config-file FILE_NAME

Replace the following:

• CLUSTER_NAME: the name of the cluster.
• FILE_NAME: the name of the configuration file.

Use surge upgrades for new auto-provisioned node pools

You can specify surge upgrade settings on all new auto-provisioned node pools by using the gcloud CLI or a configuration file. By default, GKE sets the node upgrade strategy to surge upgrades.

Note: Before you begin, make sure you enable node auto-provisioning on the cluster.

gcloud

To specify surge upgrade settings for all new auto-provisioned node pools, run the following command:

gcloud container clusters update CLUSTER_NAME \
    --enable-autoprovisioning \
    --autoprovisioning-max-surge-upgrade MAX_SURGE \
    --autoprovisioning-max-unavailable-upgrade MAX_UNAVAILABLE

Replace the following:

• CLUSTER_NAME: the name of the cluster.
• MAX_SURGE: the maximum number of nodes that can be added to the node pool during upgrades.
• MAX_UNAVAILABLE: the maximum number of nodes in the node pool that can be simultaneously unavailable during upgrades.

File

You can specify surge upgrade settings for all new auto-provisioned node pools by using a configuration file like the following:

  upgradeSettings:
    maxSurgeUpgrade: 1
    maxUnavailableUpgrade: 2

To use an auto-provisioning configuration file:

1. Copy the configuration above to a file in a location where gcloud can access it. Edit the values for maxSurgeUpgrade and maxUnavailableUpgrade. Save the file.

2. Apply the configuration to your cluster by running the following command:

gcloud container clusters update CLUSTER_NAME \
    --enable-autoprovisioning \
    --autoprovisioning-config-file FILE_NAME

Replace the following:

• CLUSTER_NAME: the name of the cluster.
• FILE_NAME: the name of the configuration file.

For more information, see the gcloud container clusters update documentation.

To switch back to using surge upgrades for new auto-provisioned node pools, run the following command:

gcloud container clusters update CLUSTER_NAME \
  --enable-autoprovisioning \
  --enable-autoprovisioning-surge-upgrade

Replace CLUSTER_NAME with the name of the cluster.

You can optionally include the flags for specific settings as in the previous commands. GKE reuses your previous configuration for the upgrade strategy, if it was set.

Use blue-green upgrades for new auto-provisioned node pools

You can use blue-green upgrades for all new auto-provisioned node pools by using the gcloud CLI. With blue-green upgrades, you can use the default settings, or tune it to optimize for your environment. To learn more about blue-green upgrades, see Blue-green upgrades.

To update the node upgrade strategy for any existing auto-provisioned node pool, see Turn on or off surge upgrade for an existing node pool and Updating an existing node pool blue-green upgrade strategy.

The following variables are used in the commands listed below:

• CLUSTER_NAME: the name of the cluster for the node pool.
• COMPUTE_ZONE: the zone for the cluster.
• NODE_POOL_NAME: the name of the node pool.
• NUMBER_NODES: the number of nodes in the node pool in each of the cluster's zones.
• BATCH_NODE_COUNT: the number of blue nodes to drain in a batch during the blue pool drain phase. Default is one. If it is set to zero, the blue pool drain phase will be skipped.
• BATCH_PERCENT: the percentage of blue nodes to drain in a batch during the blue pool drain phase. Must be in the range of [0.0, 1.0].
• BATCH_SOAK_DURATION: the duration in seconds to wait after each batch drain. Default is zero.
• NODE_POOL_SOAK_DURATION: the duration in seconds to wait after completing drain of all batches. Default is 3600 seconds.

The default settings for blue-green upgrades are:

• BATCH_NODE_COUNT = 1
• BATCH_SOAK_DURATION = 0 seconds
• NODE_POOL_SOAK_DURATION = 3600 seconds (1 hour)

Update a cluster to use blue-green upgrades for new auto-provisioned node pools

The following commands use gcloud container clusters update to update the node upgrade strategy for new auto-provisioned node pools.

You can also use these flags when:

• using the gcloud container clusters create command to create a cluster with node auto-provisioning enabled.
• using the gcoud container clusters update command to enable node auto-provisioning.

To update a cluster to use blue-green upgrades with default settings for new auto-provisioned node pools, use this command:

gcloud container clusters update CLUSTER_NAME \
  --enable-autoprovisioning \
  --enable-autoprovisioning-blue-green-upgrade

You can update a cluster to use blue-green upgrades with specific settings for new auto-provisioned node pools. These commands can also be used without the --enable-autoprovisioning-blue-green-upgrade flag to update the settings.

The following command uses BATCH_NODE_COUNT to set an absolute node count batch size:

gcloud container clusters update CLUSTER_NAME \
  --enable-autoprovisioning \
  --enable-autoprovisioning-blue-green-upgrade \
  --autoprovisioning-node-pool-soak-duration=NODE_POOL_SOAK_DURATION \
  --autoprovisioning-standard-rollout-policy=batch-node-count=BATCH_NODE_COUNT,batch-soak-duration=BATCH_SOAK_DURATION

You can also use BATCH_PERCENT to set a percentage-based batch size, replacing batch-node-count in the last command with batch-percent and using a decimal between 0 and 1 (e.g. 25% is 0.25). To see how percentage-based batch sizes are set, see Update a node pool with blue/green upgrade using percentage-based batch sizes.

Custom boot disks

Node auto-provisioning supports creating node pools with Custom boot disks.

You can customize boot disk setting using a configuration file. GKE reserves a portion of the node boot disk for the kubelet functions. For more information, see Ephemeral storage backed by node boot disk.

The following YAML configuration causes node auto-provisioning to create node pools with 100 GB SSD disks:

  diskSizeGb: 100
  diskType: pd-ssd

Specify the following:

• diskSizeGb: the size of the disk, specified in GB.
• diskType: the type of disk, which can be one of the following values:
  • pd-balanced (default)
  • pd-standard
  • pd-ssd. In GKE version 1.22 and earlier, if you specify pd-ssd, node auto-provisioning only considers N1 machine types when creating node pools.

To use an auto-provisioning configuration file:

1. Create a file with desired boot disk configuration in a location where the gcloud CLI can access it.

2. Apply the configuration to your cluster by running the following command:

   gcloud container clusters update CLUSTER_NAME \
      --enable-autoprovisioning \
      --autoprovisioning-config-file FILE_NAME

   Replace the following:

   • CLUSTER_NAME: the name of the cluster.
   • FILE_NAME: the name of the configuration file.

Separate GKE-managed Pods from your workloads

As a cluster administrator, you might want to separate Pods that GKE manages from your workloads. This separation helps to avoid scale-down issues if there are underutilized nodes in the cluster that have system Pods running.

The following example shows how you can separate managed Pods from your workloads by using a combination of node auto-provisioning and Kubernetes taints and tolerations.

Note: Using this method doesn't guarantee that all GKE-managed Pods are scheduled on the separate node pool. Some managed workloads, like critical DaemonSets, continue to run on all your nodes regardless of any applied taints. Additionally, if GKE adjusts the resource requests of some system workloads based on traffic, those workloads might not fit on the dedicated nodes in the future.

1. Create a cluster with a default node pool of e2-standard-2 VMs and apply a node taint that only lets GKE system workloads run on those nodes:

   gcloud container clusters create test-cluster \
       --machine-type=e2-standard-2 \
       --node-taints=components.gke.io/gke-managed-components=true:NoSchedule
   

2. Enable node auto-provisioning for your cluster:

   gcloud container clusters update test-cluster \
       --enable-autoprovisioning \
       --min-cpu 1 \
       --min-memory 1 \
       --max-cpu 10 \
       --max-memory 64
   

   Your cluster is able to scale between a total cluster size of 1 CPU and 1 gigabyte of memory to a maximum of 10 CPU and 64 gigabytes of memory.

3. Test this configuration by saving the following sample manifest as nginx.yaml:

   apiVersion: v1
   kind: Pod
   metadata:
     name: nginx
     labels:
       env: test
   spec:
     containers:
     - name: nginx
       image: nginx
       imagePullPolicy: IfNotPresent
     tolerations:
     - key: dedicated
       operator: Equal
       value: ui-team
       effect: NoSchedule
     nodeSelector:
       dedicated: ui-team
   

   This manifest deploys a test workload Pod in the cluster with a nodeSelector label and node taint of dedicated: ui-team. Without node auto-provisioning, this workload Pod cannot be scheduled since no node pool has the proper label and taints.

4. Apply the manifest to the cluster:

   kubectl apply -f nginx.yaml
   

   The output is similar to the following:

   pod/nginx created
   

5. See the new node pool that fits the ui-team label:

   kubectl get node --selector=dedicated=ui-team
   

   The output is similar to the following:

   NAME                                            STATUS   ROLES    AGE   VERSION
   gke-test-nap-e2-medium-14b723z1-19f89fa8-jmhr   Ready    <none>   14s   v1.21.11-gke.900
   

Your cluster separates your workloads from managed GKE Pods.

Note: If a node pool has no taints, any Pod can be scheduled on it, regardless of whether the Pod has any tolerations. Even if you taint a node pool and add matching tolerations to your Pods, those Pods might still be scheduled onto other untainted node pools if they are available.

Limit the run time of auto-provisioned nodes

In GKE version 1.31.1-gke.1146000 and later, you can limit the run time of auto-provisioned nodes using the cloud.google.com/gke-max-run-duration-seconds node selector. Add the following field to your manifest:

spec:
  nodeSelector:
    cloud.google.com/gke-max-run-duration-seconds: "MAX_RUN_DURATION"

Replace MAX_RUN_DURATION with the duration, in seconds, you want the auto-provisioned node to run before being automatically terminated. To learn about restrictions, see MaxRunDuration Restrictions.

Use accelerators for new auto-provisioned node pools

You can enable node auto-provisioning and configure GKE to provision GPU or Cloud TPU accelerators automatically to ensure the capacity required to schedule AI/ML workloads.

Configuring GPU limits

When using node auto-provisioning with GPUs, you can set the maximum limit for each GPU type in the cluster by using the gcloud CLI or the Google Cloud console. The GPU limit count is the maximum number of GPUs. For example, a VM with 16 GPUs counts as 16 not 1 for the purpose of this limit. To configure multiple types of GPU, you must use a configuration file.

To list the available resourceTypes, run gcloud compute accelerator-types list.

Note: CPU and memory limits are also required when setting GPU limits.

gcloud

gcloud container clusters update CLUSTER_NAME \
    --enable-autoprovisioning \
    --max-cpu MAXIMUM_CPU \
    --max-memory MAXIMUM_MEMORY \
    --min-accelerator type=GPU_TYPE,count=MINIMUM_ACCELERATOR \
    --max-accelerator type=GPU_TYPE,count=MAXIMUM_ACCELERATOR

Replace the following:

• CLUSTER_NAME: the name of the cluster.
• MAXIMUM_CPU: the maximum number of cores in the cluster.
• MAXIMUM_MEMORY: the maximum number of gigabytes of memory in the cluster.
• GPU_TYPE: the GPU type.
• MINIMUM_ACCELERATOR: the minimum number of GPU accelerators in the cluster.
• MAXIMUM_ACCELERATOR: the maximum number of GPU accelerators in the cluster.

The following example sets the GPU limits for the nvidia-tesla-t4 GPU accelerator type in the dev-cluster cluster:

gcloud container clusters update dev-cluster \
    --enable-autoprovisioning \
    --max-cpu 10 \
    --max-memory 64 \
    --min-accelerator type=nvidia-tesla-t4,count=1 \
    --max-accelerator type=nvidia-tesla-t4,count=4

File

You can load limits for multiple types of GPU by using a configuration file. The following YAML configuration configures two different types of GPUs:

  resourceLimits:
    - resourceType: 'cpu'
      minimum: 4
      maximum: 10
    - resourceType: 'memory'
      maximum: 64
    - resourceType: 'nvidia-tesla-t4'
      maximum: 4
    - resourceType: 'nvidia-tesla-v100'
      maximum: 2

To use an auto-provisioning configuration file:

1. Copy the configuration above to a file in a location where the gcloud CLI can access it. Edit the values for cpu and memory. Add as many values for resourceType as you need. Save the file.

2. Apply the configuration to your cluster by running the following command:

gcloud container clusters update CLUSTER_NAME \
    --enable-autoprovisioning \
    --autoprovisioning-config-file FILE_NAME

Replace the following:

• CLUSTER_NAME: the name of the cluster.
• FILE_NAME: the name of the configuration file.

For more information, see the gcloud container clusters update documentation.

Console

To enable node auto-provisioning with GPU resources, perform the following steps:

1. Go to the Google Kubernetes Engine page in Google Cloud console.

   Go to Google Kubernetes Engine

2. Click the name of the cluster.

3. In the Automation section, for Node auto-provisioning, click Edit.

4. Select the Enable node auto-provisioning checkbox.

5. Set the minimum and maximum CPU and memory usage for the cluster.

6. Click Add resource.

7. Select the type of GPU (for example, NVIDIA T4) you want to add. Set the minimum and maximum number of GPUs to add to the cluster.

8. Accept the limitations of GPUs in GKE.

9. Click Save changes.

Choose a driver version to install

In GKE version 1.29.2-gke.1108000 and later, you can select a GPU driver version for GKE to automatically install in auto-provisioned GPU nodes. Add the following field to your manifest:

spec:
  nodeSelector:
    cloud.google.com/gke-gpu-driver-version: "DRIVER_VERSION"

Replace DRIVER_VERSION with one of the following values:

• default - the default, stable driver for your node GKE version. If you omit the nodeSelector in your manifest, this is the default option.
• latest - the latest available driver version for your node GKE version.

Configuring Cloud TPUs

To learn more about how node auto-provisioning works with TPUs, see Supported machine learning accelerators.

Create a cluster and configure your Pods to use TPU resources by using the gcloud CLI. To configure multiple types of TPU, you must use a configuration file.

gcloud

1. Create a cluster and define the TPU limits:

   gcloud container clusters create CLUSTER_NAME \
       --enable-autoprovisioning \
       [--min-cpu  MINIMUM_CPU ] \
       --max-cpu MAXIMUM_CPU \
       [--min-memory MINIMUM_MEMORY ] \
       --max-memory MAXIMUM_MEMORY \
       [--min-accelerator=type=TPU_TYPE,count= MINIMUM_ACCELERATOR ] \
       --max-accelerator=type=TPU_TYPE,count= MAXIMUM_ACCELERATOR
   

   Replace the following:

   • CLUSTER_NAME: the name of the cluster.
   • MINIMUM_CPU: the minimum number of vCPUs in the cluster.
   • MAXIMUM_CPU: the maximum number of vCPUs in the cluster.
   • MINIMUM_MEMORY:the minimum number of gigabytes of memory in the cluster.
   • MAXIMUM_MEMORY: the maximum number of gigabytes of memory in the cluster.
   • TPU_TYPE: the type of TPU you choose:
     • To select TPU v4, use tpu-v4-podslice.
     • To select TPU v5e with a machine type that begins with ct5lp-, use tpu-v5-lite-podslice.
     • To select TPU v5e with a machine type that begins with ct5l-, use
     • To select TPU v5p with a machine type that begins with ct5p-, use tpu-v5p-slice.
     • To select TPU v6e, use tpu-v6e-slice. TPU v6e is in Preview.
   • MINIMUM_ACCELERATOR: the minimum number of TPU chips in the cluster.
     • Note that using MINIMUM_ACCELERATOR may block scale down of multi-host TPU slices even if count is smaller than the number of TPU chips in the slice.
   • MAXIMUM_ACCELERATOR: the maximum number of TPU chips in the cluster.
     • If the Pod configuration requests a multi-host TPU slice, GKE creates such slice atomically. Set the count value high enough to allow the provisioning of all TPU chips of the specified topology. The number of chips in each TPU slice equals the product of the topology. For example, if the topology of the multi-host TPU slice is 2x2x2, the number of TPU chips equals 8, therefore the MAXIMUM_ACCELERATOR must be higher than 8.

   The following example sets the TPU limits for the ct5lp-hightpu-1t, ct5lp-hightpu-4t, and ct5lp-hightpu-8t machine types in the dev-cluster cluster. For example, up to ten ct5lp-hightpu-4t machines could be provisioned, each with 4 TPU chips, 112 vCPU, and 192 GiB of memory.

   gcloud container clusters create dev-cluster-inference \
         --enable-autoprovisioning \
         --min-cpu 0 \
         --max-cpu 1120 \
         --min-memory 0 \
         --max-memory 1920 \
         --min-accelerator=type=tpu-v5-lite-podslice,count=0 \
         --max-accelerator=type=tpu-v5-lite-podslice,count=40
   

2. Create a Deployment specification that results in a Pod requesting TPU resources. For example, the following manifest will cause GKE to provision four ct5lp-hightpu-4t nodes:

   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: tpu-workload
     labels:
       app: tpu-workload
   spec:
     replicas: 4
     selector:
       matchLabels:
         app: nginx-tpu
     template:
       metadata:
         labels:
           app: nginx-tpu
       spec:
         nodeSelector:
           cloud.google.com/gke-tpu-accelerator: tpu-v5-lite-podslice
           cloud.google.com/gke-tpu-topology:  2x2
           cloud.google.com/reservation-name: my-reservation
         containers:
         - name: nginx
           image: nginx:1.14.2
           resources:
             requests:
               google.com/tpu: 4
             limits:
               google.com/tpu: 4
           ports:
           - containerPort: 80
   

   In the nodeSelector field, you define the TPU type, TPU topology, and accelerator count, where:

   • cloud.google.com/gke-tpu-accelerator: Defines the TPU type. For example, tpu-v4-podslice.
   • cloud.google.com/gke-tpu-topology: Defines the TPU topology, for example 2x2x1 or 4x4x8.

   To consume an existing reservation with your workload, specify additional label in the nodeSelector field: * cloud.google.com/reservation-name: Defines the name of the reservation GKE uses to auto-provision the nodes.

   Under limits: google.com/tpu you define the number of TPU chips per node.

File

You can assign limits for multiple types of TPUs by using a configuration file. The following YAML configuration configures two different types of TPUs:

  resourceLimits:
    - resourceType: 'cpu'
      maximum: 10000
    - resourceType: 'memory'
      maximum: 10000
    - resourceType: 'tpu-v4-podslice'
      maximum: 32
    - resourceType: 'tpu-v5-lite'
      maximum: 64

To use an auto-provisioning configuration file:

1. Copy the configuration above to a file in a location where the gcloud CLI can access it. Edit the values for resourceType and maximum. Add as many values for resourceType as you need. Save the file.

2. Apply the configuration to your cluster by running the following command:

gcloud container clusters update CLUSTER_NAME \
    --enable-autoprovisioning \
    --autoprovisioning-config-file FILE_NAME

Replace the following:

• CLUSTER_NAME: the name of the cluster.
• FILE_NAME: the name of the configuration file.

For more information, see the gcloud container clusters update documentation.

Node auto-provisioning locations

You set the zones where node auto-provisioning can create new node pools. Regional locations are not supported. Zones must belong to the same region as the cluster but are not limited to node locations defined on the cluster level. Changing node auto-provisioning locations doesn't affect any existing node pools.

To set locations where node auto-provisioning can create new node pools, use the gcloud CLI or a configuration file.

gcloud

Run the following command:

gcloud container clusters update CLUSTER_NAME \
  --enable-autoprovisioning --autoprovisioning-locations=ZONE

Replace the following:

• CLUSTER_NAME: the name of the cluster.
• ZONE: the zone where node auto-provisioning can create new node pools. To specify multiple zones, separate the zones by a comma (for example, ZONE1, ZONE2,...).

File

To set locations where node auto-provisioning can create new node pools, you can use a configuration file.

Add the following YAML configuration that sets the new node pools location:

    autoprovisioningLocations:
      - ZONE

Replace ZONE with the zone where node auto-provisioning can create new node pools. To specify multiple zones, add more zones to the list. Save the file.

To use an auto-provisioning configuration file:

1. Create a configuration file in a location where gcloud CLI can access it.

2. Apply the configuration to your cluster:

   gcloud container clusters update CLUSTER_NAME \
       --enable-autoprovisioning \
       --autoprovisioning-config-file FILE_NAME
   

   Replace the following:

   • CLUSTER_NAME: the name of the cluster.
   • FILE_NAME: the path to of the configuration file.

Physically closer nodes with compact placement

Starting in GKE version 1.25, node auto-provisioning supports compact placement policy. With compact placement policy, you can instruct GKE to create node pools in closer proximity with each other within a zone.

To define a compact placement policy, add a nodeSelector to the Pod specification with the following keys:

• cloud.google.com/gke-placement-group is the identifier you assign for the group of Pods that should run together, in the same compact placement group.

• cloud.google.com/machine-family is the name of the machine family name. For more information, see the machine families that support compact placement.

The following example sets a compact placement policy with a placement group identifier of placement-group-1, and a machine family of c2:

apiVersion: v1
kind: Pod
metadata:
  ...
spec:
  ...
  nodeSelector:
    cloud.google.com/gke-placement-group: placement-group-1
    cloud.google.com/machine-family: c2

For more information, see learn how to define compact placement for GKE nodes.

Disabling node auto-provisioning

When you disable node auto-provisioning for a cluster, node pools are no longer auto-provisioned.

Note: All the node pools are marked as not auto-provisioned when you disable auto-provisioning. Reenabling auto-provisioning does not automatically mark node pools as auto-provisioned.

gcloud

To disable node auto-provisioning for a cluster, run the following command:

gcloud container clusters update CLUSTER_NAME \
  --no-enable-autoprovisioning

Replace CLUSTER_NAME with the name of your cluster.

File

To disable node auto-provisioning using the Google Cloud console:

1. Go to the Google Kubernetes Engine page in Google Cloud console.

   Go to Google Kubernetes Engine

2. Click the name of the cluster.

3. In the Automation section, for Node auto-provisioning, click the Edit.

4. Clear the Enable node auto-provisioning checkbox.

5. Click Save changes.

Marking node pool as auto-provisioned

After enabling node auto-provisioning on the cluster, you can specify which node pools are auto-provisioned. An auto-provisioned node pool is automatically deleted when no workloads are using it.

To mark a node pool as auto-provisioned, run the following command:

gcloud container node-pools update NODE_POOL_NAME \
  --enable-autoprovisioning

Replace NODE_POOL_NAME with the name of the node pool.

Marking node pool as not auto-provisioned

To mark a node pool as not auto-provisioned, run the following command:

gcloud container node-pools update NODE_POOL_NAME \
  --no-enable-autoprovisioning

Replace NODE_POOL_NAME with the name of the node pool.

Using a custom machine family

You can choose a specific Compute Engine machine series, like n2 or t2d, for your workloads by setting one of the following fields in your manifest:

• Set the node affinity with the key of cloud.google.com/machine-family, operator In, and the value being the desired machine family (for example, n2).
• Add a nodeSelector with the key of cloud.google.com/machine-family and the value being the desired machine family.

Here is an example that sets the nodeAffinity to a machine family of n2:

spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: cloud.google.com/machine-family
            operator: In
            values:
            - n2

Note: You can only specify one value in the values field of a node affinity rule for a machine series.

After applying the changes, node auto-provisioning chooses the best node pool with a machine type within the specified machine series. If you specify multiple entries for matchExpressions, GKE arbitrarily chooses a machine series from the specified entries.

Control node attributes using a custom compute class

For more control over the attributes of your auto-provisioned node pools, create and use custom compute classes. Custom compute classes let you configure scaling behaviors like fallback priorities when choosing machine types for nodes or specific resource utilization thresholds that trigger consolidating workloads on nodes to free up unused resources. Custom compute classes are available to use in GKE version 1.30.3-gke.1451000 and later.

To learn about the capabilities of custom compute classes and how to use custom compute classes with node auto-provisioning, see About custom compute classes.

Minimum CPU platform

Node auto-provisioning supports creating node pools with a minimum CPU platform specified. You can specify the minimum CPU platform at the workload level (recommended) or at the cluster level.

What's next

• Learn more about cluster autoscaler.
• Learn more about node pools.
• Learn more about node upgrade strategies.