Cluster Autoscaler

Container Engine's cluster autoscaler automatically resizes clusters based on the demands of the workloads you want to run. With autoscaling enabled, Container Engine automatically adds a new node to your cluster if you've created new pods that don't have enough capacity to run; conversely, if a node in your cluster is underutilized and its pods can be run on other nodes, Container Engine can delete the node.

Cluster autoscaling allows you to pay only for resources that are needed at any given moment, and to automatically get additional resources when demand increases.

Keep in mind that when resources are deleted or moved in the course of autoscaling your cluster, your services can experience some disruption. For example, if your service consists of a controller with a single replica, that replica's pod might be restarted on a different node if its current node is deleted. Before enabling autoscaling, ensure that your services can tolerate potential disruption.

How cluster autoscaler works

Cluster autoscaler works on a per-node pool basis. For each node pool, the autoscaler periodically checks whether there are any pods that are not being scheduled and are waiting for a node with available resources. If such pods exist, and the autoscaler determines that resizing a node pool would allow the waiting pods to be scheduled, then the autoscaler expands that node pool.

Cluster autoscaler also measures the usage of each node against the node pool's total demand for capacity. If a node has had no new pods scheduled on it for a set period of time, and all pods running on that node can be scheduled onto other nodes in the pool, the autoscaler moves the pods and deletes the node.

Note that cluster autoscaler works based on Pod resource requests, that is, how many resources your Pods have requested. Cluster autoscaler does not take into account the resources your Pods are actively using. Essentially, cluster autoscaler trusts that the Pod resource requests you've provided are accurate and schedules Pods on nodes based on that assumption.

If your Pods have requested too few resources (or haven't changed the defaults, which might be insufficient) and your nodes are experiencing shortages, cluster autoscaler does not correct the situation. You can help ensure cluster autoscaler works as accurately as possible by making explicit resource requests for all of your workloads.

Operating criteria

Cluster autoscaler makes the following assumptions when resizing a node pool:

  • Cluster autoscaler assumes that all replicated pods can be restarted on some other node, possibly causing a brief disruption. If your services are not disruption-tolerant, using autoscaling is not recommended.
  • Cluster autoscaler assumes that users or administrators are not manually managing nodes; it may override any manual node management operations you perform.
  • Cluster autoscaler assumes that all nodes in a single node pool have the same set of labels.
  • Cluster autoscaler considers the relative cost of each instance type in the node pool and attempts to expand the least expensive possible node pool.
  • Cluster autoscaler does not track labels manually added after initial cluster or node pool creation. Nodes created by cluster autoscaler are assigned labels specified with --node-labels at the time of node pool creation.

Balancing across zones

If your cluster contains multiple node pools with the same instance type, cluster autoscaler will attempt to keep those node pools' sizes balanced when scaling up. This can help prevent an uneven distribution of node pool sizes when you have node pools in multiple zones.

If you have a node pool that you want to exempt from pool size balancing, you can do so by giving that node pool a custom label.

See the Kubernetes documentation FAQ for autoscaling for more information on how cluster autoscaler makes balancing decisions.

Minimum and maximum node pool size

You can specify the minimum and maximum size for each node pool in your cluster, and cluster autoscaler makes rescaling decisions within these boundaries. If the current cluster size is lower than the specified minimum or greater than the specified maximum when you enable autoscaling, the autoscaler waits to take effect until a new node is needed or a node can be safely deleted.

Limitations

Cluster autoscaler is in Beta and is subject to the following limitations:

  • Cluster autoscaler should not yet be used with large clusters (more than 100-150 nodes).
  • When scaling down, cluster autoscaler only supports a graceful termination period for a pod of up to 60 seconds. A pod is always killed after a maximum of 60 seconds, even if the pod is configured with a higher grace period. This is a known issue in cluster autoscaler.

Additional Information

You can find more information about cluster autoscaler in the Autoscaling FAQ in the open-source Kubernetes project.

Using cluster autoscaler

The following sections explain how to use cluster autoscaler.

Create a cluster with autoscaling

gcloud

The following command creates a cluster of size 30, with node autoscaling based on cluster load that scales the default node pool to a maximum of 50 nodes and a minimum of 15 nodes:

gcloud beta container clusters create [CLUSTER-NAME] --zone=[ZONE] --num-nodes=30 --enable-autoscaling --min-nodes=15 --max-nodes=50

In this command:

  • --enable-autoscaling indicates that autoscaling is enabled.
  • --min-nodes specifies the minimum number of nodes for the node pool.
  • --max-nodes specifies the maximum number of nodes for the node pool.
  • --zone specifies the compute zone in which the autoscaler should create new nodes.

Console

  1. Visit the Container Engine menu in the Google Cloud Platform Console
  2. Click Create cluster.
  3. Configure the cluster as desired, then click More at the bottom of the menu.
  4. From the Autoscaling drop-down menu, click On.
  5. Change the values of the Minimal size and Maximal size fields as desired.
  6. Click Create.

Create a node pool with autoscaling

gcloud

The following command creates a node pool of size 3 (default), with node autoscaling based on cluster load that scales the node pool to a maximum of 5 nodes and a minimum of 1 node:

gcloud beta container node-pools create [POOL-NAME] --cluster=[CLUSTER-NAME] \
--zone=[ZONE] --enable-autoscaling --min-nodes=1 --max-nodes=5

In this command:

  • --cluster indicates the cluster in which the node is created.
  • --enable-autoscaling indicates that autoscaling is enabled.
  • --min-nodes specifies the minimum number of nodes for the node pool.
  • --max-nodes specifies the maximum number of nodes for the node pool.
  • --zone specifies the compute zone in which the autoscaler should create new nodes.

Console

  1. Visit the Container Engine menu in the Google Cloud Platform Console
  2. Select the desired cluster, then click Edit.
  3. From the Node Pools menu, click Add node pool.
  4. Configure the node pool as desired, Then, from the Autoscaling drop-down menu, select On.
  5. Click Save.

Enable autoscaling on existing cluster

gcloud

To enable autoscaling for a cluster's default node pool, run the following command:

gcloud beta container clusters update [CLUSTER-NAME] --enable-autoscaling \
--min-nodes=1 --max-nodes=10 --zone=[ZONE] --node-pool=default-pool

In this command:

  • --enable-autoscaling indicates that autoscaling is enabled.
  • --node-pool specifies the node pool for which autoscaling is enabled. Default value is default-pool. You must specify this flag if you have multiple node pools.
  • --min-nodes specifies the minimum number of nodes for the node pool.
  • --max-nodes specifies the maximum number of nodes for the node pool.
  • --zone specifies the compute zone in which the autoscaler should create new nodes.

To enable autoscaling for a specific node pool, specify the node pool in the --node-pool flag:

gcloud beta container clusters update [CLUSTER-NAME] --enable-autoscaling \
--min-nodes=1 --max-nodes=10 --zone=[ZONE] --node-pool=[POOL-NAME]

Console

To enable autoscaling for an existing cluster's default node pool, perform the following steps:

  1. Visit the Container Engine menu in Google Cloud Platform Console
  2. Click the name of the desired cluster.
  3. Click Edit.
  4. From the Autoscaling drop-down menu, click On.
  5. Change the values of the Minimal size and Maximal size fields as desired.
  6. Click Save.

To enable autoscaling for a specific node pool, perform the following steps:

  1. Visit the Container Engine menu in Google Cloud Platform Console
  2. Click the name of the desired cluster.
  3. Click Edit.
  4. From the Node Pools menu, select the desired node pool by clicking its edit icon.
  5. From the Autoscaling drop-down menu, click On.
  6. Click Save.

Disable autoscaling on an existing cluster

gcloud

The following command disables cluster autoscaling for an existing cluster's default node pool. The cluster size is fixed at the cluster's current default node pool size, which can be manually updated.

gcloud beta container clusters update [CLUSTER-NAME] --zone=[ZONE] --no-enable-autoscaling

In this command:

  • --no-enable-autoscaling instructs the cluster to disable autoscaling.
  • --zone specifies the compute zone in which the autoscaler should create new nodes.

To disable autoscaling for a specific node pool, specify the node pool in the --node-pool flag:

gcloud beta container clusters update [CLUSTER-NAME] --enable-autoscaling \
--min-nodes=1 --max-nodes=10 --zone=[ZONE] --node-pool=[POOL-NAME]

Console

To disable autoscaling for the default node pool, perform the following steps:

  1. Visit the Container Engine menu in Google Cloud Platform Console
  2. Click the name of the desired cluster.
  3. Click Edit.
  4. From the Autoscaling drop-down menu, click Off.
  5. Click Save.

To disable autoscaling for a specific node pool, perform the following steps:

  1. Visit the Container Engine menu in Google Cloud Platform Console
  2. Click the name of the desired cluster.
  3. Click Edit.
  4. From the Node Pools menu, select the desired node pool by clicking its edit icon.
  5. From the Autoscaling drop-down menu, click Off.
  6. Click Save.

Monitor your resources on the go

Get the Google Cloud Console app to help you manage your projects.

Send feedback about...

Container Engine