Auto-upgrading nodes

This page shows you how to configure node auto-upgrades in Google Kubernetes Engine (GKE).

Overview

Node auto-upgrades help you keep the nodes in your cluster up-to-date with the cluster control plane (master) version when your control plane is updated on your behalf. When you create a new cluster or node pool with Google Cloud Console or the gcloud command, node auto-upgrade is enabled by default.

You can learn more about cluster and node upgrades.

Node auto-upgrades provide several benefits:

  • Lower management overhead: You don't have to manually track and update your nodes when the control plane is upgraded on your behalf.
  • Better security: Sometimes new binaries are released to fix a security issue. With auto-upgrades, GKE automatically ensures that security updates are applied and kept up to date.
  • Ease of use: Provides a simple way to keep your nodes up to date with the latest Kubernetes features.

Node pools with auto-upgrades enabled are scheduled for upgrades when they meet the selection criteria (announced in the release notes). Rollouts are phased across multiple weeks to ensure cluster and fleet stability. When the upgrade is performed, nodes are drained and re-created to match the current control plane version. Modifications on the boot disk of a node VM do not persist across node re-creations. To preserve modifications across node re-creation, use a DaemonSet.

Node auto-upgrade is not available for Alpha clusters. If you are using a cluster with Windows Server node pools, review Upgrading Windows Server node pools before enabling node auto-upgrade.

Checking the state of auto-upgrade for an existing node pool

You can check whether auto-upgrade is enabled or disabled for a node pool using Google Cloud Console or the gcloud command.

gcloud

To check the state of auto-upgrade for a node pool, run the following command:

gcloud container node-pools describe NODE_POOL_NAME \
  --cluster CLUSTER_NAME \
  --zone COMPUTE_ZONE

Replace the following:

  • NODE_POOL_NAME: the name of the node pool.
  • CLUSTER_NAME: the name of the cluster that contains the node pool.
  • COMPUTE_ZONE: the compute zone for the cluster.

Console

To check the state of auto-upgrade for a node pool, perform the following:

  1. Go to the Google Kubernetes Engine page in Cloud Console.

    Go to Google Kubernetes Engine

  2. In the cluster list, click the name of the cluster you want to view.

  3. Click the Nodes tab.

  4. Under Node Pools, click the name of the node pool you want to view.

  5. On the Node pool details page, under Management, view the value of the Auto-upgrade field.

Enabling node auto-upgrades for an existing node pool

When you create a new cluster with Google Cloud Console or the gcloud command, node auto-upgrade is enabled by default.

You can enable node auto-upgrade if it is currently disabled.

gcloud

To enable auto-upgrades for an existing node pool, run the following command:

gcloud container node-pools update NODE_POOL_NAME \
    --cluster CLUSTER_NAME \
    --zone COMPUTE_ZONE \
    --enable-autoupgrade

Replace the following:

  • NODE_POOL_NAME: the name of the node pool.
  • CLUSTER_NAME: the name of the cluster that contains the node pool.
  • COMPUTE_ZONE: the compute zone for the cluster.

Console

To enable auto-upgrades for an existing node pool, perform the following steps:

  1. Go to the Google Kubernetes Engine page in Cloud Console.

    Go to Google Kubernetes Engine

  2. In the cluster list, click the name of the cluster you want to modify.

  3. Click the Nodes tab.

  4. Under Node Pools, click the name of the node pool you want to modify.

  5. On the Node pool details page, click Edit.

  6. Under Management, select the Enable auto-upgrade checkbox.

  7. Click Save.

For more control over when nodes can be auto-upgraded, consider configuring maintenance windows and exclusions.

Checking the status of node upgrades

To check the status of an upgrade, see Checking node upgrade status.

Disabling node auto-upgrades for an existing node pool

Although not recommended, you can disable node auto-upgrade for an existing node pool if the underlying cluster is not currently enrolled in a release channel. Opting out of node auto-upgrades does not block your cluster's control plane upgrade. If you disable node auto-upgrade, you are responsible for ensuring that the cluster's nodes run a version compatible with the cluster's version, and that the version adheres to the Kubernetes version and version skew support policy.

gcloud

To disable auto-upgrades for an existing node pool, run the following command:

gcloud container node-pools update NODE_POOL_NAME \
    --cluster CLUSTER_NAME \
    --zone COMPUTE_ZONE \
    --no-enable-autoupgrade

Console

To disable auto-upgrades for an existing node pool, perform the following steps:

  1. Go to the Google Kubernetes Engine page in Cloud Console.

    Go to Google Kubernetes Engine

  2. In the cluster list, click the name of the cluster you want to modify.

  3. Click the Nodes tab.

  4. Under Node Pools, click the name of the node pool you want to modify.

  5. On the Node pool details page, click Edit.

  6. Under Management, clear the Enable auto-upgrade checkbox.

  7. Click Save to modify the cluster.

Migrate workloads to previous node version (temporary mitigation)

You cannot downgrade a node pool. However, to downgrade to a node version after the node pool was upgraded, use the following guidelines as a temporary mitigation to migrate workloads to a desired node version.

  1. Check if your subnet range and Pod IP range allows additional nodes in the cluster to host all Pods currently hosted by the current node pool. Check the max Pods and max nodes for a subnet range.
  2. Check if the node version to be migrated to is supported.

  3. Check if the nodes have workloads with PodDisruptionBudget that can slow down the migration of workloads.

  4. Create another node pool using the previous node version with a capacity (number of nodes) to host all Pods currently hosted by the current node pool.

  5. Drain nodes on the node pool with the latest version using kubectl drain <nodename>, one node at a time. Confirm that Pods on the node have moved to a node on the new node pool (that's running the previous node version).

  6. Extremely important: Confirm that all Pods have migrated to nodes on the node pool that's running the previous node version.

  7. Delete the node pool that's running the latest node pool version, if you no longer need it.

Creating a cluster or node pool with node auto-upgrades enabled

gcloud

To create a cluster with auto-upgrades enabled for the default node pool, specify the --enable-autoupgrade flag in the gcloud container clusters create command:

gcloud container clusters create CLUSTER_NAME \
    --zone COMPUTE_ZONE \
    --enable-autoupgrade

To create a node pool with auto-upgrade enabled specify the --enable-autoupgrade flag in the gcloud container node-pools create command:

gcloud container node-pools create NODE_POOL_NAME \
    --cluster CLUSTER_NAME \
    --zone COMPUTE_ZONE \
    --enable-autoupgrade

Console

Clusters and node pools created with Cloud Console have auto-upgrades enabled by default. Visit Creating a cluster or Adding and managing node pools for instructions to create clusters and node pools.

You can disable auto-upgrades for new node pools. From the cluster creation page, click the name of the node pool you want to modify, then clear Enable auto-upgrade.

Changing surge upgrade parameters

Surge Upgrades allow you to change the number of nodes GKE upgrades at one time and the amount of disruption an upgrade makes on your workloads.

The max-surge-upgrade and max-unavailable-upgrade flags are defined for each node pool. For more information on chosing the right parameters, go to Determining your optimal surge configuration.

You can change these settings when creating or updating a cluster or node pool.

The following variables are used in the commands mentioned below:

  • CLUSTER_NAME: the name of the cluster for the node pool.
  • COMPUTE_ZONE: the zone for the cluster.
  • NODE_POOL_NAME: the name of the node pool.
  • NUMBER_NODES: the number of nodes in the node pool in each of the cluster's zones.
  • SURGE_NODES: the number of extra (surge) nodes to be created on each upgrade of the node pool.
  • UNAVAILABLE_NODES: the number of nodes that can be unavailable at the same time on each upgrade of the node pool.

Creating a cluster with specific surge parameters

To create a cluster with specific settings for surge upgrades, use the max-surge-upgrade and max-unavailable-upgrade flags.

gcloud container clusters create CLUSTER_NAME \
  --max-surge-upgrade=SURGE_NODES --max-unavailable-upgrade=UNAVAILABLE_NODES

Creating a cluster with surge upgrade disabled

To create a cluster without surge upgrades, set the value for the max-surge-upgrade flag to 0.

gcloud container clusters create CLUSTER_NAME \
  --max-surge-upgrade=0 --max-unavailable-upgrade=1

Creating a node pool with specific surge parameters

To create a node pool in an existing cluster with specific settings for surge upgrades, use the max-surge-upgrade and max-unavailable-upgrade flags.

gcloud container node-pools create NODE_POOL_NAME \
  --num-nodes=NUMBER_NODES --cluster=CLUSTER_NAME \
  --max-surge-upgrade=SURGE_NODES --max-unavailable-upgrade=UNAVAILABLE_NODES

Turn on or turn off Surge Upgrade for an existing node pool

To update the upgrade settings of an existing node pool, use the max-surge-upgrade and max-unavailable-upgrade flags. If you set max-surge-upgrade to greater than 0, GKE creates surge nodes. If you set max-surge-upgrade to 0, GKE doesn't create surge nodes.

gcloud beta container node-pools update NODE_POOL_NAME \
  --cluster=CLUSTER_NAME \
  --max-surge-upgrade=SURGE_NODES --max-unavailable-upgrade=UNAVAILABLE_NODES

Checking if surge upgrades are enabled on a node pool

To see if surge upgrades are enabled on a node pool, use gcloud to describe the cluster's parameters:

gcloud container node-pools describe NODE_POOL_NAME \
--cluster=CLUSTER_NAME

Receiving upgrade notifications

GKE publishes upgrade notifications to Pub/Sub, providing you with a channel to receive information from GKE about your clusters.

For more information, see Receiving cluster upgrade notifications.

What's next