Auto-upgrading nodes

This page shows you how to configure node auto-upgrades in Google Kubernetes Engine.

Overview

Node auto-upgrades help you keep the nodes in your cluster up-to-date with the cluster control plane (master) version when your control plane is updated on your behalf. When you create a new cluster or node pool with Google Cloud Console or the gcloud command, node auto-upgrade is enabled by default.

You can learn more about cluster and node upgrades.

Node auto-upgrades provide several benefits:

• Lower management overhead: You don't have to manually track and update your nodes when the control plane is upgraded on your behalf.
• Better security: Sometimes new binaries are released to fix a security issue. With auto-upgrades, GKE automatically ensures that security updates are applied and kept up to date.
• Ease of use: Provides a simple way to keep your nodes up to date with the latest Kubernetes features.

Node pools with auto-upgrades enabled are scheduled for upgrades when they meet the selection criteria (announced in the release notes). Rollouts are phased across multiple weeks to ensure cluster and fleet stability. When the upgrade is performed, nodes are drained and re-created to match the current control plane version. Modifications on the boot disk of a node VM do not persist across node re-creations. To preserve modifications across node re- creation, use a DaemonSet.

Node auto-upgrade is not available for Alpha clusters. If you are using a cluster with Windows Server node pools, review Upgrading Windows Server node pools before enabling node auto-upgrade.

Note: Enabling auto-upgrades does not cause your nodes to upgrade immediately. For more information, see Cluster and node upgrades.

Checking the state of auto-upgrade for an existing node pool

You can check whether auto-upgrade is enabled or disabled for a node pool using Google Cloud Console or the gcloud command.

gcloud

To check the state of auto-upgrade for a node pool, run the following command:

gcloud container node-pools describe node-pool-name \
  --cluster cluster-name \
  --zone compute-zone

where:

• node-pool-name is the name of the node pool.
• cluster-name is the name of the cluster that contains the node pool.
• compute-zone is the zone for the cluster.

Console

To check the state of auto-upgrade for a node pool, perform the following:

1. Visit the Google Kubernetes Engine menu in Cloud Console.

   Visit the Google Kubernetes Engine menu

2. Click the cluster's Edit button, which looks like a pencil.

3. In the Node pools section, select the node pool you wish to examine.

4. Under the Management section, view the value for the Auto-upgrade field.

Enabling node auto-upgrades for an existing node pool

When you create a new cluster with Google Cloud Console or the gcloud command, node auto-upgrade is enabled by default.

You can enable node auto-upgrade if it is currently disabled.

gcloud

To enable auto-upgrades for an existing node pool, run the following command:

gcloud container node-pools update node-pool-name --cluster cluster-name \
    --zone compute-zone --enable-autoupgrade

where:

• node-pool-name is the name of the node pool.
• cluster-name is the name of the cluster that contains the node pool.
• compute-zone is the zone for the cluster.

Console

To enable auto-upgrades for an existing node pool, perform the following steps:

1. Visit the Google Kubernetes Engine menu in Cloud Console.

   Visit the Google Kubernetes Engine menu

2. Click the cluster's Edit button, which looks like a pencil.

3. In the Node pools section, click the name of the node pool you want to modify, and then click Edit.

4. Select Enable auto-upgrade.

5. Click Save to modify the cluster.

For more control over when nodes can be auto-upgraded, consider configuring maintenance windows and exclusions.

Disabling node auto-upgrades for an existing node pool

You can disable node auto-upgrade for an existing node pool if the underlying cluster is not currently enrolled in a release channel. If you do so, you are responsible for ensuring that the cluster's nodes run a version compatible the cluster's version.

Warning: Even with node auto-upgrades disabled, triggering IP rotation or enabling network policy on a cluster recreates all nodes at the same version as the control plane, regardless of the version selected for the node pool.

Warning: Disabling node auto-upgrades does not stop or cancel any ongoing upgrades to nodes in node pools. To cancel or stop ongoing upgrades, follow Cancelling a node upgrade. Canceling a node upgrade can be helpful in situations where you find workloads are failing on upgraded nodes and you wanted to prevent further disruption.

Note: If the upgrade is fully completed for the entire node pool, then it cannot be rolled back or cancelled. Downgrading a node pool is not possible. Instead, create a new node pool at the desired version and migrate your workloads to it. For instructions, see Migrating workloads to previous node version (temporary mitigation) section.

gcloud

To disable auto-upgrades for an existing node pool, run the following command:

gcloud container node-pools update node-pool-name --cluster cluster-name \
    --zone compute-zone --no-enable-autoupgrade

Console

To disable auto-upgrades for an existing node pool, perform the following steps:

1. Visit the Google Kubernetes Engine menu in Cloud Console.

   Visit the Google Kubernetes Engine menu

2. Click the cluster's Edit button, which looks like a pencil.

3. In the Node pools section, click the name of the node pool you want to modify, and then click Edit.

4. Clear Enable auto-upgrade.

5. Click Save to modify the cluster.

Migrate workloads to previous node version (temporary mitigation)

You cannot downgrade a node pool. However, to downgrade to a node version after the node pool was upgraded, use the following guidelines as a temporary mitigation to migrate workloads to a desired node version.

Note: These instructions assume your workloads are failing on a node pool that was recently upgraded with the latest node pool version, and that you want to downgrade to a previous node pool version where workloads were running correctly before the upgrade occurred.

1. Check if your subnet range and Pod IP range allows additional nodes in the cluster to host all Pods currently hosted by the current node pool. Check the max Pods and max nodes for a subnet range.

2. Check if the node version to be migrated to is supported.

   Note: The Kubernetes version and version skew support policy guarantees that control planes are compatible with nodes up to two minor versions older than the control plane. For example, Kubernetes 1.13 control planes are compatible with Kubernetes 1.11 nodes. For more details, see Manually upgrade a node.

3. Check if the nodes have workloads with PodDisruptionBudget that can slow down the migration of workloads.

   Note: Once the PodDisruptionBudget is set, GKE will not shut down Pods in your application if the number of pods is equal to or less than a configured limit. GKE respects a PodDisruptionBudget for up to 60 minutes. For more details, see how to Safely drain a node while respecting the PodDisruptionBudget.

4. Create another node pool using the previous node version with a capacity (number of nodes) to host all Pods currently hosted by the current node pool.

5. Drain nodes on the node pool with the latest version using kubectl drain <nodename>, one node at a time. Confirm that Pods on the node have moved to a node on the new node pool (that's running the previous node version).

6. Extremely important: Confirm that all Pods have migrated to nodes on the node pool that's running the previous node version.

7. Delete the node pool that's running the latest node pool version, if you no longer need it.

Note: For detailed steps with commands, refer to a similar approach explained in Migrating workloads to different machine types.

Creating a cluster or node pool with node auto-upgrades enabled

gcloud

To create a cluster with auto-upgrades enabled for the default node pool, specify the --enable-autoupgrade flag in the gcloud container clusters create command:

gcloud container clusters create cluster-name --zone compute-zone \
    --enable-autoupgrade

Note: By default, subsequent node pools do not have auto-upgrades enabled.

To create a node pool with auto-upgrade enabled:

gcloud container node-pools create node-pool-name --cluster cluster-name \
    --zone compute-zone --enable-autoupgrade

Console

Clusters and node pools created with Cloud Console have auto-upgrades enabled by default. Visit Creating a cluster or the Adding and managing node pools for instructions to create clusters and node pools.

You can disable auto-upgrades for new node pools. From the cluster creation page, click the name of the node pool you want to modify, then clear Enable auto-upgrade.

Changing surge upgrade parameters

Surge Upgrades allow you to change the number of nodes GKE upgrades at one time and the amount of disruption an upgrade makes on your workloads.

The max-surge-upgrade and max-unavailable-upgrade flags are defined for each node pool. For more information on chosing the right parameters, go to Determining your optimal surge configuration.

Note: All new node pools are automatically configured to use surge upgrades maxSurge=1 maxUnavailable=0. To tune your configuration, see Determining your optimal surge configuration.

You can change these settings when creating or updating a cluster or node pool.

The following variables are used in the commands mentioned below:

• CLUSTER_NAME: the name of the cluster for the node pool.
• COMPUTE_ZONE: the zone for the cluster.
• NODE_POOL_NAME: the name of the node pool.
• NUMBER_NODES: the number of nodes in the node pool in each of the cluster's zones.
• SURGE_NODES: the number of extra (surge) nodes to be created on each upgrade of the node pool.
• UNAVAILABLE_NODES: the number of nodes that can be unavailable at the same time on each upgrade of the node pool.

Creating a cluster with specific surge parameters

To create a cluster with specific settings for surge upgrades, use the max-surge-upgrade and max-unavailable-upgrade flags.

gcloud container clusters create CLUSTER_NAME \
  --max-surge-upgrade=SURGE_NODES --max-unavailable-upgrade=UNAVAILABLE_NODES

Creating a cluster with surge upgrade disabled

To create a cluster without surge upgrades, set the value for the max-surge-upgrade flag to 0.

gcloud container clusters create CLUSTER_NAME \
  --max-surge-upgrade=0 --max-unavailable-upgrade=1

Creating a node pool with specific surge parameters

To create a node pool in an existing cluster with specific settings for surge upgrades, use the max-surge-upgrade and max-unavailable-upgrade flags.

gcloud container node-pools create NODE_POOL_NAME \
  --num-nodes=NUMBER_NODES --cluster=CLUSTER_NAME \
  --max-surge-upgrade=SURGE_NODES --max-unavailable-upgrade=UNAVAILABLE_NODES

Turn on or turn off Surge Upgrade for an existing node pool

To update the upgrade settings of an existing node pool, use the max-surge-upgrade and max-unavailable-upgrade flags. If you set max-surge-upgrade to greater than 0, GKE creates surge nodes. If you set max-surge-upgrade to 0, GKE doesn't create surge nodes.

gcloud beta container node-pools update NODE_POOL_NAME \
  --cluster=CLUSTER_NAME \
  --max-surge-upgrade=SURGE_NODES --max-unavailable-upgrade=UNAVAILABLE_NODES

Checking if surge upgrades are enabled on a node pool

To see if surge upgrades are enabled on a node pool, use gcloud to describe the cluster's parameters:

gcloud container node-pools describe NODE_POOL_NAME \
--cluster=CLUSTER_NAME

Note: While recreating nodes does not require additional Compute Engine resources, surge upgrading nodes does. Resource allocation is subjected to Compute Engine quota. Depending on your configuration, this quota can limit the number of parallel upgrades or even cause the upgrade to fail.

For more information about quota, go to Node upgrades and quota.

Receiving upgrade notifications

Beta

This product or feature is covered by the Pre-GA Offerings Terms of the Google Cloud Platform Terms of Service. Pre-GA products and features may have limited support, and changes to pre-GA products and features may not be compatible with other pre-GA versions. For more information, see the launch stage descriptions.

GKE publishes upgrade notifications to Pub/Sub, providing you with a channel to receive information from GKE about your clusters.

For more information, see Receiving cluster upgrade notifications.

What's next

• Learn more about node pools.
• Learn more about cluster and node pool upgrades.
• Manually upgrade a cluster or node pool.