Auto-upgrading nodes

This page shows you how to configure node auto-upgrades in Google Kubernetes Engine.

Overview

Node auto-upgrades help you keep the nodes in your cluster up to date with the cluster master version when your master is updated on your behalf. When you create a new cluster using Google Cloud Platform Console, node auto-upgrade is enabled by default.

For more information about upgrading, refer to Upgrading a Container Cluster.

Some benefits of using auto-upgrades:

• Lower management overhead: You don't have to manually track and update your nodes when the master is upgraded on your behalf.
• Better security: Sometimes new binaries are released to fix a security issue. With auto-upgrades, GKE automatically ensures that security updates are applied and kept up to date.
• Ease of use: Provides a simple way to keep your nodes up to date with the latest Kubernetes features.

Node pools with auto-upgrades enabled are automatically scheduled for upgrades soon after a cluster's masters are upgraded. When the upgrade is performed, nodes are drained and re-created to match the current cluster master version. Modifications on the boot disk of a node VM do not persist across node re-creations. To preserve modifications across node re-creation, use a DaemonSet.

Node auto-upgrade is not available for Alpha Clusters.

If you want more control over when upgrades to your nodes occur, you can schedule a maintenance window.

You can disable auto-upgrades on one or more node pools at any time. However, disabling auto-upgrades does not halt in-progress upgrades.

Note: Enabling auto-upgrades does not cause your nodes to upgrade immediately. Automatic upgrades occur at regular intervals at the discretion of the GKE team. To learn how to manually upgrade your nodes, refer to Upgrading a Container Cluster.

How node auto-upgrade works

When you enable node auto-upgrade for a node pool, your nodes are upgraded to match the version of your master shortly after the master is upgraded.

First, here are some guidelines about node auto-upgrade:

• Only one node pool's nodes are upgraded at a time.
• In a given node pool, one node is upgraded at a time. The order is chosen at random.
• Nodes are auto-upgraded during a maintenance window.

When a node is automatically upgraded, the following things happen:

1. The node is cordoned and drained. At this point, it cannot be scheduled to run new Pods.
   • PodDisruptionBudget is respected for 1 hour.
   • GracefulTerminationPeriod is limited to 1 hour.
2. Pods on the node are rescheduled onto other nodes. If a Pod can't be rescheduled, that Pod stays in PENDING state until the node is recreated.
3. The node is deleted, then re-created at the new version.
4. If the new node fails to register as healthy, auto-upgrade of the entire node pool is disabled.
5. If a significant number of node auto-upgrades to a given version result in unhealthy nodes across the GKE fleet, upgrades to that version are halted while the problem is investigated.

Creating a cluster or node pool with node auto-upgrades

gcloud

To create a cluster with auto-upgrades enabled for the default node pool, specify the --enable-autoupgrade flag in the gcloud container clusters create command:

gcloud container clusters create [CLUSTER_NAME] --zone [COMPUTE_ZONE] \
    --enable-autoupgrade

Note: By default, subsequent node pools do not have auto-upgrades enabled.

To create a node pool with auto-upgrades:

gcloud container node-pools create [NODE_POOL] --cluster [CLUSTER_NAME] \
    --zone [COMPUTE_ZONE] --enable-autoupgrade

Console

Clusters and node pools created with GCP Console have auto-upgrades enabled by default. Visit Creating a Cluster or the Adding and Managing Node Pools for instructions to create clusters and node pools.

You can disable auto-upgrades for new clusters or node pools. To do so, click Advanced edit, then deselect Enable auto-upgrade while configuring the cluster or node pool.

Enabling node auto-upgrades for an existing node pool

gcloud

To enable auto-upgrades for an existing node pool, run the following command:

gcloud container node-pools update [NODE_POOL] --cluster [CLUSTER_NAME] \
    --zone [COMPUTE_ZONE] --enable-autoupgrade

Console

To enable auto-upgrades for an existing node pool, perform the following steps:

1. Visit the Google Kubernetes Engine menu in GCP Console.

   Visit the Google Kubernetes Engine menu

2. Click the cluster's Edit button, which looks like a pencil.

3. In the Node pools section, click the Advanced edit button for the node pool you want to modify.

4. Select Enable auto-upgrade.

5. Click Save to exit the Advanced edit overlay.

6. Click Save again to modify the cluster.

Disabling node auto-upgrades for an existing node pool

Warning: Even with node auto-upgrades disabled, triggering IP rotation or enabling network policy on a cluster recreates all nodes at the same version as the master, regardless of the version selected for the node pool.

gcloud

To disable auto-upgrades for an existing node pool, run the following command:

gcloud container node-pools update [NODE_POOL] --cluster [CLUSTER_NAME] \
    --zone [COMPUTE_ZONE] --no-enable-autoupgrade

Console

To disable auto-upgrades for an existing node pool, perform the following steps:

1. Visit the Google Kubernetes Engine menu in GCP Console.

   Visit the Google Kubernetes Engine menu

2. Click the cluster's Edit button, which looks like a pencil.

3. In the Node pools section, click the Advanced edit button for the node pool you want to modify.

4. Deselect Enable auto-upgrade.

5. Click Save to exit the Advanced edit overlay.

6. Click Save again to modify the cluster.

What's next

• Learn more about node pools.