Upgrading a cluster

This page explains how to upgrade or downgrade the version of Kubernetes that your Google Kubernetes Engine cluster runs.

See Versioning and Upgrades to learn about how Kubernetes versions are made available for your cluster master and nodes. To learn more about clusters, refer to Cluster Architecture.

Before you begin

To prepare for this task, perform the following steps:

• Ensure that you have enabled the Google Kubernetes Engine API.
Enable Google Kubernetes Engine API
• Ensure that you have installed the Cloud SDK.
• Set your default project ID:

  gcloud config set project [PROJECT_ID]

• If you are working with zonal clusters, set your default compute zone:

  gcloud config set compute/zone [COMPUTE_ZONE]

• If you are working with regional clusters, set your default compute region:

  gcloud config set compute/region [COMPUTE_REGION]

• Update gcloud to the latest version:

  gcloud components update

Note: You can override these default settings in gcloud commands using the --project, --zone, and --region operational flags.

About upgrading clusters

A cluster's master and nodes are upgraded separately.

Supported versions

To find the supported Kubernetes master and node versions for upgrades and downgrades, run the following command:

gcloud container get-server-config

Downgrading a cluster is not recommended. A cluster can be downgraded to a patch version older than the current master version. You cannot downgrade a cluster from one minor version to another. For example, if a cluster is running GKE 1.11.5, you can downgrade to 1.11.4 if it is still available, but you cannot downgrade to 1.10.9. Instead, an error like the following is generated:

ERROR: (gcloud.container.clusters.upgrade) ResponseError: code=400,
message=Master cannot be upgraded to "1.10.9-gke.7": specified version is not
newer than the current version.

To downgrade a cluster to a previous patch version, update the cluster master version using the gcloud command line tool.

GKE versions are rolled out according to the schedule in Supported Versions. The roll out takes a few days to reach all zones. You can check which versions are available in a zone or region using gcloud container get-server-config.

Downgrading a node pool is not possible. Instead, create a new node pool at the desired version and migrate your workloads to it. If you enable node auto-upgrade on the node pool, nodes are upgraded to match the master version.

Saving your data to persistent disks

Before upgrading a node pool, you must ensure that any data you wish to keep is stored in a Pod using persistent volumes which use persistent disks. Persistent disks are unmounted, rather than erased, during upgrades, and their data is "handed off" between Pods.

The following restrictions pertain to persistent disks:

• The nodes on which Pods are running must be Compute Engine VMs
• Those VMs need to be in the same Compute Engine project and zone as the persistent disk

To learn how to add a persistent disk to an existing node instance, see Adding or Resizing Persistent Disks.

Upgrading the master

In the following sections, you can substitute the word "upgrading" with "downgrading". Before downgrading a cluster or a node pool, be sure you understand the caveats of downgrading clusters.

Automatically upgrading the master

Google upgrades cluster masters automatically. To learn more about automatic master upgrades, see Versioning and upgrades.

You can initiate a manual upgrade any time after a new version becomes available.

Manually upgrade the master

Note: You cannot upgrade your cluster master more than one minor version at a time. For example, you can upgrade a cluster from version 1.12.x to 1.13.x, but not directly from 1.11.x to 1.13.x. For more information, refer to Versioning and Upgrades.

When initiating a cluster master upgrade, you can't modify the cluster's configuration for several minutes, until the control plane is accessible again. If you need to prevent downtime during master upgrades, consider using a regional cluster

You can manually upgrade your masters using the GCP Console or the gcloud command-line tool. Once you've upgraded your cluster's master, you can upgrade the nodes to the same version. If node auto-upgrade is enabled, Google upgrades your nodes after upgrading the control plane.

gcloud

To upgrade your cluster master's version, first run the following command to see the available versions:

gcloud container get-server-config

To upgrade to the latest version, run the following command:

gcloud container clusters upgrade [CLUSTER_NAME] --master

To upgrade to a specific version that is not the default, run the following command:

gcloud container clusters upgrade [CLUSTER_NAME] --master --cluster-version [CLUSTER_VERSION]

Refer to the gcloud container clusters upgrade documentation.

Console

To manually update your cluster master, perform the following steps:

1. Visit the Google Kubernetes Engine menu in Google Cloud Platform Console.

   Visit the Google Kubernetes Engine menu

2. Select the desired cluster.

3. Click the Upgrade available link next to Master version.

4. Select the desired version, then click Change.

5. Click the arrow at the top of the screen to go back to the cluster overview page.

Upgrading nodes

When you upgrade cluster nodes, GKE stops scheduling new Pods onto the node being upgraded and attempts to reschedule those Pods onto different nodes. Replacement nodes are recreated with the same name as their predecessors, just like any other event that recreates a node.

Note: During automatic or manual node upgrades, PodDisruptionBudgets (PDBs) are respected for a maximum of 1 hour. If Pods running on a node cannot be scheduled onto new nodes within 1 hour, the upgrade is initiated, regardless.

For more information about what to expect during node termination in general, see the topic about Pods.

The upgrade is only complete when all nodes have been recreated and the cluster is in the desired state. When a newly-upgraded node registers with the master, GKE marks the nodes as schedulable.

New node instances run the desired Kubernetes version as well as:

• The base OS image
• The Docker daemon
• kubelet
• kube-proxy

Node auto-upgrade

Node auto-upgrade is enabled by default for new clusters. When node auto-upgrade is enabled, Google ensures that the version of GKE on your nodes is kept up to date with the version running on the master.

To enable automatic node upgrades, refer to the Node Auto-Upgrades documentation.

Manually upgrade a node

You can only upgrade a node pool version to match the version of the master or to a previous version that is still available and is compatible with the master. Kubernetes guarantees that masters are compatible with nodes up to two minor versions older than the master. For example, Kubernetes 1.13 masters are compatible with Kubernetes 1.11 nodes.

You cannot downgrade a node pool. Instead, create a new node pool using the older version and migrate workloads to the new node pool.

You can manually upgrade your nodes to a version compatible with the master, using the Google Cloud Platform Console or the gcloud command-line tool.

gcloud

The following command upgrades your nodes to the version that your master is running:

gcloud container clusters upgrade [CLUSTER_NAME] --node-pool=[NODE-POOL-NAME]

where [CLUSTER_NAME] is the name of the cluster to be upgraded.

To specify a different version of GKE on nodes, use the optional --cluster-version flag:

gcloud container clusters upgrade [CLUSTER_NAME] --node-pool=[NODE-POOL-NAME] --cluster-version [CLUSTER_VERSION]

where [CLUSTER_VERSION] is the Kubernetes version to which the nodes are upgraded. For example, --cluster-version=1.7.2 or cluster-version=latest.

For more information about specifying versions, refer to the Versioning and Upgrades documentation.

For more information, refer to the gcloud container clusters upgrade documentation.

Console

To upgrade a node pool using GCP Console, perform the following steps:

1. Visit the Google Kubernetes Engine menu in GCP Console.

   Visit the Google Kubernetes Engine menu

2. Click the cluster's Edit button, which looks like a pencil.

3. In the Node Pools section, expand the disclosure arrow for the node pool you want to upgrade.

4. Click the Change link next to Node version.

5. Select the desired version, then click Change.

6. Click Save.

Note: It may take several minutes for the node version to change.

Checking node upgrade status

Beta

This product or feature is in a pre-release state and might change or have limited support. For more information, see Product launch stages.

You can check the status of an upgrade using gcloud beta container operations.

Note: To use gcloud beta commands, you must configure gcloud to use the v1beta1 API.

To see a list of every running and completed operation in the cluster, run the following command:

gcloud beta container operations list

Each operation is assigned an operation ID and an operation type as well as start and end times, target cluster, and status. The list appears similar to the following example:

NAME                              TYPE                ZONE           TARGET              STATUS_MESSAGE  STATUS  START_TIME                      END_TIME
operation-1505407677851-8039e369  CREATE_CLUSTER      us-west1-a     my-cluster                          DONE    20xx-xx-xxT16:47:57.851933021Z  20xx-xx-xxT16:50:52.898305883Z
operation-1505500805136-e7c64af4  UPGRADE_CLUSTER     us-west1-a     my-cluster                          DONE    20xx-xx-xxT18:40:05.136739989Z  20xx-xx-xxT18:41:09.321483832Z
operation-1505500913918-5802c989  DELETE_CLUSTER      us-west1-a     my-cluster                          DONE    20xx-xx-xxT18:41:53.918825764Z  20xx-xx-xxT18:43:48.639506814Z

To get more information about a specific operation, specify the operation ID in the following command:

gcloud beta container operations describe [OPERATION_ID]

For example:

gcloud beta container operations describe operation-1507325726639-981f0ed6
endTime: '20xx-xx-xxT21:40:05.324124385Z'
name: operation-1507325726639-981f0ed6
operationType: UPGRADE_CLUSTER
selfLink: https://container.googleapis.com/v1/projects/.../kubernetes-engine/docs/zones/us-central1-a/operations/operation-1507325726639-981f0ed6
startTime: '20xx-xx-xxT21:35:26.639453776Z'
status: DONE
targetLink: https://container.googleapis.com/v1/projects/.../kubernetes-engine/docs/zones/us-central1-a/clusters/...
zone: us-central1-a

Canceling a node upgrade

Beta

This is a Beta release of gcloud container operations. This feature is not covered by any SLA or deprecation policy and might be subject to backward-incompatible changes.

You can cancel an upgrade at any time. When you cancel an upgrade:

• Nodes that have started the upgrade complete it
• Nodes that have not started the upgrade do not upgrade
• Nodes that have already successfully completed the upgrade are unaffected and are not rolled back

First, get the upgrade's operation ID using the following command:

gcloud container operations list

Then, to cancel the upgrade, run the following command:

gcloud beta container operations cancel [OPERATION_ID]

Refer to the gcloud container operations cancel documentation.

Rolling back a node upgrade

You can roll back node pools that failed to upgrade, or whose upgrades were canceled, to their previous version of Kubernetes. You cannot roll back node pools once they have been successfully upgraded. Nodes that have not started an upgrade are unaffected.

To roll back an upgrade, run the following command:

gcloud container node-pools rollback [POOL_NAME] --cluster [CLUSTER_NAME]

where:

• [POOL_NAME] is the name of the node pool to roll back
• [CLUSTER_NAME] is the name of the cluster from which to roll back the node pool
• [COMPUTE_ZONE] is your project's compute zone

Refer to the gcloud container node-pools rollback documentation.

What's next

• Learn more about clusters.