Manually upgrading a cluster or node pool

Standard

---

By default, Google Kubernetes Engine clusters and node pools are upgraded automatically by Google. This page explains how to manually request an upgrade or downgrade for a GKE cluster or its nodes. You can learn more about how automatic and manual cluster upgrades work. You can also control when auto-upgrades can and cannot occur by configuring maintenance windows and exclusions.

Note about terminology: A cluster's control plane (master) and nodes do not necessarily run the same version at all times. In this topic, cluster upgrade and control plane upgrade are used interchangeably, and are differentiated from node upgrades.

New versions of GKE are announced regularly. To learn about about available versions, see Versioning. To learn more about clusters, see Cluster architecture. For guidance on upgrading clusters, see Best practices for upgrading clusters.

Before you begin

Before you start, make sure you have performed the following tasks:

• Ensure that you have enabled the Google Kubernetes Engine API.
Enable Google Kubernetes Engine API
• Ensure that you have installed the Cloud SDK.

Set up default gcloud settings using one of the following methods:

• Using gcloud init, if you want to be walked through setting defaults.
• Using gcloud config, to individually set your project ID, zone, and region.

Using gcloud init

If you receive the error One of [--zone, --region] must be supplied: Please specify location, complete this section.

1. Run gcloud init and follow the directions:

   gcloud init

   If you are using SSH on a remote server, use the --console-only flag to prevent the command from launching a browser:

   gcloud init --console-only

2. Follow the instructions to authorize gcloud to use your Google Cloud account.
3. Create a new configuration or select an existing one.
4. Choose a Google Cloud project.
5. Choose a default Compute Engine zone for zonal clusters or a region for regional or Autopilot clusters.

Note: You can override these default settings in gcloud commands using the --project, --zone, and --region flags.

Using gcloud config

• Set your default project ID:

  gcloud config set project PROJECT_ID

• If you are working with zonal clusters, set your default compute zone:

  gcloud config set compute/zone COMPUTE_ZONE

• If you are working with Autopilot or regional clusters, set your default compute region:

  gcloud config set compute/region COMPUTE_REGION

• Update gcloud to the latest version:

  gcloud components update

Note: You can override these default settings in gcloud commands using the --project, --zone, and --region flags.

Save your data to persistent disks

Before upgrading a node pool, you must ensure that any data you wish to keep is stored in a Pod using persistent volumes which use persistent disks. Persistent disks are unmounted, rather than erased, during upgrades, and their data is "handed off" between Pods.

The following restrictions pertain to persistent disks:

• The nodes on which Pods are running must be Compute Engine VMs
• Those VMs need to be in the same Compute Engine project and zone as the persistent disk

To learn how to add a persistent disk to an existing node instance, see Adding or resizing zonal persistent disks in the Compute Engine documentation.

About upgrading clusters

A cluster's control plane and nodes are upgraded separately.

Limitations

Alpha clusters cannot be upgraded.

Supported versions

The release notes announce when new versions become available and when older versions are no longer available. At any time, you can list all supported cluster and node versions using this command:

gcloud container get-server-config

Known issues

If you have Anthos Service Mesh or OSS Istio installed on your cluster, depending on your PodDisruptionBudget settings for the Istio components, user nodes might fail to upgrade to the control plane version after repeated attempts. To prevent this failure, we recommend that you increase the Horizontal Pod Autoscaling minReplicas setting from 1 to 2 for the components in the istio-system namespace before you upgrade. This will ensure that you always have an instance of the ASM control plane running.

If you have Anthos Service Mesh 1.5+ or OSS Istio 1.5+:

kubectl patch hpa -n istio-system istio-ingressgateway -p '{"spec":{"minReplicas": 2}}' --type=merge

kubectl patch hpa -n istio-system istiod -p '{"spec":{"minReplicas": 2}}' --type=merge

If you have Anthos Service Mesh 1.4.x or OSS Istio 1.4.x:

kubectl patch hpa -n istio-system istio-galley -p '{"spec":{"minReplicas": 2}}' --type=merge

kubectl patch hpa -n istio-system istio-ingressgateway -p '{"spec":{"minReplicas": 2}}' --type=merge

kubectl patch hpa -n istio-system istio-nodeagent -p '{"spec":{"minReplicas": 2}}' --type=merge

kubectl patch hpa -n istio-system istio-pilot -p '{"spec":{"minReplicas": 2}}' --type=merge

kubectl patch hpa -n istio-system istio-sidecar-injector -p '{"spec":{"minReplicas": 2}}' --type=merge

Although automatic upgrades might encounter the issue, the automatic upgrade process forces the nodes to upgrade. However, the upgrade takes an extra hour for every node in the istio-system namespace that violates the PodDisruptionBudget.

Downgrading limitations

Downgrading a cluster is not recommended. Nodes can be downgraded to a patch version older than the cluster version. You cannot downgrade a cluster from one minor version to another. For example, if a cluster is running GKE 1.11.5, you can downgrade to 1.11.4 if it is still available, but you cannot downgrade to 1.10.9. Instead, an error like the following is generated:

ERROR: (gcloud.container.clusters.upgrade) ResponseError: code=400,
message=Master cannot be upgraded to "1.10.9-gke.9": specified version is not
newer than the current version.

Upgrading the cluster

Google upgrades clusters and nodes automatically. For more control over which auto-upgrades your cluster and its nodes receive, you can enroll it in a release channel.

To learn more about managing your cluster's GKE version, see Upgrades.

You can initiate a manual upgrade any time after a new version becomes available.

Manually upgrading the control plane

Note: You cannot upgrade your cluster more than one minor version at a time. For example, you can upgrade a cluster from version 1.12.x to 1.13.x, but not directly from 1.11.x to 1.13.x. For more information, refer to Versioning.

When initiating a cluster upgrade, you can't modify the cluster's configuration for several minutes, until the control plane is accessible again. If you need to prevent downtime during control plane upgrades, consider using a regional cluster.

You can manually upgrade your cluster using the Cloud Console or the gcloud command-line tool. After upgrading your cluster, you can upgrade its nodes. By default, nodes created using the Google Cloud Console have auto-upgrade enabled, so this happens automatically.

gcloud

To upgrade your cluster control plane's version, first run the following command to see the available versions:

gcloud container get-server-config

To upgrade to the default cluster version, run the following command:

gcloud container clusters upgrade CLUSTER_NAME --master

To upgrade to a specific version that is not the default, run the following command:

gcloud container clusters upgrade CLUSTER_NAME \
  --master --cluster-version VERSION

Note: If your cluster is subscribed to a release channel, the VERSION must be a valid version for the release channel.

Refer to the gcloud container clusters upgrade documentation.

Console

To manually update your cluster control plane, perform the following steps:

1. Visit the Google Kubernetes Engine menu in Google Cloud Console.

   Visit the Google Kubernetes Engine menu

2. Click the desired cluster name.

3. Under Cluster basics, click Upgrade Available next to Version.

4. Select the desired version, then click Save Changes.

Downgrading clusters

Note: Before attempting to downgrade a cluster, be sure you understand the related limitations.

To downgrade a cluster to a previous patch version, change the cluster control plane version with the following command:

gcloud container clusters upgrade CLUSTER_NAME \
  --master --cluster-version VERSION

Note: If your cluster is subscribed to a release channel, the VERSION must be a valid version for the release channel.

Disabling cluster auto-upgrades

It is not possible to disable cluster auto-upgrades. Although it is not recommended, you can disable node auto-upgrade.

Upgrading nodes

By default, a cluster's nodes have auto-upgrade enabled, and it is recommended that you do not disable it.

When a node pool is upgraded, you can configure surge upgrade settings to control how many nodes GKE upgrades at once as well as how disruptive the upgrade is to workloads. By default, GKE upgrades one node at a time.

While a node is being upgraded, GKE stops scheduling new Pods onto it, and attempts to schedule its running Pods onto other nodes. This is similar to other events that re-create the node, such as enabling or disabling a feature on the node pool.

Note: During automatic or manual node upgrades, PodDisruptionBudgets (PDBs) are respected for a maximum of 1 hour. If Pods running on a node cannot be scheduled onto new nodes within 1 hour, the upgrade is initiated, regardless.
For more information about what to expect during node termination in general, see the topic about Pods.

The upgrade is only complete when all nodes have been recreated and the cluster is in the desired state. When a newly-upgraded node registers with the control plane, GKE marks the node as schedulable.

New node instances run the desired Kubernetes version as well as:

• The node image
• The Docker daemon, if applicable
• kubelet
• kube-proxy

Manually upgrade a node

You can manually upgrade a node pool version to match the version of the control plane or to a previous version that is still available and is compatible with the control plane. The Kubernetes version and version skew support policy guarantees that control planes are compatible with nodes up to two minor versions older than the control plane. For example, Kubernetes 1.13 control planes are compatible with Kubernetes 1.11 nodes.

Note: Upgrading a node pool may disrupt workloads running in that node pool. To avoid this, you can create a new node pool with the desired version and migrate the workload. After migration, you can delete the old node pool.

You can manually upgrade your nodes to a version compatible with the control plane, using the Google Cloud Console or the gcloud command-line tool.

gcloud

The following command upgrades your nodes to the version that your control plane is running:

gcloud container clusters upgrade CLUSTER_NAME \
  --node-pool=NODE_POOL_NAME

Replace CLUSTER_NAME with the name of the cluster to be upgraded.

To specify a different version of GKE on nodes, use the optional --cluster-version flag:

gcloud container clusters upgrade CLUSTER_NAME \
  --node-pool=NODE_POOL_NAME \
  --cluster-version VERSION

Replace VERSION with the Kubernetes version to which the nodes are upgraded. For example, --cluster-version=1.7.2 or cluster-version=latest.

For more information about specifying versions, see Versioning.

For more information, refer to the gcloud container clusters upgrade documentation.

Console

To upgrade a node pool using Cloud Console, perform the following steps:

1. Visit the Google Kubernetes Engine menu in Cloud Console.

   Visit the Google Kubernetes Engine menu

2. Next to the cluster you want to edit, click Actions, then click Edit.

3. On the Cluster details page, click the Nodes tab.

4. In the Node Pools section, click the name of the node pool that you want to upgrade.

5. Click Edit.

6. Click Change under Node version.

7. Select the desired version from the Node version drop-down list, then click Change.

Note: It may take several minutes for the node version to change.

Downgrading nodes

Downgrading a node pool is not possible. Instead, create a new node pool at the desired version and migrate your workloads to it. Nodes with auto-upgrade enabled are upgraded to match the control plane version.

Checking node upgrade status

Beta

This product or feature is covered by the Pre-GA Offerings Terms of the Google Cloud Platform Terms of Service. Pre-GA products and features may have limited support, and changes to pre-GA products and features may not be compatible with other pre-GA versions. For more information, see the launch stage descriptions.

You can check the status of an upgrade using gcloud beta container operations.

Note: To use gcloud beta commands, you must configure gcloud to use the v1beta1 API.

To see a list of every running and completed operation in the cluster, run the following command:

gcloud beta container operations list

Each operation is assigned an operation ID and an operation type as well as start and end times, target cluster, and status. The list appears similar to the following example:

NAME                              TYPE                ZONE           TARGET              STATUS_MESSAGE  STATUS  START_TIME                      END_TIME
operation-1505407677851-8039e369  CREATE_CLUSTER      us-west1-a     my-cluster                          DONE    20xx-xx-xxT16:47:57.851933021Z  20xx-xx-xxT16:50:52.898305883Z
operation-1505500805136-e7c64af4  UPGRADE_CLUSTER     us-west1-a     my-cluster                          DONE    20xx-xx-xxT18:40:05.136739989Z  20xx-xx-xxT18:41:09.321483832Z
operation-1505500913918-5802c989  DELETE_CLUSTER      us-west1-a     my-cluster                          DONE    20xx-xx-xxT18:41:53.918825764Z  20xx-xx-xxT18:43:48.639506814Z

To get more information about a specific operation, specify the operation ID in the following command:

gcloud beta container operations describe OPERATION_ID

For example:

gcloud beta container operations describe operation-1507325726639-981f0ed6
endTime: '20xx-xx-xxT21:40:05.324124385Z'
name: operation-1507325726639-981f0ed6
operationType: UPGRADE_CLUSTER
selfLink: https://container.googleapis.com/v1/projects/.../kubernetes-engine/docs/zones/us-central1-a/operations/operation-1507325726639-981f0ed6
startTime: '20xx-xx-xxT21:35:26.639453776Z'
status: DONE
targetLink: https://container.googleapis.com/v1/projects/.../kubernetes-engine/docs/zones/us-central1-a/clusters/...
zone: us-central1-a

Canceling a node upgrade

Beta — gcloud container operations

This feature is covered by the Pre-GA Offerings Terms of the Google Cloud Platform Terms of Service. Pre-GA features may have limited support, and changes to pre-GA features may not be compatible with other pre-GA versions. For more information, see the launch stage descriptions.

You can cancel an upgrade at any time. When you cancel an upgrade:

• Nodes that have started the upgrade complete it.
• Nodes that have not started the upgrade do not upgrade.
• Nodes that have already successfully completed the upgrade are unaffected and are not rolled back.

1. Get the upgrade's operation ID using the following command:

   gcloud container operations list
   

2. Run the following command to cancel the upgrade:

   gcloud beta container operations cancel OPERATION_ID
   

Refer to the gcloud container operations cancel documentation.

Rolling back a node upgrade

You can roll back node pools that failed to upgrade, or whose upgrades were canceled, to their previous version of Kubernetes. You cannot roll back node pools once they have been successfully upgraded. Nodes that have not started an upgrade are unaffected.

To roll back an upgrade, run the following command:

gcloud container node-pools rollback NODE_POOL_NAME \
  --cluster CLUSTER_NAME

Replace the following:

• NODE_POOL_NAME: the name of the node pool to roll back.
• CLUSTER_NAME: the name of the cluster from which to roll back the node pool.

Refer to the gcloud container node-pools rollback documentation.

Changing surge upgrade parameters

Surge Upgrades allow you to change the number of nodes GKE upgrades at one time and the amount of disruption an upgrade makes on your workloads.

The max-surge-upgrade and max-unavailable-upgrade flags are defined for each node pool. For more information on chosing the right parameters, go to Determining your optimal surge configuration.

Note: All new node pools are automatically configured to use surge upgrades maxSurge=1 maxUnavailable=0. To tune your configuration, see Determining your optimal surge configuration.

You can change these settings when creating or updating a cluster or node pool.

The following variables are used in the commands mentioned below:

• CLUSTER_NAME: the name of the cluster for the node pool.
• COMPUTE_ZONE: the zone for the cluster.
• NODE_POOL_NAME: the name of the node pool.
• NUMBER_NODES: the number of nodes in the node pool in each of the cluster's zones.
• SURGE_NODES: the number of extra (surge) nodes to be created on each upgrade of the node pool.
• UNAVAILABLE_NODES: the number of nodes that can be unavailable at the same time on each upgrade of the node pool.

Creating a cluster with specific surge parameters

To create a cluster with specific settings for surge upgrades, use the max-surge-upgrade and max-unavailable-upgrade flags.

gcloud container clusters create CLUSTER_NAME \
  --max-surge-upgrade=SURGE_NODES --max-unavailable-upgrade=UNAVAILABLE_NODES

Creating a cluster with surge upgrade disabled

To create a cluster without surge upgrades, set the value for the max-surge-upgrade flag to 0.

gcloud container clusters create CLUSTER_NAME \
  --max-surge-upgrade=0 --max-unavailable-upgrade=1

Creating a node pool with specific surge parameters

To create a node pool in an existing cluster with specific settings for surge upgrades, use the max-surge-upgrade and max-unavailable-upgrade flags.

gcloud container node-pools create NODE_POOL_NAME \
  --num-nodes=NUMBER_NODES --cluster=CLUSTER_NAME \
  --max-surge-upgrade=SURGE_NODES --max-unavailable-upgrade=UNAVAILABLE_NODES

Turn on or turn off Surge Upgrade for an existing node pool

To update the upgrade settings of an existing node pool, use the max-surge-upgrade and max-unavailable-upgrade flags. If you set max-surge-upgrade to greater than 0, GKE creates surge nodes. If you set max-surge-upgrade to 0, GKE doesn't create surge nodes.

gcloud beta container node-pools update NODE_POOL_NAME \
  --cluster=CLUSTER_NAME \
  --max-surge-upgrade=SURGE_NODES --max-unavailable-upgrade=UNAVAILABLE_NODES

Checking if surge upgrades are enabled on a node pool

To see if surge upgrades are enabled on a node pool, use gcloud to describe the cluster's parameters:

gcloud container node-pools describe NODE_POOL_NAME \
--cluster=CLUSTER_NAME

Note: While recreating nodes does not require additional Compute Engine resources, surge upgrading nodes does. Resource allocation is subjected to Compute Engine quota. Depending on your configuration, this quota can limit the number of parallel upgrades or even cause the upgrade to fail.

For more information about quota, go to Node upgrades and quota.

What's next

• Learn about Cluster architecture.
• Learn about Release channels.