By default, Google Kubernetes Engine clusters and node pools are upgraded automatically by Google. This page explains how to manually request an upgrade or downgrade for a GKE cluster or its nodes. You can learn more about how automatic and manual cluster upgrades work. You can also control when auto-upgrades can and cannot occur by configuring maintenance windows and exclusions.
New versions of GKE are announced regularly. See Versioning and Upgrades to learn about available versions. To learn more about clusters, see Cluster Architecture.
Before you begin
To prepare for this task, perform the following steps:
- Ensure that you have enabled the Google Kubernetes Engine API. Enable Google Kubernetes Engine API
- Ensure that you have installed the Cloud SDK.
- Set your default project ID:
gcloud config set project [PROJECT_ID]
- If you are working with zonal clusters, set your default compute zone:
gcloud config set compute/zone [COMPUTE_ZONE]
- If you are working with regional clusters, set your default compute region:
gcloud config set compute/region [COMPUTE_REGION]
- Update
gcloud
to the latest version:gcloud components update
Save your data to persistent disks
Before upgrading a node pool, you must ensure that any data you wish to keep is stored in a Pod using persistent volumes which use persistent disks. Persistent disks are unmounted, rather than erased, during upgrades, and their data is "handed off" between Pods.
The following restrictions pertain to persistent disks:
- The nodes on which Pods are running must be Compute Engine VMs
- Those VMs need to be in the same Compute Engine project and zone as the persistent disk
To learn how to add a persistent disk to an existing node instance, see Adding or Resizing Persistent Disks.
About upgrading clusters
A cluster's master and nodes are upgraded separately.
Limitations
Clusters enrolled in release channels, and their nodes, cannot be manually upgraded or downgraded. They are always upgraded when a new version is available in their enrolled release channel.
Alpha clusters cannot be upgraded.
Supported versions
The release notes announce when new versions become available and when older versions are no longer available. At any time, you can list all supported cluster and node versions using this command:
gcloud container get-server-config
Downgrading limitations
Downgrading a cluster is not recommended. Nodes can be downgraded to a patch version older than the cluster version. You cannot downgrade a cluster from one minor version to another. For example, if a cluster is running GKE 1.11.5, you can downgrade to 1.11.4 if it is still available, but you cannot downgrade to 1.10.9. Instead, an error like the following is generated:
ERROR: (gcloud.container.clusters.upgrade) ResponseError: code=400,
message=Master cannot be upgraded to "1.10.9-gke.9": specified version is not
newer than the current version.
Upgrading the cluster
Google upgrades clusters and nodes automatically. For more control over which auto-upgrades your cluster and its nodes receive, you can enroll it in a release channel.
To learn more about managing your cluster's GKE version, see Versioning and upgrades.
You can initiate a manual upgrade any time after a new version becomes available.
Manually upgrading the master
When initiating a cluster upgrade, you can't modify the cluster's configuration for several minutes, until the control plane is accessible again. If you need to prevent downtime during master upgrades, consider using a regional cluster
You can manually upgrade your cluster using the Cloud Console or the
gcloud
command-line tool. After upgrading your cluster, you can
upgrade its nodes. By default, nodes created using the
Google Cloud Console have auto-upgrade enabled, so this happens automatically.
It is not possible (or necessary) to manually upgrade a cluster enrolled in a release channel.
gcloud
To upgrade your cluster master's version, first run the following command to see the available versions:
gcloud container get-server-config
To upgrade to the latest version, run the following command:
gcloud container clusters upgrade [CLUSTER_NAME] --master
To upgrade to a specific version that is not the default, run the following command:
gcloud container clusters upgrade [CLUSTER_NAME] --master --cluster-version [CLUSTER_VERSION]
Refer to the gcloud container clusters upgrade
documentation.
Console
To manually update your cluster master, perform the following steps:
Visit the Google Kubernetes Engine menu in Google Cloud Console.
Select the desired cluster.
Click the Upgrade available link next to Master version.
Select the desired version, then click Change.
Click the arrow at the top of the screen to go back to the cluster overview page.
Downgrading clusters
To downgrade a cluster to a previous patch version, update the cluster master
version using the gcloud
command line tool.
Disabling cluster auto-upgrades
It is not possible to disable cluster upgrades. Although it is not recommended, you can disable node auto-upgrade.
Upgrading nodes
By default, a cluster's nodes have auto-upgrade enabled, and it is recommended that you do not disable it.
When a node pool is upgraded, GKE upgrades one node at a time.
While a node is being upgraded, GKE stops scheduling new Pods onto it, and attempts to schedule its running Pods onto other nodes. The node is then recreated at the new version, but using the same name as before. This is similar to other events that re-create the node, such as enabling or disabling a feature on the node pool.
It is not possible (or necessary) to manually upgrade a node whose cluster is enrolled in a release channel. When you use release channels, the cluster and its nodes use the same version, except during the brief span of time between auto-upgrading the cluster and its nodes.
The upgrade is only complete when all nodes have been recreated and the cluster is in the desired state. When a newly-upgraded node registers with the master, GKE marks the node as schedulable.
New node instances run the desired Kubernetes version as well as:
- The node image
- The Docker daemon, if applicable
kubelet
kube-proxy
Manually upgrade a node
You can manually upgrade a node pool version to match the version of the master or to a previous version that is still available and is compatible with the master. The Kubernetes version and version skew support policy guarantees that masters are compatible with nodes up to two minor versions older than the master. For example, Kubernetes 1.13 masters are compatible with Kubernetes 1.11 nodes.
You can manually upgrade your nodes to a version compatible with the master, using
the Google Cloud Console or the gcloud
command-line tool.
It is not possible (or necessary) to manually upgrade a node whose cluster is enrolled in a release channel. When you use release channels, the cluster and its nodes use the same version, except during the brief span of time between auto-upgrading the cluster and its nodes.
gcloud
The following command upgrades your nodes to the version that your master is running:
gcloud container clusters upgrade [CLUSTER_NAME] --node-pool=[NODE-POOL-NAME]
where [CLUSTER_NAME] is the name of the cluster to be upgraded.
To specify a different version of GKE on nodes, use the
optional --cluster-version
flag:
gcloud container clusters upgrade [CLUSTER_NAME] --node-pool=[NODE-POOL-NAME] --cluster-version [CLUSTER_VERSION]
where [CLUSTER_VERSION] is the Kubernetes version to which the nodes are
upgraded. For example, --cluster-version=1.7.2
or cluster-version=latest
.
For more information about specifying versions, refer to the Versioning and Upgrades documentation.
For more information, refer to the
gcloud container clusters upgrade
documentation.
Console
To upgrade a node pool using Cloud Console, perform the following steps:
Visit the Google Kubernetes Engine menu in Cloud Console.
Click the cluster's Edit button, which looks like a pencil.
In the Node Pools section, expand the disclosure arrow for the node pool you want to upgrade.
Click the Change link next to Node version.
Select the desired version, then click Change.
Click Save.
Downgrading nodes
Downgrading a node pool is not possible. Instead, create a new node pool at the desired version and migrate your workloads to it. Nodes with auto-upgrade enabled are upgraded to match the master version.
Checking node upgrade status
You can check the status of an upgrade using gcloud beta container operations
.
To see a list of every running and completed operation in the cluster, run the following command:
gcloud beta container operations list
Each operation is assigned an operation ID and an operation type as well as start and end times, target cluster, and status. The list appears similar to the following example:
NAME TYPE ZONE TARGET STATUS_MESSAGE STATUS START_TIME END_TIME
operation-1505407677851-8039e369 CREATE_CLUSTER us-west1-a my-cluster DONE 20xx-xx-xxT16:47:57.851933021Z 20xx-xx-xxT16:50:52.898305883Z
operation-1505500805136-e7c64af4 UPGRADE_CLUSTER us-west1-a my-cluster DONE 20xx-xx-xxT18:40:05.136739989Z 20xx-xx-xxT18:41:09.321483832Z
operation-1505500913918-5802c989 DELETE_CLUSTER us-west1-a my-cluster DONE 20xx-xx-xxT18:41:53.918825764Z 20xx-xx-xxT18:43:48.639506814Z
To get more information about a specific operation, specify the operation ID in the following command:
gcloud beta container operations describe [OPERATION_ID]
For example:
gcloud beta container operations describe operation-1507325726639-981f0ed6 endTime: '20xx-xx-xxT21:40:05.324124385Z' name: operation-1507325726639-981f0ed6 operationType: UPGRADE_CLUSTER selfLink: https://container.googleapis.com/v1/projects/.../kubernetes-engine/docs/zones/us-central1-a/operations/operation-1507325726639-981f0ed6 startTime: '20xx-xx-xxT21:35:26.639453776Z' status: DONE targetLink: https://container.googleapis.com/v1/projects/.../kubernetes-engine/docs/zones/us-central1-a/clusters/... zone: us-central1-a
Canceling a node upgrade
You can cancel an upgrade at any time. When you cancel an upgrade:
- Nodes that have started the upgrade complete it
- Nodes that have not started the upgrade do not upgrade
- Nodes that have already successfully completed the upgrade are unaffected and are not rolled back
First, get the upgrade's operation ID using the following command:
gcloud container operations list
Then, to cancel the upgrade, run the following command:
gcloud beta container operations cancel [OPERATION_ID]
Refer to the
gcloud container operations cancel
documentation.
Rolling back a node upgrade
You can roll back node pools that failed to upgrade, or whose upgrades were canceled, to their previous version of Kubernetes. You cannot roll back node pools once they have been successfully upgraded. Nodes that have not started an upgrade are unaffected.
To roll back an upgrade, run the following command:
gcloud container node-pools rollback [POOL_NAME] --cluster [CLUSTER_NAME]
where:
- [POOL_NAME] is the name of the node pool to roll back
- [CLUSTER_NAME] is the name of the cluster from which to roll back the node pool
- [COMPUTE_ZONE] is your project's compute zone
Refer to the gcloud container node-pools rollback
documentation.
Changing surge upgrade parameters
Surge Upgrades allow you to change the number of nodes GKE upgrades at one time and the amount of disruption an upgrade makes on your workloads.
The
max-surge-upgrade
and max-unavailable-upgrade
flags are defined for
each node pool.
Creating a cluster with different surge parameters
To create a cluster with different settings for surge upgrades, use the max-surge-upgrade and max-unavailable-upgrade flags.gcloud beta container clusters create [CLUSTER_NAME] \ --max-surge-upgrade=[SURGE_NODES] --max-unavailable-upgrade=[UNAVAILABLE_NODES]
Creating a cluster with surge upgrade disabled
To create a cluster without surge upgrades, set the max-surge-upgrade flag to 0.gcloud beta container clusters create [CLUSTER_NAME] \ --max-surge-upgrade=0 --max-unavailable-upgrade=1
Creating a node pool with different surge parameters
To create a node pool in an existing cluster with different settings for surge upgrades, use the max-surge-upgrade and max-unavailable-upgrade flags.gcloud beta container node-pools create [NODE_POOL_NAME] \ --num-nodes=[NUMBER_NODES] --cluster=[CLUSTER_NAME] \ --max-surge-upgrade=[SURGE_NODES] --max-unavailable-upgrade=[UNAVAILABLE_NODES]
Checking if surge upgrades are enabled on a node pool
To see if surge upgrades are enabled on a node pool, use gcloud to describe the cluster's parameters.gcloud beta container node-pools describe [NODE_POOL_NAME] \ --cluster=[CLUSTER_NAME]