Ensure resources for node upgrades

This page explains how to ensure that you can use node upgrade strategies to make changes to your cluster's nodes. We recommend verifying whether your project has enough resources to update your Google Kubernetes Engine (GKE) Standard node pools depending on the strategy that you choose.

Required resources for node upgrades

Node upgrades might require GKE to temporarily create additional nodes, depending on which node strategy and configuration that you chose. Because of this, GKE might require additional resources for a node pool upgrade.

All nodes that GKE creates, including nodes used for upgrades, are subject to the resource quota of your project, resource availability, and reservation capacity, for node pools with specific reservation affinity.

We strongly recommend verifying you have adequate resources if you have special machine type requirements and capacity constraints. You might need to configure a strategy that requires less resources to ensure that you can upgrade your nodes, or take other action such as creating a reservation. To learn more, see Upgrade in a resource-constrained environment.

Resources needed for different upgrade strategies

The following sections describe the additional resources are required to use the available upgrade strategies. If you're experiencing upgrade issues, check upgrade operations and follow the instructions to resolve upgrade errors.

Resources for surge upgrades

Surge upgrades are the default node upgrade strategy. As surge upgrades create extra VMs—if maxSurge is set to more than zero—before draining and deleting old nodes, upgrades can fail if your project does not have enough resources.

If the available resources are less than the number of nodes specified in maxSurge, then the number of parallel upgrades is less than maxSurge plus maxUnavailable. If you have no additional capacity, see how to upgrade in a resource-constrained environment.

The following table describes examples to demonstrate different upgrade behaviors:

Upgrade Settings Number of additional nodes allowed by available resources Result
maxSurge: 5 maxUnavailable: 0 5 Upgrades 5 nodes in parallel.
maxSurge: 5 maxUnavailable: 0 2 Upgrades only 2 nodes in parallel.
maxSurge: 5 maxUnavailable: 0 0 Upgrade fails, since it's not possible to bring up additional nodes and restarting existing ones is prohibited by the upgrade settings. GKE continues to try automatically upgrading until the operation succeeds.
maxSurge: 5 maxUnavailable: 1 5 Upgrades 6 nodes in parallel, while ensuring the node pool temporarily loses only one node due to upgrade.
maxSurge: 5 maxUnavailable: 1 2 Upgrades only 3 nodes in parallel, while ensuring the node pool is no more than one node short due to upgrade.
maxSurge: 5 maxUnavailable: 1 0 Upgrades only 1 node at a time by recreating each node with a rolling upgrade method.

Resources for blue-green upgrades

Blue-green upgrades temporarily double the number of nodes in a node pool, which means that your node pool temporarily needs twice as many resources. If you don't have enough resources available and you configured your node pool to use this strategy, configure surge upgrades instead if you need to unblock upgrades. When configuring this strategy, ensure that you configure surge upgrade parameters that'll work with the available resources for the node pool's nodes. To learn more, see the previous section, Resources for surge upgrades.

Check upgrade operations for resource-related failures

You can find more information about your upgrade operations as well as information about if and why any upgrades failed by checking your upgrade operation objects. To list the upgrade operation objects, run the following command:

gcloud container operations list \
    --filter="STATUS=DONE AND TYPE=UPGRADE_NODES AND targetLink:https://container.googleapis.com/v1/projects/PROJECT_ID/zones/COMPUTE_ZONE/clusters/CLUSTER_NAME"

Replace the following:

  • PROJECT_ID: the ID of your Google Cloud project.
  • COMPUTE_ZONE: the Compute Engine zone for your cluster.
  • CLUSTER_NAME: the name of your cluster.

If your most recent upgrade failed due to insufficient resource quota, the output is similar to the following:

gcloud container operations describe operation-1234567891234-1abc2d3e
detail: "Insufficient quota to satisfy the request: waiting on IG: instance https://www.googleapis.com/compute/v1/projects/my-project-123/zones/us-central1-a/instances/gke-my-cluster-default-pool-123ab45c-de67\
  \ is still CREATING. Last attempt errors: [QUOTA_EXCEEDED] Instance 'gke-my-cluster-default-pool-123ab45c-de67'\
  \ creation failed: Quota 'IN_USE_ADDRESSES' exceeded.  Limit: 50.0 in region us-central1.\
...

If the reason was due to insufficient reservation, the output is similar to the following:

gcloud container operations describe operation-1234567891234-1abc2d3e
detail: "Reservation does not have enough resources for the request: waiting on IG:\
  \ instance https://www.googleapis.com/compute/v1/projects/my-project-123/zones/us-central1-a/instances/gke-my-cluster-default-pool-123ab45c-de67\
  \ is still CREATING. Last attempt error: [CONDITION_NOT_MET] Instance 'gke-my-cluster-default-pool-123ab45c-de67'\
  \ creation failed: Specified reservation 'foo' does not have available resources\
  \ for the request."

If the reason was due to insufficient capacity, you'll see resource availability error messages.

Resolving upgrade errors

If your upgrade failed due to insufficient resources, try these steps:

  1. Consider a less resource-intensive upgrade strategy. If you're using blue-green upgrades, consider surge upgrades.
  2. Check if there are any Compute Engine resources in your project that are consuming resources and are no longer needed. If you find any, remove them and retry the upgrade.
  3. If the first two steps can't unblock your upgrade, request a quota increase or increase the size of the specific reservation.
  4. If you're operating in a resource-constrained environment with, for example, special machine type requirements and capacity constraints, see the next section for steps to unblock upgrades.

Upgrade in a resource-constrained environment

If Google Cloud has limited capacity for a specific resource (for example, GPUs, or TPUs) that your node pool is using, we recommend that you use a reservation to ensure that you have adequate resources. If you don't have adequate resources, you might not be able to upgrade your nodes until you can access more capacity, if you don't have a reservation for your existing capacity.

If you can't get any additional capacity, you can't use blue-green upgrades. For surge upgrades, maxSurge requires additional nodes. With, maxUnavailable, if the capacity isn't from a reservation, GKE temporarily releases the capacity of the node when upgrading it, meaning that if there is limited capacity, you risk losing the capacity.

Use the following guidance, depending on whether you have a reservation for your node pool that needs to be upgraded.

Upgrade with limited capacity without a reservation

If you don't have a reservation for your node pool's nodes, we recommend creating a reservation. However, if you can't, we recommend using surge upgrades with the default configuration of maxSurge=1;maxUnavailable=0. With this configuration, you don't risk losing existing capacity. However, surge upgrades only creates a surge node if the capacity is available. So, if no capacity is available, upgrades might take longer and your node pool could remain in a partially upgraded state until capacity is secured to upgrade each node.

We recommend that you create a reservation or increase the size of an existing reservation.

Upgrade with limited capacity with a reservation

If you have a reservation for your node pool's nodes, you can upgrade your node pools in a capacity-constrained environment, tuning surge upgrade settings for speed and reliability.

If you have additional capacity in the reservation, you can use maxSurge to create surge nodes. Set maxSurge depending on how many nodes you want to disrupt at one time, and how much additional capacity you have. For example, set maxSurge=1;maxUnavailable=0 if you only want to disrupt one node at a time, or if you only have capacity to create one additional node.

If you don't have any additional capacity, you can use maxUnavailable with a configuration such as maxSurge=0;maxUnavailable=1, as the capacity is reserved. This configuration disrupts one node a time, recreating the node to update it.

What's next