This page shows you how to diagnose and resolve issues that prevent cluster autoscaler from scaling down your Google Kubernetes Engine (GKE) nodes.
This page is for Application developers who want to resolve an unexpected or negative situation with their app or service and Platform admins and operators who want to prevent interruption to delivery of products and services.
Understand when cluster autoscaler scales down your nodes
Before you proceed to the troubleshooting steps, it can be helpful to understand when cluster autoscaler would try to scale down your nodes. It could be the case that cluster autoscaler didn't scale down because it didn't need to.
Cluster autoscaler determines if a node is underutilized and eligible for scale down by calculating a utilization factor. The utilization factor is calculated by dividing the vCPU and memory requested by the Pods on the node by the allocatable vCPU and memory on the node.
Every 10 seconds cluster autoscaler checks the utilization factor of your nodes
to see if it's below the required threshold. If you're using a balanced
autoscaling profile,
the utilization factor threshold is 0.5. If you're using the
optimize-utilization
profile, the utilization factor varies. When the
utilization factor is less than the required threshold for both vCPU and memory,
cluster autoscaler considers the node underutilized.
When a node is underutilized, cluster autoscaler marks the node for removal and monitors the node for the next 10 minutes to make sure the utilization factor stays under the required threshold. If the node is still underutilized after 10 minutes, cluster autoscaler should remove the node.
Example: Utilization factor calculation
You have a cluster with cluster autoscaler enabled and you're using the
balanced
autoscaling profile. A node on this cluster is provisioned with the
e2-standard-4
machine type, which offers 4 vCPUs and 16 GB of memory. A
Pod on this node requests 0.5 vCPU and 10 GB of memory, so cluster autoscaler
calculates the following utilization factors:
- vCPU utilization factor: 0.5 vCPU / 4 vCPUs = 0.125
- Memory utilization factor: 10 GB / 16 GB = 0.625
In this scenario, cluster autoscaler would not consider this node underutilized because the memory utilization factor (0.625) exceeds the threshold of 0.5. Even though the vCPU utilization is low, the higher memory usage prevents scale down to ensure sufficient resources remain available for the Pod's workload.
Check if the issue is caused by a limitation
If you observe a cluster with low utilization for more than 10 minutes and it's not scaling down, make sure your issue isn't caused by one of the limitations for the cluster autoscaler.
View errors
If your issue wasn't caused by a limitation, you can often diagnose the cause by viewing error messages:
If you've already seen an error message, see the error messages table for advice on resolving the error.
If you haven't seen a message yet, use one of the following options:
- Issues less than 72 hours old: View error notifications in the Google Cloud console.
- Issues over 72 hours old: View errors in events in Cloud Logging.
View errors in notifications
If the issue you observed happened less than 72 hours ago, view notifications about errors in the Google Cloud console. These notifications provide valuable insights into why cluster autoscaler didn't scale down and offer advice on how to resolve the error and view relevant logs for further investigation.
To view the notifications in the Google Cloud console, complete the following steps:
In the Google Cloud console, go to the Kubernetes clusters page.
Review the Notifications column. The following notifications are associated with scale down issues:
Can't scale down nodes
Scale down blocked by pod
Click the relevant notification to see a pane with details about what caused the issue and recommended actions to resolve it.
Optional: To view the logs for this event, click Logs. This action takes you to Logs Explorer with a pre-populated query to help you further investigate the scaling event. To learn more about how scale down events work, see View cluster autoscaler events.
If you're still experiencing issues after reviewing the advice in the notification, consult the error messages tables for further help.
View errors in events
If the issue you observed happened over 72 hours ago, view events in Cloud Logging. When there has been an error, it's often recorded in an event.
To view cluster autoscaler logs in the Google Cloud console, complete the following steps:
In the Google Cloud console, go to the Kubernetes clusters page.
Select the name of the cluster that you want to investigate to view its Cluster details page.
On the Cluster details page, click the Logs tab.
On the Logs tab, click the Autoscaler Logs tab to view the logs.
Optional: To apply more advanced filters to narrow the results, click the button with the arrow on the right side of the page to view the logs in Logs Explorer.
To learn more about scale down events work, see View cluster autoscaler events. For one example of how to use Cloud Logging, see the following troubleshooting example.
Example: Troubleshoot an issue over 72 hours old
The following example shows you how you might investigate and resolve an issue with a cluster not scaling down.
Scenario:
One week ago, you were looking at the GKE Enterprise dashboard and noticed that your cluster has only utilized only 10% of its CPU and memory. Despite the low utilization, cluster autoscaler didn't delete the node as you expected. When you look at the dashboard now, the issue seems to have been resolved, but you decide to find out what happened so that you can avoid it happening again.
Investigation:
- Because the issue happened over 72 hours ago, you investigate the issue using Cloud Logging instead of looking at the notification messages.
- In Cloud Logging, you find the logging details for cluster autoscaler events, as described in View errors in events.
- You search for
scaleDown
events that contain the nodes belonging to the cluster that you're investigating in thenodesToBeRemoved
field. You could filter the log entries, including filtering by a particular JSON field value. Learn more in Advanced logs queries. - You don't find any
scaleDown
events. However, if you did find ascaleDown
event, you could search for aneventResult
event that contains the associatedeventId
. You could then search for an error in theerrorMsg
field. You decide to continue your investigation by searching for
noScaleDown
events that have the node that you're investigating in the nodes field.You find a
noScaleDown
event that contains a reason for your node not scaling down. The message ID is"no.scale.down.node.pod.not.backed.by.controller"
and there's a single parameter:"test-single-pod"
.
Resolution:
You consult the error messages table, and discover this message
means that the Pod is blocking scale down because it's not backed by a
controller. You find out that one solution is to add a
"cluster-autoscaler.kubernetes.io/safe-to-evict": "true"
annotation to the
Pod. You investigate test-single-pod
and see that a colleague added the
annotation and after applying the annotation, cluster autoscaler scaled down the
cluster correctly. You decide to add the annotation to all other Pods where it's
safe to do so to avoid the issue from happening again.
Resolve scale down errors
After you have identified your error, use the following tables to help you understand what caused the error and how to resolve it.
ScaleDown errors
You can find error event messages for scaleDown
events in the corresponding
eventResult
event, in the resultInfo.results[].errorMsg
field.
Event message | Details | Parameter | Mitigation |
---|---|---|---|
"scale.down.error.failed.to.mark.to.be.deleted" |
A node couldn't be marked for deletion. | Failing node name. | This message should be transient. If it persists, contact Cloud Customer Care for further investigation. |
"scale.down.error.failed.to.evict.pods" |
Cluster autoscaler can't scale down because some of the Pods couldn't be evicted from a node. | Failing node name. | Review the PodDisruptionBudget for the Pod and make sure the rules allow for eviction of application replicas when acceptable. To learn more, see Specifying a Disruption Budget for your Application in the Kubernetes documentation. |
"scale.down.error.failed.to.delete.node.min.size.reached" |
Cluster autoscaler can't scale down because a node couldn't be deleted due to the cluster already being at minimal size. | Failing node name. | Review the minimum value set for node pool autoscaling and adjust the settings as necessary. To learn more, see the Error: Nodes in the cluster have reached minimum size. |
Reasons for a noScaleDown event
A noScaleDown
event is periodically emitted when there are nodes which are
blocked from being deleted by cluster autoscaler. noScaleDown
events are
best-effort, and don't cover all possible cases.
NoScaleDown top-level reasons
Top-level reason messages for noScaleDown
events appear in the
noDecisionStatus.noScaleDown.reason
field. The message contains a top-level
reason why cluster autoscaler can't scale the cluster down.
Event message | Details | Mitigation |
---|---|---|
"no.scale.down.in.backoff" |
Cluster autoscaler can't scale down because scaling down is in a backoff period (temporarily blocked). | This message should be transient, and can occur when there has been a recent scale up event. If the message persists, contact Cloud Customer Care for further investigation. |
"no.scale.down.in.progress" |
Cluster autoscaler can't scale down because a previous scale down was still in progress. |
This message should be transient, as the Pod will eventually be removed. If this message occurs frequently, review the termination grace period for the Pods blocking scale down. To speed up the resolution, you can also delete the Pod if it's no longer needed. |
NoScaleDown node-level reasons
Node-level reason messages for noScaleDown
events appear in the
noDecisionStatus.noScaleDown.nodes[].reason field
. The message contains a
reason why cluster autoscaler can't remove a particular node.
Event message | Details | Parameters | Mitigation |
---|---|---|---|
"no.scale.down.node.scale.down.disabled.annotation" |
Cluster autoscaler can't remove a node from the node pool because
the node is annotated with
cluster-autoscaler.kubernetes.io/scale-down-disabled: true .
|
N/A | Cluster autoscaler skips nodes with this annotation without considering their utilization and this message is logged regardless of the node's utilization factor. If you want cluster autoscaler to scale down these nodes, remove the annotation. |
"no.scale.down.node.node.group.min.size.reached" |
Cluster autoscaler can't scale down when node group size has exceeded minimum size limit. This happens because removing nodes would violate the cluster-wide minimal resource limits defined in your node auto-provisioning settings. |
N/A | Review the minimum value set for node pool autoscaling. If you want cluster autoscaler to scale down this node, adjust the minimum value. |
"no.scale.down.node.minimal.resource.limits.exceeded" |
Cluster autoscaler can't scale down nodes because it would violate cluster-wide minimal resource limits. These are the resource limits set for node auto-provisioning. |
N/A | Review your limits for memory and vCPU and, if you want cluster autoscaler to scale down this node, increase the limits. |
"no.scale.down.node.no.place.to.move.pods" |
Cluster autoscaler can't scale down because there's no place to move Pods. | N/A | If you expect that the Pod should be rescheduled, review the scheduling requirements of the Pods on the underutilized node to determine if they can be moved to another node in the cluster. To learn more, see the Error: No place to move Pods. |
"no.scale.down.node.pod.not.backed.by.controller" |
Pod is blocking scale down because it's not backed by a controller. Specifically, the cluster autoscaler is unable to scale down an underutilized node due to a Pod that lacks a recognized controller. Allowable controllers include ReplicationController, DaemonSet, Job, StatefulSet, or ReplicaSet. |
Name of the blocking Pod. | Set the annotation
"cluster-autoscaler.kubernetes.io/safe-to-evict": "true"
for the Pod or define an acceptable controller. |
"no.scale.down.node.pod.has.local.storage" |
Pod is blocking scale down because it has local storage. | Name of the blocking Pod. | Set an annotation
"cluster-autoscaler.kubernetes.io/safe-to-evict": "true"
for the Pod if the data in the local storage for the Pod is not critical.
This error only occurs for clusters using a version earlier than 1.22. |
"no.scale.down.node.pod.not.safe.to.evict.annotation" |
A Pod on the node has the safe-to-evict=false annotation. |
Name of the blocking Pod. | If the Pod can be safely evicted, edit the manifest of the Pod and
update the annotation to
"cluster-autoscaler.kubernetes.io/safe-to-evict": "true" . |
"no.scale.down.node.pod.kube.system.unmovable" |
Pod is blocking scale down because it's a non-DaemonSet,
non-mirrored, Pod without a PodDisruptionBudget in the
kube-system namespace. |
Name of the blocking Pod. | By default, Pods in the To resolve this issue, either add a PodDisruptionBudget
for the |
"no.scale.down.node.pod.not.enough.pdb" |
Pod is blocking scale down because it doesn't have enough PodDisruptionBudget. | Name of the blocking Pod. | Review the PodDisruptionBudget for the Pod and consider making it less restrictive. To learn more, see Error: Not enough PodDisruptionBudget. |
"no.scale.down.node.pod.controller.not.found" |
Pod is blocking scale down because its controller (for example, a Deployment or ReplicaSet) can't be found. | N/A | To determine what actions were taken that left the Pod running after its controller was removed, review the logs. To resolve this issue, manually delete the Pod. |
"no.scale.down.node.pod.unexpected.error" |
Pod is blocking scale down because of an unexpected error. | N/A | The root cause of this error is unknown. Contact Cloud Customer Care for further investigation. |
Conduct further investigation
The following sections provide guidance on how to use Logs Explorer and
gcpdiag
to gain additional insights into your errors.
Investigate errors in Logs Explorer
If you want to further investigate your error message, you can view logs specific to your error:
In the Google Cloud console, go to the Logs Explorer page.
In the query pane, enter the following query:
resource.type="k8s_cluster" log_id("container.googleapis.com/cluster-autoscaler-visibility") jsonPayload.resultInfo.results.errorMsg.messageId="ERROR_MESSAGE"
Replace
ERROR_MESSAGE
with the message that you want to investigate. For example,scale.down.error.failed.to.delete.node.min.size.reached
.Click Run query.
Debug some errors with gcpdiag
gcpdiag
is an open source tool created with support from Google Cloud
technical engineers. It isn't an officially supported Google Cloud product.
If you've experienced one of the following error messages, you can use
gcpdiag
to help troubleshoot the issue:
scale.down.error.failed.to.evict.pods
no.scale.down.node.node.group.min.size.reached
For a list and description of all gcpdiag
tool flags, see the
gcpdiag
usage instructions.
Resolve complex scale down errors
The following sections offer guidance on resolving errors where the mitigations involve multiple steps and errors that don't have a cluster autoscaler event message associated with them.
Error: Nodes in the cluster have reached minimum size
If you see the following errors, cluster autoscaler couldn't delete a node because the number of nodes in the cluster was already at the minimum size:
Notification
Scale down of underutilized node is blocked because cluster autoscaler minimal resource limits are reached.
Event
"scale.down.error.failed.to.delete.node.min.size.reached"
To resolve this issue, review and update the minimum limits for autoscaling:
In the Google Cloud console, go to the Kubernetes clusters page:
Click the name of the cluster identified in the notification or Cloud Logging.
On the Cluster details page, go to the Nodes tab.
Review the value in the Number of nodes column and compare it with the minimum number of nodes listed in the Autoscaling column. For example, if you see 4 - 6 nodes listed in the Autoscaling column, and the number of nodes in the node pool is 4, the number of node pools is already equal to the minimum size, so cluster autoscaler cannot scale down the nodes any further.
If the configuration is correct and the value for number of nodes is equal to the minimum defined for Autoscaling, cluster autoscaler is working as intended. If the minimum number of nodes is too high for your needs, reduce the minimum size so that the nodes can scale down.
Error: No place to move Pods
The following errors occur when cluster autoscaler tries to scale down a node but can't, because a Pod on that node can't be moved to another node:
Notification
Scale down of underutilized node is blocked because it has Pod which cannot be moved to another node in the cluster.
Event
"no.scale.down.node.no.place.to.move.pods"
If you don't want this Pod to be rescheduled, then this message is expected and
no changes are required. If you do want the Pod to be rescheduled, investigate
the following definitions in the pod.spec block
in the manifest of the Pod:
- NodeAffinity: Review the scheduling requirements of the Pods on the underutilized node. You can review these requirements by examining the Pod manifest and looking for any NodeAffinity or NodeSelector rules. If the Pod has a nodeSelector defined and there are no other nodes (from other nodes pools) in the cluster that match this selector, cluster autoscaler is unable to move the Pod to another node, which in turn prevents it from removing any underutilized nodes.
maxPodConstraint
: IfmaxPodConstraint
is configured to any other number other than the default number of 110, then confirm if this was an intended change. Lowering this value increases the likelihood of issues. Cluster autoscaler cannot reschedule Pods to other nodes, if all other nodes in the cluster have already reached the value defined inmaxPodConstraint
, leaving no space for new Pods to be scheduled. Increasing themaxPodConstraint
value allows more Pods to be scheduled on nodes and cluster autoscaler will have space to reschedule Pods and scale down underutilized nodes. When definingmaxPodConstraint
, keep in mind that there are approximately 10 system Pods on each node.hostPort
: Specifying ahostPort
for the Pod means only one Pod can run on that node. This can make it difficult for cluster autoscaler to reduce the number of nodes because the Pod might not be able to move to another node if that node's port is already in use. This is expected behavior.
Error: kube-system Pod unmoveable
The following errors occur when a system Pod is preventing scale down:
Notification
Pod is blocking scale down because it's a non-DaemonSet, non-mirrored, Pod without a PodDisruptionBudget in the
kube-system
namespace.
Event
"no.scale.down.node.pod.kube.system.unmovable"
A Pod in the kube-system
namespace is considered a system Pod. By default,
cluster autoscaler doesn't remove Pods in the kube-system
namespace.
To resolve this error, choose one of the following resolutions:
Add a
PodDisruptionBudget
for thekube-system
Pods. For more information about manually adding aPodDisruptionBudget
for thekube-system
Pods, see the Kubernetes cluster autoscaler FAQ.Creating a PodDisruptionBudget might affect the availability of system workloads which can cause downtime on the cluster. Cluster autoscaler reschedules these system workloads on different worker nodes during the scale down process.
Use a combination of node pools taints and tolerations to separate
kube-system
Pods from your application Pods. For more information, see node auto-provisioning in GKE.
Verify that nodes have kube-system
Pods
If you're not sure that your nodes are running kube-system
Pods, and want to
verify, complete the following steps:
Go to the Logs Explorer page in the Google Cloud console.
Click Query builder.
Use the following query to find all network policy log records:
- resource.labels.location="CLUSTER_LOCATION" resource.labels.cluster_name="CLUSTER_NAME" logName="projects/PROJECT_ID/logs/container.googleapis.com%2Fcluster-autoscaler-visibility" jsonPayload.noDecisionStatus.noScaleDown.nodes.node.mig.nodepool="NODE_POOL_NAME"
Replace the following:
CLUSTER_LOCATION
: The region your cluster is in.CLUSTER_NAME
: The name of your cluster.PROJECT_ID
: the ID of the project that your cluster belongs to.NODE_POOL_NAME
: The name of your node pool.
If there are
kube-system
Pods running on your node pool, the output includes the following:"no.scale.down.node.pod.kube.system.unmovable"
Error: Not enough PodDisruptionBudget
The following errors occur when your PodDisruptionBudget is preventing scale down:
Notification
Scale down of underutilized node is blocked because it has a Pod running on it which doesn't have enough Pod Disruption Budget to allow eviction of the Pod.
Event
NoScaleDownNodePodNotEnoughPdb: "no.scale.down.node.pod.not.enough.pdb"
To see if a PodDisruptionBudget is too restrictive, review its settings:
kubectl get pdb --all-namespaces
The output is similar to the following:
NAMESPACE NAME MIN AVAILABLE MAX UNAVAILABLE ALLOWED DISRUPTIONS AGE
example-app-one one_pdb N/A 1 1 12d
example-app-two two_pdb N/A 0 0 12d
In this example, any Pods matching the two_pdb
label selector won't be
evicted by cluster autoscaler. The maxUnavailable: 0
setting in this
PodDisruptionBudget dictates that all replicas must remain available at all
times. Additionally, disruptionsAllowed: 0
prohibits any disruptions to these
Pods. Consequently, nodes running these Pods cannot be scaled down, as doing so
would cause a disruption and violate the PodDisruptionBudget.
If your PodDisruptionBudget is working the way you want, no further action is
required. If you'd like to adjust your PodDisruptionBudget so that Pods on an
underutilized node can be moved, edit the manifest of the PodDisruptionBudget.
For example, if you had set maxUnavailable
to 0
, you could change it
to 1
so that cluster autoscaler can scale down.
Issue: Node stay in cordoned status and isn't removed
Errors similar to the following happen when cluster autoscaler can't reduce the node pool size because the Google service account doesn't have the Editor role:
Required 'compute.instanceGroups.update' permission for 'INSTANCE_GROUP_NAME'.
A common symptom of this issue is when cluster autoscaler tries to reduce the node pool size, but the node doesn't change status.
To resolve this issue, check if the default service account
(PROJECT_NUMBER@cloudservices.gserviceaccount.com
)
has the Editor role (roles/editor
) on the project. If the service account
doesn't have this role, add it. GKE uses this service
account to manage your project's resources. To learn how to do this, see
Grant or revoke a single role
in the IAM documentation.
What's next
Review the following questions in the Kubernetes cluster autoscaler FAQ:
Watch a YouTube video about troubleshooting and resolving scaling issues.
If you need additional assistance, reach out to Cloud Customer Care.