Problem
Horizontal Pod Autoscaling(HPA) is not scaling up/down as expected even if the resource utilization isn't meeting the current size of the workload.
The logs would show that the HPA is re-sizing the cluster as expected. However, lots of pods will be showing in a Terminating/Terminated state.
Environment
- Google Kubernetes Engine: prior to v1.22.x
Solution
Apply only one of the following solutions:
- Upgrade the cluster to Google Kubernetes Engine v1.22
- Manually remove the Failed/Terminated pods
- If the previous two solutions aren't suitable for you, use this script as a possible automated solution for the versions prior to 1.22.
Cause
When the node is preempted several pods are passed to the new node in a Terminated state and retained there in that state until manual removal or clean up is performed by the Garbage collector. In this case, HPA still considers these pods to be a healthy part of the deployment and will not scale up or down to fulfill the desired state based on the metrics.
The root cause of the issue is the Graceful node shutdown feature. When pods are evicted during the graceful node shutdown, they are marked as shutdown. Running kubectl get pods shows the status of the evicted pods as Terminated.