Problem
You notice that the API server responds to the client with higher latency than normal. If there is heavy usage of Custom Resource in the cluster, the response time related to Custom Resource requests exhibits significantly higher latency than the other API server requests. You may also observe disk throttling with the ETCD resource usage on the Control Plane side.
Environment
- Google Kubernetes Engine
Solution
- Upgrade the stored API versions which have been deprecated. Follow the Kubernetes documentation about upgrading the existing objects to a new stored version.
Cause
The root cause of the issue is that the Custom Resources in the cluster are operating with a deprecated API version. The API server logs for workloads resources will show a lot of conversion Webhook calls. This indicates that requests seem to target a specific deprecated API version for example v1alpha2.