View Horizontal Pod Autoscaler events

Autopilot Standard

---

This page provides information about decision events emitted by the Horizontal Pod Autoscaler in Google Kubernetes Engine (GKE). By analyzing these events, you can both gain insights into how the Horizontal Pod Autoscaler controller manages your workload scaling and understand the decision-making process behind its actions.

The Horizontal Pod Autoscaler emits decision events, which are stored as log entries in Cloud Logging.

Before you begin

Make sure to complete the following prerequisites:

Select or create a project

You can use an existing project, or create a new one for this tutorial.

1. Sign in to your Google Cloud account. If you're new to Google Cloud, create an account to evaluate how our products perform in real-world scenarios. New customers also get $300 in free credits to run, test, and deploy workloads.

2. In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

   Note: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project.

   Go to project selector

3. Make sure that billing is enabled for your Google Cloud project.

4. In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

   Note: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project.

   Go to project selector

5. Make sure that billing is enabled for your Google Cloud project.

Enable the APIs

Enable the GKE, and Cloud Logging APIs.

Enable the APIs

Set up Cloud Shell

In this tutorial, you use Cloud Shell to run gcloud and kubectl commands. Cloud Shell is a shell environment for managing resources hosted on Google Cloud. It comes preinstalled with the Google Cloud CLI and kubectl command-line tool.

In the Google Cloud console, activate Cloud Shell.

Activate Cloud Shell

A Cloud Shell session opens inside a frame lower on the console.

Before you run commands in this tutorial, make sure that your default project is set to the project ID where you want to deploy the sample app. If it's not already set, run the following command in the Cloud Shell:

gcloud config set project PROJECT_ID

Replace PROJECT_ID with your project ID.

Required roles and permissions

To get the permissions that you need to enable log generation, and to access and process logs, ask your administrator to grant you the following IAM roles on the project:

• To enable Horizontal Pod Autoscaler event logging in your cluster: Kubernetes Engine Cluster Admin (roles/container.clusterAdmin)
• To access logs and to use Logs Explorer and Log Analytics: Logs Viewer (roles/logging.viewer)

For more information about granting roles, see Manage access to projects, folders, and organizations.

You might also be able to get the required permissions through custom roles or other predefined roles.

Requirements

• Your GKE cluster must be running version 1.31.5-gke.1090000 or later, or 1.32.1-gke.1260000 or later.
• Enable Cloud Logging in your GKE cluster. The Cloud Logging pricing applies.

Enable Horizontal Pod Autoscaler decision events

Caution: Existing clusters might have specific logging components already configured. When you update an existing cluster using the Google Cloud CLI, or an API call, you must include the existing logging configuration in your update request to avoid overwriting the existing configurations.

Note: If you choose to send logs to Cloud Logging, you must send system logs. Make sure to enable the system logs component when you enable logs from additional sources, such as KCP_HPA.

To create a new cluster with the KCP_HPA decision logs enabled, run the following command:

gcloud container clusters create CLUSTER_NAME \
    --location=LOCATION \
    --project=PROJECT_ID \
    --logging=SYSTEM,KCP_HPA

To enable the KCP_HPA decision logs on an existing cluster, run the following command:

gcloud container clusters update CLUSTER_NAME \
    --location=LOCATION \
    --project=PROJECT_ID \
    --logging=SYSTEM,KCP_HPA

Replace the following:

• CLUSTER_NAME: name of the cluster you want to create or update.
• PROJECT_ID: your Google Cloud project ID.
• LOCATION: Compute regions or zones for your cluster.

These commands enable exporting the logs generated by KCP_HPA and save them at the destination logName="projects/PROJECT_ID/logs/container.googleapis.com%2Fhpa-controller" within Cloud Logging.

Retrieve the updated logging configuration of the cluster and review the list of logs to make sure that the KCP_HPA log is enabled:

gcloud container clusters describe CLUSTER_NAME \
    --location=LOCATION \
    --flatten=loggingConfig \
    --format='csv[delimiter=",",no-heading](componentConfig.enableComponents)'

The output is similar to the following:

SYSTEM_COMPONENTS,APISERVER,CONTROLLER_MANAGER,SCHEDULER,KCP_HPA

Disable Horizontal Pod Autoscaler decision events

Caution: Existing clusters might have specific logging components already configured. When you update an existing cluster using the Google Cloud CLI, or an API call, you must include the existing logging configuration in your update request to avoid overwriting the existing configurations.

Note: System logs are necessary for troubleshooting and support. If you disable System logs, GKE customer support is offered on a best-effort basis and might require additional effort from your engineering team.

Update a cluster to remove the KCP_HPA component from the --logging flag:

gcloud container clusters update CLUSTER_NAME \
    --location=LOCATION \
    --project=PROJECT_ID \
    --logging=SYSTEM

Replace the following:

• CLUSTER_NAME: name of the cluster you want to create or update.
• PROJECT_ID: your Google Cloud project ID.
• LOCATION: Compute regions or zones for your cluster.

This command disables exporting the logs generated by KCP_HPA. You can't retrieve them using the logName="projects/PROJECT_ID/logs/container.googleapis.com%2Fhpa-controller" filter within Cloud Logging.

Retrieve the updated logging configuration of the cluster and review the list of logs to make sure that the KCP_HPA log is disabled:

gcloud container clusters describe CLUSTER_NAME \
    --location=LOCATION \
    --flatten=loggingConfig \
    --format='csv[delimiter=",",no-heading](componentConfig.enableComponents)'

The output is similar to the following:

SYSTEM_COMPONENTS,APISERVER,CONTROLLER_MANAGER,SCHEDULER

Types of Logs

The decision events for the Horizontal Pod Autoscaler are stored in Cloud Logging, at the logName="projects/PROJECT_ID/logs/container.googleapis.com%2Fhpa-controller" location that's in the _Default bucket in the same project as your GKE cluster. All logged events are in the JSON format and can be found in the jsonPayload field of a log entry.

Make sure that you understand the storage needs of the potential log volumes, in addition to any performance or cost implications. The following examples explain how frequently The Horizontal Pod Autoscaler generates each type of decision event:

• Atomic recommendation: the Horizontal Pod Autoscaler generates one atomic recommendation event every 15 seconds for each metric being monitored by each HPA object in your cluster. For example, if you have two HPA objects in your cluster and each HPA object monitors three metrics, then six atomic recommendations will be logged every 15 seconds.

• Final recommendation: the Horizontal Pod Autoscaler generates one final recommendation event every 15 seconds for each HPA object in your cluster. For example, if you have two HPA objects in your cluster, then two final recommendations will be logged every 15 seconds.

In total, with two HPA objects that each monitor three metrics, your KCP_HPA log will receive eight total decision event entries every 15 seconds.

Atomic recommendation

An atomic recommendation log describes a recommendation based on an individual metric specified on your Horizontal Pod Autoscaler.

An atomic log includes the following fields:

Field	Description
start_time	Indicates when HPA started to compute a recommendation.
hpa	The name of the HPA object associated with the recommendation.
pod_count	Indicates the number of total Pods associated with the HPA when making the recommendation. This number also includes ready, unready, and ignored Pods.
metric	Provides information about the specification and status of the metric used for the recommendation. The metric field contains the following subfields: index: index of the metric in the Spec metrics array. type: metric type with values from the MetricSourceType (for example Resource, External). spec: name of the metric and a target set on that metric. status: Status conditions with respect to scalability and scaling limitations. newest_sample_time: timestamp of the freshest metric sample. newest_sample_age_seconds: age of the most recent sample, measured in seconds, since the recommendation computation began. A negative value signifies that the metrics sample predates the start of the computation.
summary	The summary field contains information regarding the recommendation result, including the suggested replica number. If a recommendation can't be proposed, an error message is displayed. The summary field contains the following subfields: dampening: HPA applies dampening to the recommendation and its direction to try and reduce the magnitude of a potential scale. Dampening can happen in the following ways: up: an up-dampening direction means HPA assumes missing-metrics Pods with 100% usage of the metric. down: a down-dampening direction means HPA assumes missing-metrics Pods or unready Pods that consume 0% usage of the metric. none: no dampening is applied. override: message that provides a reason when the recommendation proposed by HPA is not applied (for example, due to tolerance); or none if no override happens. result: result of the recommendation. It either proposes a recommended number of replicas or displays an error message if the recommendation can't be computed.

Example atomic recommendation log:

{
  "insertId": "xiu4bty9k5b279wu",
  "jsonPayload": {
    "instance": {
      "vm_name": "my-unique-vm-identifier",
      "zone": "us-central1-a"
    },
    "atomicRecommendation": {
      "startTime": "2025-02-06T20:07:00.573419526Z",
      "hpa": "gke-managed-cim/kube-state-metrics",
      "metric": {
        "newestSampleAgeSeconds": -39.573419526,
        "status": {
          "averageValue": "25849856"
        },
        "newestSampleTime": "2025-02-06T20:06:21Z",
        "type": "Resource",
        "spec": {
          "target": {
            "averageValue": "400Mi"
          },
          "name": "memory"
        }
      },
      "podCount": {
        "ready": 1,
        "total": 1
      },
      "summary": {
        "override": "none",
        "replicas": 1,
        "dampening": "none"
      }
    }
  },
  "resource": {
    "type": "k8s_control_plane_component",
    "labels": {
      "project_id": "my-project-id",
      "cluster_name": "my-cluster",
      "location": "us-central1-a",
      "component_location": "us-central1-a",
      "component_name": "hpa-controller"
    }
  },
  "timestamp": "2025-02-06T20:07:00.593777835Z",
  "severity": "INFO",
  "labels": {
    "compute.googleapis.com/resource_name": "my-unique-vm-identifier"
  },
  "logName": "projects/my-project-id/logs/container.googleapis.com%2Fhpa-controller",
  "sourceLocation": {
    "file": "event_logger.go",
    "line": "61"
  },
  "receiveTimestamp": "2025-02-06T20:07:05.284753647Z"
}

Final recommendation

A final recommendation log describes a consolidated recommendation proposed by Horizontal Pod Autoscaler. The Horizontal Pod Autoscaler combines all the atomic recommendations from different metrics to create a final recommendation and actuates the final recommendations. Actuation means that the HPA instructs the Deployment to adjust the number of replicas to match the recommended value. If the final recommendation suggests a different number of Pods than the number of running Pods, the Horizontal Pod Autoscaler triggers a scale-up or scale-down event to adjust the Deployment accordingly.

A final recommendation log includes the following fields:

Field	Description
start_time	Indicates when HPA started to compute a recommendation.
hpa	The name of the HPA object associated with the recommendation.
target_ref	Indicates the HPA ScaleTargetRef object associated with a recommendation.
configured_size	The last recorded number of replicas before HPA computes and applies this recommendation.
top_level_override	Provides a reason if the recommendation proposed by HPA is not applied (for example, due to tolerance); or none if no override happens.
top_level_limit	Provides a reason if the recommendation proposed by HPA needs to be adjusted (for example, due to the number of replicas that are defined by the MinReplicas or MaxReplicas fields in the HPA specification).
leading_metric_index	The leading metric index in the Spec metrics array is the metric whose associated atomic recommendation is used as the final recommendation.
normalization	Provides stabilization and limitation summary as follows, if present: stabilization: describes the stabilization state if it was applied. Stabilization is used to restrict the flapping of replica count when the metrics used for scaling keep fluctuating. stabilization field consists of the following subfields: replicas: replica number after stabilization. reason: applied stabilization type — scaleUp or scaleDown. stabilization_window: associated stabilization time period, in seconds. replicas_before_stabilization: recommended replica number before stabilization. limitation: describes how scaling limitations are handled if they are applied. This behavior modifies the recommendations proposed by the HPA based on the limitations in place. limitation field consists of the following subfields: replicas: replica number after limitation. reason: reason for not scaling beyond the min or max replica count. scaling_policy: the scaling policy that's applied. selectPolicy: specifies how a policy is selected when scaling in a certain direction. MaxChange corresponds to MaxChangePolicySelect; MinChange corresponds to MinChangePolicySelect. If scaling is disabled, the selectPolicy field won't be present. replicas_before_limitation: recommended replica number before limitation.
replicas	Recommended replica number.
actuation_error	An error message associated with the failure, if actuation failed.
actuation_time	Timestamp of the actuation on success.
actuation_latency_seconds	Time that passed, in seconds, from the start of the recommendation computation until the actuation succeeded.

Example final recommendation log:

{
  "insertId": "qzyv7alfv1sm19ns",
  "jsonPayload": {
    "finalRecommendation": {
      "actuationTime": "2025-02-06T20:06:57.487786873Z",
      "targetRef": {
        "name": "kube-state-metrics",
        "kind": "StatefulSet",
        "apiVersion": "apps/v1"
      },
      "topLevelLimit": "none",
      "hpa": "gke-managed-cim/kube-state-metrics",
      "topLevelOverride": "noRecommendation",
      "replicas": 1,
      "configuredSize": 1,
      "actuationLatencySeconds": 0.003722451,
      "startTime": "2025-02-06T20:06:57.484064422Z"
    },
    "instance": {
      "vm_name": "my-unique-vm-identifier",
      "zone": "us-central1-a"
    }
  },
  "resource": {
    "type": "k8s_control_plane_component",
    "labels": {
      "cluster_name": "my-cluster",
      "component_location": "us-central1-a",
      "component_name": "hpa-controller",
      "location": "us-central1-a",
      "project_id": "my-project-id"
    }
  },
  "timestamp": "2025-02-06T20:06:57.488193527Z",
  "severity": "INFO",
  "labels": {
    "compute.googleapis.com/resource_name": "my-unique-vm-identifier"
  },
  "logName": "projects/my-project-id/logs/container.googleapis.com%2Fhpa-controller",
  "sourceLocation": {
    "file": "event_logger.go",
    "line": "61"
  },
  "receiveTimestamp": "2025-02-06T20:06:57.844898727Z"
}

Troubleshooting

This section describes issues and resolution steps related to Horizontal Pod Autoscaler events.

No events

If you don't see any Horizontal Pod Autoscaler decision events, make sure you have done all of the following:

• You have enabled Cloud Logging for the cluster.
• You have enabled the KCP_HPA logs for the cluster.
• You have deployed at least one correctly configured hpa object to your cluster.

To view your hpa object configuration, run the following command:

  kubectl describe hpa $HPA_NAME

If you still don't see any KCP_HPA logs, reach out to Google Cloud Support.

What's next

• See About GKE logs.
• Understand your GKE logs.
• Learn how to find specific GKE logs using sample queries.
• Learn how to optimize Pod autoscaling based on metrics.