Horizontal pod autoscaling (HPA)

Stay organized with collections Save and categorize content based on your preferences.

Google Cloud Managed Service for Prometheus supports horizontal pod autoscaling (HPA) by using the popular prometheus-adapter library.

Existing prometheus-adapter configs can be used to autoscale with only a few changes. Configuring prometheus-adapter to scale using Managed Service for Prometheus has two additional restrictions compared to scaling using upstream Prometheus:

As data can take slightly longer to be available within Managed Service for Prometheus compared to upstream Prometheus, configuring overly-eager autoscaling logic might cause unwanted behavior. Although there is no guarantee on data freshness, data is typically available to query 3-7 seconds after it is sent to Managed Service for Prometheus, excluding any network latency.

All queries issued by prometheus-adapter are global in scope. This means that if you have applications in two namespaces that emit identically named metrics, an HPA configuration using that metric scales using data from both applications. We recommend always using namespace or cluster filters in your PromQL to avoid scaling using incorrect data.

Deploy an example HPA configuration

To set up an example HPA configuration using prometheus-adapter and managed collection, use the following steps:

  1. Set up managed collection in your cluster.
  2. Deploy the Prometheus frontend UI proxy in your cluster. If you use Workload Identity, you must also configure and authorize a service account.
  3. Install prometheus-adapter in your cluster using instructions from the prometheus-adapter deployment README. You might need to install an internal firewall rule on port 6443 from the control plane to your nodes. This is third-party code that is not installed, managed, or supported by Google; you have to set up prometheus-adapter yourself.
  4. Deploy the manifests in the examples/hpa/ directory within the prometheus-engine repo:
    • example-app.yaml: An example deployment and service that emits metrics.
    • pod-monitoring.yaml: A resource that configures scraping the example metrics.
    • hpa.yaml: The HPA resource that configures scaling for your workload.
    • cm.yaml: A ConfigMap that overrides the default prometheus-adapter adapter-config ConfigMap and queries the http_requests_per_second metric from the example deployment.
  5. Edit the custom-metrics-apiserver Deployment within the prometheus-adapter repo and change the prometheus-url value as follows: - --prometheus-url=http://frontend.default.svc:9090/. You might need to reload the config after performing these steps by deleting the pod and letting it restart. Metrics won't be available to query until load is generated against the example application.
  6. Run the following commands, each in a separate terminal session:
    1. Generate HTTP load against the prometheus-example-app service:
      kubectl run -i --tty load-generator --rm --image=busybox:1.28 --restart=Never -- /bin/sh -c "while sleep 0.01; do wget -q -O- http://prometheus-example-app; done"
    2. Watch the horizontal pod autoscaler:
      kubectl get hpa prometheus-example-app --watch
    3. Watch the workload scale up:
      kubectl get po -lapp.kubernetes.io/name=prometheus-example-app --watch
  7. Stop HTTP load generation by using Ctrl+C and watch the workload scale back down.

Using the Custom Metrics - Stackdriver Adapter

The Custom Metrics - Stackdriver Adapter, custom-metrics-stackdriver-adapter, does not work with Managed Service for Prometheus. Additionally, this adapter uses resource definitions with the same names as those in the Prometheus Adapter, prometheus-adapter. This overlap in names means that running more than one adapter in the same cluster causes errors.

Installing the Prometheus Adapter in a cluster that previously had the Custom Metrics - Stackdriver Adapter installed might throw errors such as FailedGetObjectMetric due to colliding names. To resolve this, you might have to delete the v1beta1.external.metrics.k8s.io, v1beta1.custom.metrics.k8s.io, and v1beta2.custom.metrics.k8s.io apiservices previously registered by the Custom Metrics Adapter.