Configure autoscaling for LLM workloads on GPUs with Google Kubernetes Engine (GKE)

Autopilot Standard

This page shows how to set up your autoscaling infrastructure by using the GKE Horizontal Pod Autoscaler (HPA) to deploy the Gemma large language model (LLM) with the Text Generation Interface (TGI) serving framework from Hugging Face.

To learn more about selecting metrics for autoscaling, see Best practices for autoscaling LLM workloads with GPUs on GKE.

Before you begin

Before you start, make sure you have performed the following tasks:

Enable the Google Kubernetes Engine API.

Enable Google Kubernetes Engine API

If you want to use the Google Cloud CLI for this task, install and then initialize the gcloud CLI. If you previously installed the gcloud CLI, get the latest version by running gcloud components update.
Note: For existing gcloud CLI installations, make sure to set the compute/region and compute/zone properties. By setting default locations, you can avoid errors in gcloud CLI like the following: One of [--zone, --region] must be supplied: Please specify location.

Familiarize yourself with the workflow in Serve Gemma open models using GPUs on GKE with Hugging Face TGI.

Autoscale using server metrics

You can use the workload-specific performance metrics that are emitted by the TGI inference server to direct autoscaling for your Pods. To learn more about these metrics, see Server metrics.

To set up custom-metric autoscaling with server metrics, follow these steps:

Export the metrics from the TGI server to Cloud Monitoring. You use Google Cloud Managed Service for Prometheus, which simplifies deploying and configuring your Prometheus collector. Google Cloud Managed Service for Prometheus is enabled by default in your GKE cluster; you can also enable it manually.

The following example manifest shows how to set up your PodMonitoring resource definition to direct Google Cloud Managed Service for Prometheus to scrape metrics from your Pods at recurring intervals of 15 seconds:
```
apiVersion: monitoring.googleapis.com/v1
kind: PodMonitoring
metadata:
  name: gemma-pod-monitoring
spec:
  selector:
    matchLabels:
      app: gemma-server
  endpoints:
  - port: 8000
    interval: 15s
```
Install the Custom Metrics Stackdriver Adapter. This adapter makes the custom metric that you exported to Monitoring visible to the HPA controller. For more details, see Horizontal pod autoscaling in the Google Cloud Managed Service for Prometheus documentation.

The following example command shows how to install the adapter:
```
kubectl apply -f https://raw.githubusercontent.com/GoogleCloudPlatform/k8s-stackdriver/master/custom-metrics-stackdriver-adapter/deploy/production/adapter_new_resource_model.yaml
```

Set up the custom metric-based HPA resource. Deploy an HPA resource that is based on your preferred custom metric. For more details, see Horizontal pod autoscaling in the Google Cloud Managed Service for Prometheus documentation.

Select one of these tabs to see examples of how to configure the HorizontalPodAutoscaler resource in your manifest:

Queue size

This example uses the tgi_queue_size TGI server metrics, which represents the number of requests in the queue.

To determine the right queue size threshold for HPA, see Best practices for autoscaling LLM inference workloads with GPUs.

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: gemma-server
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: tgi-gemma-deployment
  minReplicas: 1
  maxReplicas: 5
  metrics:
  - type: Pods
    pods:
      metric:
        name: prometheus.googleapis.com|tgi_queue_size|gauge
      target:
        type: AverageValue
        averageValue:  $HPA_AVERAGEVALUE_TARGET

Batch size

This example uses the tgi_batch_size TGI server metric, which represents the number of requests in the current batch.

To determine the right batch size threshold for HPA, see Best practices for autoscaling LLM inference workloads with GPUs.

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: gemma-server
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: tgi-gemma-deployment
  minReplicas: 1
  maxReplicas: 5
  metrics:
  - type: Pods
    pods:
      metric:
        name: prometheus.googleapis.com|tgi_batch_current_size|gauge
      target:
        type: AverageValue
        averageValue:  $HPA_AVERAGEVALUE_TARGET

Autoscale using GPU metrics

You can use the usage and performance metrics emitted by the GPU to direct autoscaling for your Pods. To learn more about these metrics, see GPU metrics.

To set up custom-metric autoscaling with GPU metrics, follow these steps:

Export the GPU metrics to Cloud Monitoring. If your GKE cluster has system metrics enabled, it automatically sends the GPU utilization metric to Cloud Monitoring through the container/accelerator/duty_cycle system metric, every 60 seconds.
- To learn how to enable GKE system metrics, see Configure metrics collection.
- To set up managed collection, see Get started with managed collection in the Google Cloud Managed Service for Prometheus documentation.
- For additional techniques to monitor your GPU workload performance in GKE, see the Run GPUs in GKE Standard node pools.
The following example manifest shows how to set up your PodMonitoring resource definition to ingest metrics from the NVIDIA DCGM workload:
```
apiVersion: monitoring.googleapis.com/v1
kind: PodMonitoring
metadata:
  name: nvidia-dcgm-exporter-for-hpa
  namespace: gmp-public
  labels:
    app.kubernetes.io/name: nvidia-dcgm-exporter
    app.kubernetes.io/part-of: google-cloud-managed-prometheus
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: nvidia-dcgm-exporter
  endpoints:
    - port: metrics
      interval: 15s
      metricRelabeling:
        - action: keep
          sourceLabels: [__name__]
        - action: replace
          sourceLabels: [__name__]
          targetLabel: __name__
          regex: DCGM_FI_DEV_GPU_UTIL
          replacement: dcgm_fi_dev_gpu_util
```
In the code, make sure to change the DCGM metric name to use in HPA to lowercase. This is because there's a known issue where HPA doesn't work with uppercase external metric names.
Install the Custom Metrics Stackdriver Adapter. This adapter makes the custom metric you exported to Monitoring visible to the HPA controller. For more details, see Horizontal pod autoscaling in the Google Cloud Managed Service for Prometheus documentation.

The following example command shows how to execute this installation:
```
kubectl apply -f https://raw.githubusercontent.com/GoogleCloudPlatform/k8s-stackdriver/master/custom-metrics-stackdriver-adapter/deploy/production/adapter_new_resource_model.yaml
```

Set up the custom metric-based HPA resource. Deploy a HPA resource based on your preferred custom metric. For more details, see Horizontal pod autoscaling in the Google Cloud Managed Service for Prometheus documentation.

Identify an average value target for HPA to trigger autoscaling. You can do this experimentally; for example, generate increasing load on your server and observe where your GPU utilization peaks. Be mindful of the HPA tolerance, which defaults to a 0.1 no-action range around the target value to dampen oscillation.
We recommend using the locust-load-inference tool for testing. You can also create a Cloud Monitoring custom dashboard to visualize the metric behavior.

Select one of these tabs to see an example of how to configure the HorizontalPodAutoscaler resource in your manifest:

Duty cycle (GKE system)

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: gemma-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: tgi-gemma-deployment
  minReplicas: 1
  maxReplicas: 5
  metrics:
  - type: External
    external:
      metric:
        name: kubernetes.io|container|accelerator|duty_cycle
        selector:
          matchLabels:
            resource.labels.container_name: inference-server
            resource.labels.namespace_name: default
      target:
        type: AverageValue
        averageValue:  $HPA_AVERAGEVALUE_TARGET

Duty cycle (DCGM)

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: gemma-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: tgi-gemma-deployment
  minReplicas: 1
  maxReplicas: 5
  metrics:
  - type: External
    external:
      metric:
        name: prometheus.googleapis.com|dcgm_fi_dev_gpu_util|unknown
        selector:
          matchLabels:
            metric.labels.exported_container: inference-server
            metric.labels.exported_namespace: default
      target:
        type: AverageValue
        averageValue:  $HPA_AVERAGEVALUE_TARGET

What's next

Learn how to optimize Pod autoscaling based on metrics from Cloud Monitoring.
Learn more about Horizontal Pod Autoscaling from the open source Kubernetes documentation.