Use Prometheus exemplars

This document describes how you can use Google Cloud Managed Service for Prometheus to ingest Prometheus exemplars on histogram metrics. You must use one of the following:

  • Managed Service for Prometheus managed collection version 0.7.0 or higher
  • Managed Service for Prometheus self-deployed collection version 2.35.0-gmp.5-gke.0 or higher
  • The OpenTelemetry Collector

Exemplars are typically used to associate trace identifiers with metric data collected in a time interval, but they can be used to associate any non-metric data with the collected metrics. Storing trace information with the metric data lets you quickly identify the traces associated with a sudden change in metric values; you don't have to manually cross-reference trace information and metric data by using timestamps to identify what was happening in the application when the metric data was recorded.

Managed Service for Prometheus exemplars are compatible both with Cloud Trace and with 3rd-party trace tools such as Grafana Tempo.

The following screenshot shows a Grafana chart of sets of histogram buckets and associated exemplars, with one exemplar expanded:

Grafana chart showing histogram buckets and exemplars.

Prometheus exemplars in Managed Service for Prometheus

Exemplars are supported in Prometheus by using the OpenMetrics exposition format. An OpenMetrics exemplar has the following structure:

  • The value of the exemplar: must be either an integer or a 64-bit float.
  • (Optional) A timestamp: must be Unix Epoch seconds, can be negative.
  • A number of labels for the exemplar, typically including trace_id and span_id values; the total length of all label names and values must not exceed 128 UTF-8 characters, or 128 runes in Go applications.

Google Cloud Managed Service for Prometheus can only ingest exemplars attached to histogram metrics. Exemplars attached to counter metrics can't be ingested. Prometheus histogram metrics are converted to the analogous Cloud Monitoring Distribution type, which supports exemplars. Non-distribution metrics in Cloud Monitoring do not support exemplars.

Exemplars remain available for querying for 24 months. Compare this retention period to upstream Prometheus, where exemplars are retained only while the data is in-memory, typically less than 14 days. Managed Service for Prometheus stores exemplars with the time series, so the exemplars are retained according to the retention policy for the time series. For more information about data retention in Cloud Monitoring, see Data retention.

Enable ingestion of exemplars

If you are using Managed Service for Prometheus with managed collection version 0.7.0 or higher, then ingestion of exemplars is automatically enabled.

If you are using Managed Service for Prometheus with self-deployed collection, then you must enable ingestion of exemplars when you start the server by specifying the following feature flag:

--enable-feature=exemplar-storage

Write metrics with exemplars

Each Prometheus client has its own instructions for properly using exemplars in that client. Follow these instructions so that your application exposes metrics in the OpenMetrics format for Prometheus to scrape from the application's /metrics endpoint. For examples, see the following:

Integrate with Cloud Trace

You can write Prometheus histograms with exemplars that are compatible with Cloud Trace. You can display the data, including the exemplars, on a Cloud Monitoring heatmap chart, where you can use the exemplars to view details about the related trace spans. You can also analyze the trace data in Trace. For more information, see Show exemplars on a heatmap chart and View trace details.

To create exemplars that are compatible with Trace, your Prometheus exemplars must contain the following labels:

  • project_id
  • trace_id
  • span_id

If your exemplar includes these three labels, then the labels are used to create a Trace-compatible SpanContext object that is attached to the exemplar created for the Cloud Monitoring Distribution bucket.

All labels other than those that make up a complete SpanContext are added to a DroppedLabels object attached to the exemplar for a Distribution bucket.

Query metrics with exemplars

The Prometheus API exposes the api/v1/query_exemplars endpoint for querying exemplars.

Query interfaces set up to interact with Google Cloud Managed Service for Prometheus automatically include exemplar querying. For information about setting up query interfaces, see Query using Grafana or Query using the Prometheus API or UI.