Google Cloud Managed Service for Prometheus charges for the number of samples ingested into Cloud Monitoring and for read requests to the Monitoring API. The number of samples ingested is the primary contributor to your cost.
This document describes how you can control costs associated with metric ingestion and how to identify sources of high-volume ingestion.
For more information about the pricing for Managed Service for Prometheus, see Managed Service for Prometheus pricing summary.
View your bill
To view your Google Cloud bill, do the following:
In the Google Cloud console, go to the Billing page.
If you have more than one billing account, select Go to linked billing account to view the current project's billing account. To locate a different billing account, select Manage billing accounts and choose the account for which you'd like to get usage reports.
In the Cost management section of the Billing navigation menu, select Reports.
From the Services menu, select the Cloud Monitoring option.
From the SKUs menu, select the following options:
- Prometheus Samples Ingested
- Monitoring API Requests
The following screenshot shows the billing report for Managed Service for Prometheus from one project:
Reduce your costs
To reduce the costs associated with using Managed Service for Prometheus, you can do the following:
- Reduce the number of time series you send to the managed service by filtering the metric data you generate.
- Reduce the number of samples that you collect by changing the scraping interval.
- Limit the number of samples from potentially misconfigured high-cardinality metrics.
Reduce the number of time series
Open source Prometheus documentation rarely recommends filtering metric volume, which is reasonable when costs are bounded by machine costs. But when paying a managed-service provider on a unit basis, sending unlimited data can cause unnecessarily high bills.
The exporters included in the
service in particular—can emit a lot of metric data.
For example, the
kube-state-metrics service emits hundreds of metrics,
many of which might be completely valueless to you as a consumer. A
fresh three-node cluster using the
kube-prometheus project sends
approximately 900 samples per second to Managed Service for Prometheus.
Filtering these extraneous metrics might be enough by itself to get your bill
down to an acceptable level.
To reduce the number of metrics, you can do the following:
- Modify your scrape configs to scrape fewer targets.
- Filter the collected metrics as described in the following:
- Filter exported metrics when using managed collection.
- Filter exported metrics when using self-deployed collection.
If you are using the
kube-state-metrics service, you could add a
Prometheus relabeling rule with a
action. For managed collection, this rule goes in your PodMonitoring or
ClusterPodMonitoring definition. For
self-deployed collection, this rule goes in your Prometheus scrape
config or your
ServiceMonitor definition (for
For example, using the following filter on a fresh three-node cluster reduces your sample volume by approximately 125 samples per second:
metricRelabeling: - action: keep regex: kube_(daemonset|deployment|pod|namespace|node|statefulset)_.+ sourceLabels: [__name__]
Sometimes, you might find an entire exporter to be unimportant. For example,
kube-prometheus package installs the following service monitors by
default, many of which are unnecessary in a managed environment:
To reduce the number of metrics that you export, you can delete, disable, or
stop scraping the service monitors you don't need. For example, disabling
kube-apiserver service monitor on a fresh three-node cluster reduces
your sample volume by approximately 200 samples per second.
Reduce the number of samples collected
Managed Service for Prometheus charges on a per-sample basis. You can reduce the number of samples ingested by increasing the length of the sampling period. For example:
- Changing a 10-second sampling period to a 30-second sampling period can reduce your sample volume by 66%, without much loss of information.
- Changing a 10-second sampling period to a 60-second sampling period can reduce your sample volume by 83%.
For information about how samples are counted and how the sampling period affects the number of samples, see Pricing examples based on samples ingested.
You can usually set the scraping interval on a per-job or a per-target basis.
For managed collection, you set the scrape interval in the
PodMonitoring resource by using the
For self-deployed collection, you set the sampling interval in your scrape
by setting an
Configure local aggregation (self-deployed collection only)
If you are configuring the service by using self-deployed collection,
for example with kube-prometheus, prometheus-operator, or by manually
deploying the image, then you can reduce your samples sent to Managed Service for Prometheus
by aggregating high-cardinality metrics locally. You can
use recording rules to aggregate away labels such as
instance and use the
--export.match flag or the
variable to only send aggregated data to
For example, assume you have three metrics,
low_cardinality_metric. You want to reduce the samples sent for
high_cardinality_metric_1 and eliminate all samples sent for
high_cardinality_metric_2, while keeping all raw data stored locally (perhaps
for alerting purposes). Your setup might look something like this:
- Deploy the Managed Service for Prometheus image.
- Configure your scrape configs to scrape all raw data into the local server (using as few filters as desired).
Configure your recording rules to run local aggregations over
high_cardinality_metric_2, perhaps by aggregating away the
instancelabel or any number of metric labels, depending on what provides the best reduction in the number of unneeded time series. You might run a rule that looks like the following, which drops the
instancelabel and sums the resulting time series over the remaining labels:
record: job:high_cardinality_metric_1:sum expr: sum without (instance) (high_cardinality_metric_1)
See aggregation operators in the Prometheus documentation for more aggregation options.
Deploy the Managed Service for Prometheus image with the following filter flag, which prevents raw data from the listed metrics from being sent to Monarch:
export.matchflag uses comma-separated selectors with the
!=operator to filter out unwanted raw data. If you add additional recording rules to aggregate other high-cardinality metrics, then you also have to add a new
__name__selector to the filter so that the raw data is discarded. By using a single flag containing multiple selectors with the
!=operator to filter out unwanted data, you only need to modify the filter when you create a new aggregation instead of whenever you modify or add a scrape config.
Certain deployment methods, such as prometheus-operator, might require you to omit the single quotes surrounding the brackets.
This workflow might incur some operational overhead in creating and managing
recording rules and
export.match flags, but it's likely that you can cut a lot
of volume by focusing only on metrics with exceptionally high cardinality. For
information about identifying which metrics might benefit the most from local
pre-aggregation, see Identify high-volume metrics.
Do not implement federation when using Managed Service for Prometheus. This workflow makes using federation servers obsolete, as a single self-deployed Prometheus server can perform any cluster-level aggregations you might need. Federation may cause unexpected effects such as "unknown"-typed metrics and doubled ingestion volume.
Limit samples from high-cardinality metrics (self-deployed collection only)
You can create extremely high-cardinality metrics by adding labels that have
a large number of potential values, like a user ID or IP address. Such
metrics can generate a very large number of samples. Using labels
with a large number of values is typically a misconfiguration. You can
guard against high-cardinality metrics in your self-deployed collectors
by setting a
in your scrape configs.
If you use this limit, we recommend that you set it to a very high value, so that it only catches obviously misconfigured metrics. Any samples over the limit are dropped, and it can be very hard to diagnose issues caused by exceeding the limit.
Using a sample limit is not a good way to manage sample ingestion, but
the limit can protect you against accidental misconfiguration. For more
sample_limit to avoid overload.
Identify and attribute costs
You can use Cloud Monitoring to identify the Prometheus metrics that are writing the largest numbers of samples. These metrics are contributing the most to your costs. After you identify the most expensive metrics, you can modify your scrape configs to filter these metrics appropriately.
The following sections describe ways to analyze the number of samples that you are sending to Managed Service for Prometheus and attribute high volume to specific metrics, Kubernetes namespaces, and Google Cloud regions.
Identify high-volume metrics
To identify the Prometheus metrics with the largest ingestion volumes, do the following:
- In the Google Cloud console, go to the Monitoring page.
- In the Monitoring navigation pane, click Metrics Explorer.
- Select the Configuration tab, and then use the
following information to complete the fields:
- For the Resource and metric field, enter or select Metric Ingestion Attribution and Samples written by attribution id.
- For the Group by field, select metric_type and attribution_dimension.
- For Filters, select attribution_dimension = namespace. This must be done after grouping by attribution_dimension.
- For the Aggregator field, select sum.
The chart now shows the ingestion volumes for each metric type.
- To identify the metrics with the largest ingestion volumes, click Value in the chart legend.
The resulting chart, which shows your 300 top metrics by volume ranked by mean, looks like the following screenshot:
Identify high-volume namespaces
You can also use the metric and resource types from the prior example to attribute ingestion volume to specific Kubernetes namespaces and then take appropriate action. For example:
To correlate overall ingestion volume with namespaces, select the following labels for the Group by field, then filter to attribution_dimension = namespace:
To correlate ingestion volume of individual metrics with namespaces, select the following labels for the Group by field, then filter to attribution_dimension = namespace:
To identify the namespaces responsible for a specific high-volume metric:
- Identify the metric type for the high-volume metric by using one
of the other examples to identify high-volume metric types. The
metric type is the string in the chart legend that begins
To restrict the chart data to a specific metric type, add a filter for the metric type in the Filters field. For example:
Select the following labels for the Group by field, then filter to attribution_dimension = namespace::
- Identify the metric type for the high-volume metric by using one of the other examples to identify high-volume metric types. The metric type is the string in the chart legend that begins with
To see ingestion by Google Cloud region, add the location label to the Group by field.
To see ingestion by Google Cloud project, add the resource_container label to the Group by field.