You can use Prometheus Query Language (PromQL) to query all metrics in Cloud Monitoring, including Google Cloud system metrics, Kubernetes metrics, custom metrics, and log-based metrics.
Mapping Cloud Monitoring metric names to PromQL
Cloud Monitoring metric names include two components, a domain (such as
compute.googleapis.com/
) and a path (such as
instance/disk/max_read_ops_count
). Because PromQL only supports the
special characters :
and _
, you must first make Monitoring
metric names compatible with PromQL. To map Monitoring
metric names to PromQL, apply the following rules:
- Replace the first
/
with:
. - Replace all other special characters (including
.
and other/
characters) with_
.
The following table lists some metric names and their PromQL equivalents:
Cloud Monitoring metric name | PromQL metric name |
---|---|
kubernetes.io/container/cpu/limit_cores |
kubernetes_io:container_cpu_limit_cores |
compute.googleapis.com/instance/cpu/utilization |
compute_googleapis_com:instance_cpu_utilization |
logging.googleapis.com/log_entry_count |
logging_googleapis_com:log_entry_count |
custom.googleapis.com/opencensus/opencensus.io/ |
custom_googleapis_com:opencensus_opencensus_io_ |
agent.googleapis.com/disk/io_time |
agent_googleapis_com:disk_io_time |
Cloud Monitoring distribution-valued metrics can be queried
like Prometheus histograms, with _count
, _sum
, or _bucket
suffix
appended to the metric name:
Cloud Monitoring metric name | PromQL metric names |
---|---|
networking.googleapis.com/vm_flow/rtt |
networking_googleapis_com:vm_flow_rtt_sum networking_googleapis_com:vm_flow_rtt_count networking_googleapis_com:vm_flow_rtt_bucket
|
Specifying a monitored resource type
When a metric is associated with only a single Cloud Monitoring monitored resource type, PromQL querying will work without manually specifying a resource type. However, some metrics within Cloud Monitoring, including some system metrics and many of those generated by log-based metrics, map to more than one resource type.You can see which monitored resource types map to a metric by doing one of the following:
- For Google-curated metrics, you can consult the lists of metrics available, including Google Cloud metrics and Kubernetes metrics. Each entry in the documentation lists the associated monitored resource types in the first column of each entry below the type. If no monitored resource types are listed, then the metric can be associated with any type.
In Metrics Explorer, you can do the following:
- Enter your metric name in the Select a metric field and then navigate through the menus to select the metric. The resource menu lists valid resource types for that metric, for example, "VM Instance".
- In the toolbar of the query-builder pane, select the button whose name is either code MQL or code PromQL.
Verify that MQL is selected in the Language toggle. The language toggle is in the same toolbar that lets you format your query.
The displayed query shows the resource type. In particular, MQL is useful for metrics that can be associated with many monitored resource types, for example log-based metrics, custom metrics, or any user-defined metric.
If a metric is associated with more than one resource type, then you must
specify the resource type in your PromQL query. There is a special label,
monitored_resource
, that you can use to select the resource type.
Monitored resource types are in most cases a short string,
like gce_instance
, but occasionally they appear as full URIs, like
monitoring.googleapis.com/MetricIngestionAttribution
. Well-formed PromQL
queries might look like the following:
logging_googleapis_com:byte_count{monitored_resource="k8s_container"}
custom_googleapis_com:opencensus_opencensus_io_http_server_request_count_by_method{monitored_resource="global"}
loadbalancing_googleapis_com:l3_external_egress_bytes_count{monitored_resource="loadbalancing.googleapis.com/ExternalNetworkLoadBalancerRule"}
The value of ""
for the monitored_resource
label is special and refers to
the default prometheus_target
resource type that is used for
Managed Service for Prometheus metrics.
If you don't use the monitored_resource
label when it is needed, then you
receive the following error:
metric is configured to be used with more than one monitored resource type;
series selector must specify a label matcher on monitored resource name
Using metadata labels
You can use metadata labels in PromQL just like any other
label, but like metric names, metadata labels also need to be made
PromQL-compatible. The syntax for referring to a metadata system label
version
is metadata_system_version
, and the syntax for metadata user
label version
is metadata_user_version
. Well-formed PromQL queries
using metadata labels might look like the following:
compute_googleapis_com:instance_cpu_utilization{monitored_resource="gce_instance",metadata_user_env="prod"}
sum(compute_googleapis_com:instance_cpu_utilization) by (metadata_system_region)
The only special character you can use in metadata label keys is the _
character. Aggregation using metadata labels with the type MULTI_STRING
or
KEY_VALUE
is not supported.
Resolving label conflicts
In Cloud Monitoring, labels can belong to either the metric or the resource.
If a metric label has the same key name as a resource label, you can refer to
the metric label specifically by adding the prefix metric_
to the label key
name in your query.
For example, suppose you have a resource label and a
metric label both named pod_name
in the metric
example.googleapis.com/user/widget_count
.
To filter on the value of the resource label, use
example_googleapis_com:user_widget_count{pod_name="RESOURCE_LABEL_VALUE"}
To filter on the value of the metric label, use
example_googleapis_com:user_widget_count{metric_pod_name="METRIC_LABEL_VALUE"}
Additional topics
Rules and alerts
You can use Cloud Monitoring metrics in both recording and alerting rules in Managed Service for Prometheus. For instructions, see Managed rule evaluation and alerting or Self-deployed rule evaluation and alerting.
Learning PromQL
To learn the basics of using PromQL, we recommend consulting open source documentation. Some helpful links might be:
PromQL differences
PromQL for Managed Service for Prometheus might function slightly differently than upstream PromQL. For a list of these differences, see PromQL differences.