Mapping Cloud Monitoring metrics to PromQL

You can use Prometheus Query Language (PromQL) to query all metrics stored by Cloud Monitoring, including Google Cloud system metrics, Kubernetes metrics, custom metrics, and log-based metrics.

Mapping Cloud Monitoring metric names to PromQL

Cloud Monitoring metric names include two components, a domain (such as compute.googleapis.com/) and a path (such as instance/disk/max_read_ops_count). Because PromQL only supports the special characters : and _, you must first make Monitoring metric names compatible with PromQL. To map Monitoring metric names to PromQL, apply the following rules:

• Replace the first / with :.
• Replace all other special characters (including . and other / characters) with _.

The following table lists some metric names and their PromQL equivalents:

Cloud Monitoring metric name	PromQL metric name
kubernetes.io/container/cpu/limit_cores	kubernetes_io:container_cpu_limit_cores
compute.googleapis.com/instance/cpu/utilization	compute_googleapis_com:instance_cpu_utilization
logging.googleapis.com/log_entry_count	logging_googleapis_com:log_entry_count
custom.googleapis.com/opencensus/opencensus.io/ http/server/request_count_by_method	custom_googleapis_com:opencensus_opencensus_io_ http_server_request_count_by_method
agent.googleapis.com/disk/io_time	agent_googleapis_com:disk_io_time

Cloud Monitoring distribution-valued metrics can be queried like Prometheus histograms, with _count, _sum, or _bucket suffix appended to the metric name:

Cloud Monitoring metric name	PromQL metric names
networking.googleapis.com/vm_flow/rtt	networking_googleapis_com:vm_flow_rtt_sum networking_googleapis_com:vm_flow_rtt_count networking_googleapis_com:vm_flow_rtt_bucket

Specifying a monitored resource type

When a metric is associated with only a single Cloud Monitoring monitored resource type, PromQL querying will work without manually specifying a resource type. However, some metrics within Cloud Monitoring, including some system metrics and many of those generated by log-based metrics, map to more than one resource type.

You can see which monitored resource types map to a metric by doing one of the following:

• For Google-curated metrics, you can consult the lists of metrics available, including Google Cloud metrics and Kubernetes metrics. Each entry in the documentation lists the associated monitored resource types in the first column of each entry below the type. If no monitored resource types are listed, then the metric can be associated with any type.

• In Metrics Explorer, you can do the following:

  1. Enter your metric name in the Select a metric field and then navigate through the menus to select the metric. The resource menu lists valid resource types for that metric, for example, "VM Instance".
  2. In the toolbar of the query-builder pane, select the button whose name is either  MQL or  PromQL.

  3. Verify that MQL is selected in the Language toggle. The language toggle is in the same toolbar that lets you format your query.

     The displayed query shows the resource type. In particular, MQL is useful for metrics that can be associated with many monitored resource types, for example log-based metrics, custom metrics, or any user-defined metric.

If a metric is associated with more than one resource type, then you must specify the resource type in your PromQL query. There is a special label, monitored_resource, that you can use to select the resource type.

Monitored resource types are in most cases a short string, like gce_instance, but occasionally they appear as full URIs, like monitoring.googleapis.com/MetricIngestionAttribution. Well-formed PromQL queries might look like the following:

• logging_googleapis_com:byte_count{monitored_resource="k8s_container"}
• custom_googleapis_com:opencensus_opencensus_io_http_server_request_count_by_method{monitored_resource="global"}
• loadbalancing_googleapis_com:l3_external_egress_bytes_count{monitored_resource="loadbalancing.googleapis.com/ExternalNetworkLoadBalancerRule"}

The value of "" for the monitored_resource label is special and refers to the default prometheus_target resource type that is used for Managed Service for Prometheus metrics.

If you don't use the monitored_resource label when it is needed, then you receive the following error:

metric is configured to be used with more than one monitored resource type; series selector must specify a label matcher on monitored resource name

Using metadata labels

You can use metadata labels in PromQL just like any other label, but like metric names, metadata labels also need to be made PromQL-compatible. The syntax for referring to a metadata system label version is metadata_system_version, and the syntax for metadata user label version is metadata_user_version. Well-formed PromQL queries using metadata labels might look like the following:

• compute_googleapis_com:instance_cpu_utilization{monitored_resource="gce_instance",metadata_user_env="prod"}
• sum(compute_googleapis_com:instance_cpu_utilization) by (metadata_system_region)

The only special character you can use in metadata label keys is the _ character. Aggregation using metadata labels with the type MULTI_STRING or KEY_VALUE is not supported.

Resolving label conflicts

In Cloud Monitoring, labels can belong to either the metric or the resource. If a metric label has the same key name as a resource label, you can refer to the metric label specifically by adding the prefix metric_ to the label key name in your query.

For example, suppose you have a resource label and a metric label both named pod_name in the metric example.googleapis.com/user/widget_count.

• To filter on the value of the resource label, use
  example_googleapis_com:user_widget_count{pod_name="RESOURCE_LABEL_VALUE"}

• To filter on the value of the metric label, use
  example_googleapis_com:user_widget_count{metric_pod_name="METRIC_LABEL_VALUE"}