Select metrics for charts on dashboards

This document describes how to configure a chart on a custom dashboard to display time-series data collected by your project. Charts can display only numeric time-series data. For information on configuring the style of a chart, see Set chart display options.

Select the data to display

To configure which time series to display on a chart, you can build a query by making selections from menus, or you can write a query. When you write a query, you select your query language and then you use a query editor or a text-based interface:

Monitoring Query Language (MQL) queries specify the time series and how those time series are grouped and aligned. The MQL interface supports a code editor with suggestions and syntax checking.

It isn't generally possible to convert MQL queries into forms that can be used by the other interfaces. Your unsaved queries are discarded when you switch to or from the MQL tab.
Prometheus Query Language (PromQL) queries specify time series and how those time series are grouped and aligned. The PromQL interface supports an editor with suggestions.

It isn't generally possible to convert PromQL queries into forms that can be used by the other interfaces. Your unsaved queries are discarded when you switch to or from the PromQL tab.
Monitoring filter queries specify the time series but don't include grouping or alignment statements.

Any time series that Monitoring can chart can be specified by using a Monitoring filter. For example, to chart the number of processes running on a VM, you must use a Monitoring filter that specifies a function.

It isn't always possible to convert a Monitoring filter into the form required by other interfaces. Therefore, your query might be discarded if you switch to a different interface.

Queries typically specify a metric type, a resource type, and filters:

A metric type identifies the measurements to be collected from a resource. It includes a description of what is being measured and how the measurements are interpreted. A metric type is sometimes referred to as a metric. An example of a metric is "CPU utilization". For conceptual information, see Metric types.
A resource type specifies from which resource the metric data is captured. Resource type is sometimes called the monitored resource type or the resource. An example of a resource is a "Compute Engine virtual machine (VM) instance". For conceptual information, see Monitored resources.

Both MQL and PromQL queries include grouping and alignment statements. However, when you write a Monitoring filter or use menus to select the time series to chart, you configure the grouping and alignment settings by using menus.

Build queries by using menus

To build your query by using menus, do the following:

In the Google Cloud console, select Monitoring, and then select Dashboards, or click the following button:
Go to Dashboards
Do one of the following:
- To create a new dashboard, select Create dashboard.
- To update an existing dashboard, find the dashboard in the list of dashboards and then select its name.
In the toolbar, click Add widget.
In the Add widget dialog, select Metric.
In the toolbar of the query pane, do the following:
1. Expand the Select a metric menu.
2. Optional: To reduce the number of options shown in the menu, enter the metric or resource name in the Filter bar. For example, by entering util you restrict the menu to show entries that include util. Entries are shown when they pass a case-insensitive "contains" test.
  
  To view a list of all metrics, even those without recent data, enable the Show only active resources & metrics toggle. By default, only metrics and resources with data are shown in the menus.
3. In the Resources menu, select the resource from which the metric data is captured.
  
  When a metric isn't written against a resource, select Unspecified.
4. In the Metric categories menu, select the category of the metric.
  
  The category is typically the first term following the service domain. For example, in the Compute Engine metric type compute.googleapis.com/instance/utilization, the service domain is compute.googleapis.com and the category is instance.
5. In the Metrics menu, select the metric to chart.
6. Click Apply.
For example, to chart the CPU utilization of a Compute Engine virtual machine, do the following:
1. Enter util in the Filter bar.
2. In the Resources menu, select VM instance.
3. In the Metric categories menu, select Instance.
4. In the Metrics menu, select CPU utilization and then click Apply.
After the resource type and metric are selected, the chart shows the available time series for that pair:

The previous chart contains more data than can be displayed; charts are limited to 300 displayable lines. The chart provides a notice that there is too much data to display. To reduce the amount of data, click Add sort and limit and then complete the dialog. For more information, see Set view options.

You can also use the filtering and aggregation options to reduce the amount of charted data. These techniques make the charts more useful for diagnostics and analysis, and they increase the performance and responsiveness of the user interface itself.
Optional: Add filters to restrict which time series are shown. The next section describes the filtering options.
Optional: Configure how the time series are grouped and aligned. For more information, see Choose how to display charted data.
To save your changes, click Apply. To discard your changes, click Cancel.

You can reduce the amount of data to be charted by specifying filter criteria, applying aggregation, or by sorting the time series that match a query, and then limiting the number of time series that are charted. For more information, see Show outliers.

Filters ensure that only time series that meet some set of criteria are used. When you apply filters, you might reduce the number of lines on the chart, which can improve the performance of the chart.

The remainder of this section applies to configuring a chart when using a menu-driven interface. This section isn't applicable to MQL, PromQL, or to Monitoring configurations.

When you supply multiple filtering criteria, the corresponding chart shows only the time series that meet all criteria, a logical AND. Typically, you can filter by resource group, by name, by resource label, by zone, and by metric label.

To add a filter when you use the menu-driven interface, click Add filter and then specify the filter label, the comparison, and the value or range of values:

Click Label and then select an entry from the menu.

To find a specific label, you can use the scrollbar or you can enter text into the Filter text area. When you enter text, the menu lists only those entries that contain the entered text.

The following screenshot shows the known filter-by labels for a specific metric:
Click Comparison and then select an entry from the menu. You can choose between four operators: equals, =, not equals, !=, regular expression match, =~, and regular expression does not match, !=~:
Click Value and then do one of the following:
- For a direct comparison, = or !=, select the value from the menu or enter a value and click Done. Entered values can be simple values such as us-central1-a, or you can create a filter string that begins with starts_with or ends_with. For example, to display data for any us-central1 zone you could enter the filter string starts_with("us-central1"). See Monitoring filters for more information on filter strings.
  
  Because the menu entries are derived from the received time series, when a monitored resource isn't generating data for the selected metric, you must enter a value for the label.
  
  The following screenshot shows the value menu that is displayed for a particular project when the zone resource label is selected:
- For a regular expression comparison, =~ or !=~, enter a RE2 regular expression into the Value field and click Done. For example, the regular expression us-central1-.* matches all us-central1 zones.
  
  To match any US zone that ends with “a”, you could use the regular expression ^us.*.a$.
  
  You can't use regular expressions to filter the project_id resource label.
  
  For example, to view only the time series from one of the us-central1 zones, apply a zone="starts_with("us-central1")" or zone=~"us-central1.*" filter:

You can specify multiple filter criteria, and you can use the same label multiple times. These capabilities let you specify a filter for a range of values. All filter criteria must be met; they constitute a logical AND. For example, the following a configuration that you can use both starts_with and ends_with filter strings to show only “a” zones in the US:

Example using multiple filters.

Write MQL and PromQL queries

This document doesn't describe how to use MQL or PromQL. For information about these languages, see the following documents:

To enter an MQL or a PromQL query, do the following:

In the Google Cloud console, select Monitoring, and then select Dashboards, or click the following button:
Go to Dashboards
Do one of the following:
- To create a new dashboard, select Create dashboard.
- To update an existing dashboard, find the dashboard in the list of all dashboards and then select its name.
In the toolbar, click Add widget.
In the Add widget dialog, select Metric.
In the toolbar of the Select a metric pane, select the button whose name starts with Code.
Select your query language by using the Language toggle. This toggle is in the same toolbar that lets you format your query.
Optional: Disable the Auto-run toggle.
Enter your query and then click Run query.

When the Auto-run toggle is enabled, the Run query button isn't shown.

Write Monitoring-filter queries

When you want to do any of the following, you must use direct filter mode, which lets you enter a Monitoring filter:

Display a service-level objective (SLO).
Display the count of processes running on virtual machines (VMs).
Display a custom metric for which you don't yet have data.
Filter a time series based on a label for which you don't yet have data.

A Monitoring filter, or equivalently a metric filter, is an expression that Monitoring uses to identify the time series to chart. For example, the following expression results in a chart displaying a count of processes whose name includes nginx:

 select_process_count("monitoring.regex.full_match(\".*nginx.*\")")
 resource.type="gce_instance"

You can also use Monitoring filters to identify time series by their resource and metric type. The following expression results in a chart displaying the count of log entries for all Google Cloud virtual machine instances in the us-east1-b zone:

metric.type="logging.googleapis.com/log_entry_count"
resource.type="gce_instance"
resource.label."zone"="us-east1-b"

To enter a Monitoring filter, do the following:

In the Google Cloud console, select Monitoring, and then select Dashboards, or click the following button:
Go to Dashboards
Do one of the following:
- To create a new dashboard, select Create dashboard.
- To update an existing dashboard, find the dashboard in the list of dashboards and then select its name.
In the toolbar, click Add widget.
In the Add widget dialog, select Metric.
Click Help on the Select a metric menu, and then select Direct Filter Mode.

On the page, a text box is shown. If you selected a resource type, metric, or filters before switching to Direct Filter Mode mode, then those settings are shown in the text box.
Enter a Monitoring filter expression in the text box. For syntax information, see the following documents:
When you use direct filter mode and no data is available that satisfies the filter, an error is shown. Common error messages include Chart definition invalid and No data is available for the selected timeframe.
Optional: Configure how the time series are grouped and aligned. For more information, see Choose how to display charted data.

To return to the menu-driven interface, click Standard mode.

Choose how to display charted data

After the time series data is selected, the next step is to determine how that data is displayed. For example, do you want to display each time series or do you want to combine time series together?

This section covers how to set the aggregation fields. Aggregation consists of alignment of data points within a time series, and combining different time series together. For a detailed explanation of aggregation, see Filtering and aggregation: manipulating time series.

For information about view options, see Set chart display options.
For more information about interacting with the chart, see Explore charted data.

The contents of this section doesn't apply when you've selected the data to chart by using a MQL or PromQL.

Align data

Alignment is the process of converting the time series data received by Monitoring into a new time series which has data points at fixed intervals. The process of alignment consists of collecting all data points received in a fixed length of time, applying a function to combine those data points, and assigning a timestamp to the result. That function might compute the average of all samples or it might extract the maximum of all samples.

The Min alignment period field specifies the minimum time interval to be used when aligning time series data. When there are too many data points to chart in the selected display period, then the alignment period is automatically increased so that every data point is represented. The default setting for this field is one minute.

The Alignment function field specifies the function used to combine all the data points in an alignment period. Most of the aligners perform common mathematical functions. For example, if you select min, then the aligned data point is the minimum of all data points within the alignment period.

For example, consider a metric with a sampling period of one minute. If a chart is configured to display 1 hour of data, then the chart can display all 60 data points. If the alignment period is set to 10 minutes, then the chart displays 6 data points. When the Alignment function field has a value of mean, then each point on the chart is the average of all points in an alignment period. However, if you now configure the chart to display one week of data, then there are too many points to display in the chart so the period is automatically modified. In this example, the modified alignment period is one hour.

While most of the aligners perform common mathematical functions, some perform more complicated actions:

next older: To retain only the most recent sample within an alignment period, use the next older aligner. This aligner is commonly used with uptime checks and is a good choice when you only care about the most recent value.

This aligner is valid only for gauge metrics.
percentile: To display a distribution metric on a plot type of line chart, stacked area chart, or stacked bar chart, you must select which percentile in the distribution to display. One way to specify this percentile is to select a percentile aligner. You can select the 5th, 50th, 95th, and the 99th percentiles. The aligned data point is determined by computing the specified percentile by using all data points in the alignment period.

This aligner is valid only for gauge and delta metrics when they have a distribution data type.
delta: To convert a cumulative metric or a delta metric into a delta metric with one sample per alignment period, use this aligner. This aligner might result in data interpolation. For an example, see Kinds, types, and conversions.

This aligner is valid only for cumulative and delta metrics.
rate: To convert a cumulative or delta metric into a gauge metric, use this aligner. If you choose this aligner, you can think of the time series being transformed as it was with a delta aligner and then divided by the alignment period. For example, if the unit of the original time series is MiB and the unit of the alignment period is second, then the chart has a unit of MiB/second. For more information, see Kinds, types, and conversions.

This aligner is valid only for cumulative and delta metrics.

For more information on the available aligners, see Aligner in the API reference.

To access the alignment fields, click More Options in the query pane, and then go to the Datapoint alignment section.

The following screenshot illustrates the CPU utilization of the Compute Engine VM instances in a particular Google Cloud project. In this image, the alignment fields are at the default values: the alignment function is set to mean and the alignment period is set to 1 minute:

CPU utilization of VM instances using default alignment settings.

For comparison, the following screenshot illustrates the effect of changing the period from 1 minute to 5 minutes:

CPU utilization of VM instances using default with a 5 minute alignment period.

By increasing the period, the resulting chart has fewer points, decreasing from 60 points per time series to 10 points per time series. Each point on the chart is computed by averaging the time series points in an alignment period. By increasing the alignment period, more points are averaged together, which has a smoothing effect on the plotted data.

Combine time series

You can reduce the amount of data returned for a metric by combining different time series. To combine multiple time series, you typically specify a grouping and a function. Grouping is done by label values. The function defines how all time-series data within a group are combined into a new time series.

By default, a grouping is added after you select your resource type and metric type. This grouping is configured to average all time series, and as a result, the chart displays a single line:

To group by specific labels, in the Group by section, click Labels and make your selections. You can group by multiple labels.

When the Labels field is empty, all time series are combined and the chart displays one line.
To show all time series, on the Group by entry, click Delete grouping.

For example, if you set the Labels field to user_labels.version and set the Grouping function field to sum, then there is one time series for each value of the label user_labels.version. The data points in each time series are computed from the sum of all the values for individual time series for a specific version:

Showing time series' grouped by user_labels.version.

When you group time series by using multiple labels, the grouping function is applied to the set of time series that have the same values for the selected labels. The resulting chart displays one time series for each combination of label values. The order in which you specify the labels doesn't matter.

For example, the following screenshot illustrates grouping by user_labels.version and system_labels.machine_image:

Showing time series' grouped by version and machine image.

As illustrated, if you group by both the labels, you get one time series for each pair of values. The fact that you get a time series for each combination of labels means that this technique can easily create more data than you can usefully put on a single chart.

When you specify grouping or if you select a grouping function the charted time series only contains required labels, such as the project identifier, and the labels specified by the grouping.

Remove group-by conditions

To remove the grouping, click Delete grouping.

Secondary aggregation

When you have multiple time series that already represent aggregations, you can reduce all the time series on the chart to a single time series by choosing a Secondary Aggregator. For example, if you group data by zone, your chart shows one time series for each zone. To create a chart with a single time series, use the secondary aggregation fields.

When you set the Transform field to a value other than None, all other fields are the secondary aggregation settings.