Select metrics for charts on dashboards

This document describes how to configure a chart on a custom dashboard to display time-series data collected by your project. Charts can display only numeric time-series data. For information on configuring the style of a chart, see Set chart display options.

Select the data to display

To configure which time series to display when you add charts to a dashboard, you can build a query by making selections from menus, or you can write a query. When you write a query, you select your query language and then you use a query editor or a text-based interface:

Monitoring Query Language (MQL) queries specify the time series and how those time series are grouped and aligned. The MQL interface supports a Query Editor with suggestions and syntax checking.

It isn't generally possible to convert MQL queries into forms that can be used by the other interfaces. Your unsaved queries are discarded when you switch to or from the MQL tab.
Prometheus Query Language (PromQL) queries specify time series and how those time series are grouped and aligned. The PromQL interface supports an editor with suggestions.

It isn't generally possible to convert PromQL queries into forms that can be used by the other interfaces. Your unsaved queries are discarded when you switch to or from the PromQL tab.
Monitoring filter queries specify the time series but don't include grouping or alignment statements.

Any time series that Monitoring can chart can be specified by using a Monitoring filter. For example, to chart the number of processes running on a VM, you must use a Monitoring filter that specifies a function.

It isn't always possible to convert a Monitoring filter into the form required by other interfaces. Therefore, your query might be discarded if you switch to a different interface.

Queries typically specify a metric type, a resource type, and filters:

A metric type identifies the measurements to be collected from a resource. It includes a description of what is being measured and how the measurements are interpreted. A metric type is sometimes referred to as a metric. An example of a metric is "CPU utilization". For conceptual information, see Metric types.
A resource type specifies from which resource the metric data is captured. Resource type is sometimes called the monitored resource type or the resource. An example of a resource is a "Compute Engine virtual machine (VM) instance". For conceptual information, see Monitored resources.

Both MQL and PromQL queries include grouping and alignment statements. However, when you write a Monitoring filter or use menus to select the time series to chart, you configure the grouping and alignment settings by using menus.

Build queries by using menus

To build your query by using menus, do the following:

In the Google Cloud console, select Monitoring or click the following button:
Go to Monitoring
In the navigation pane, select Dashboards, then select the dashboard that you want to view or edit.
If the Edit dashboard button is shown, then click it.
Drag a widget from the widget library to the dashboard, or select a widget that is on the dashboard.

Dashboard widgets are automatically configured with sample selections when they are added to a dashboard. The next steps describe how to change those selections.
Select the Basic tab or the Advanced tab.

The Basic tab provides minimal configuration options, while the Advanced tab provides access to most grouping and alignment fields. You can switch from the Basic tab to the Advanced tab and preserve the configuration. However, you might lose configuration settings when switching from the Advanced tab to the Basic tab.
Select the metric and resource to chart:
1. Expand the Select a metric menu.
2. Optional: To reduce the number of options shown in the menu, enter the metric or resource name in the Filter bar. For example, by entering util you restrict the menu to show entries that include util. Entries are shown when they pass a case-insensitive contains test.
  
  To view a list of all metrics, even those without recent data, set the Show only active resources & metrics toggle to disabled. By default, the this toggle is set to enabled so only metrics and resources with data are shown in the menus.
3. In the Resource menu, select the resource from which the metric data is captured.
  
  When a metric isn't written against a resource, select Unspecified.
4. In the Metric category menu, select the category of the metric.
  
  The category is typically the first term following the metrics prefix. For example, for the metric compute.googleapis.com/instance/utilization, the category is instance.
5. In the Metric menu, select the metric to chart.
6. Click Apply.
For example, to chart the CPU utilization of a virtual machine, you might do the following:
1. Enter util in the Filter bar.
2. In the Resources menu, select VM instance.
3. In the Metric categories menu, select Instance.
4. In the Metrics menu, select CPU utilization and then click Apply.
After resource type and metric are selected, the chart shows all the available time series for that pair. The following screenshot shows a chart after the resource type and metric are selected:

The previous chart contains more data that can be displayed; charts are limited to 300 displayable lines. The chart provides a notice that there is too much data to display, and suggests using outlier mode, which greatly reduces the amount of data to display. To access the outlier mode controls, click Settings. For more information, see Set chart display options.

You can also use the filtering and aggregation options to reduce the amount of charted data. These techniques make the charts more useful for diagnostics and analysis, and they increase the performance and responsiveness of the user interface itself.
Optional: Add filters to restrict which time series are shown. The next section describes the filtering options.
Optional: Configure how the time series are grouped and aligned. For more information, see Choose how to display charted data.

You can reduce the amount of data to be charted by specifying filter criteria, applying aggregation, or by using outlier mode. Filters ensure that only time series that meet some set of criteria are used. When you apply filters, you might reduce the number of lines on the chart, which can improve the performance of the chart.

The remainder of this section applies to configuring a chart when using a menu-driven interface. This section isn't applicable to MQL, PromQL, or to Monitoring configurations.

When you supply multiple filtering criteria, the corresponding chart shows only the time series that meet all criteria, a logical AND. Typically, you can filter by resource group, by name, by resource label, by zone, and by metric label.

To add a filter when you select the Basic or Advanced mode, click Add filter and then specify the filter label, the comparison, and the value or range of values:

Click Label and then select an entry from the menu.

To find a specific label, you can use the scrollbar or you can enter text into the Filter text area. When you enter text, the menu lists only those entries that contain the entered text.

The following screenshot shows the known filter-by labels for a specific metric:
Click Comparison and then select an entry from the menu. You can choose between four operators: equals, =, not equals, !=, regular expression match, =~, and regular expression does not match, !=~:
Click Value and then do one of the following:
- For a direct comparison, = or !=, select the value from the menu or enter a value and click Done. Entered values can be simple values such as us-central1-a, or you can create a filter string that begins with starts_with or ends_with. For example, to display data for any us-central1 zone you could enter the filter string starts_with("us-central1"). See Monitoring filters for more information on filter strings.
  
  Because the menu entries are derived from the received time series, when a monitored resource isn't generating data for the selected metric, you must enter a value for the label.
  
  The following screenshot shows the value menu that is displayed for a particular project when the zone resource label is selected:
- For a regular expression comparison, =~ or !=~, enter a RE2 regular expression into the Value field and click Done. For example, the regular expression us-central1-.* matches all us-central1 zones.
  
  To match any US zone that ends with “a”, you could use the regular expression ^us.*.a$.
  
  You can't use regular expressions to filter the project_id resource label.
  
  For example, to view only the time series from one of the us-central1 zones, apply a zone="starts_with("us-central1")" or zone=~"us-central1.*" filter:

You can specify multiple filter criteria, and you can use the same label multiple times. These capabilities let you specify a filter for a range of values. All filter criteria must be met; they constitute a logical AND. For example, the following a configuration that you can use both starts_with and ends_with filter strings to show only “a” zones in the US:

Example using multiple filters.

Write MQL and PromQL queries

This document doesn't describe how to use MQL or PromQL. For information about these languages, see the following documents:

To enter a MQL and PromQL query, do the following:

In the Google Cloud console, select Monitoring or click the following button:
Go to Monitoring
In the navigation pane, select Dashboards, then select the dashboard that you want to view or edit.
If the Edit dashboard button is shown, then click it.
Drag a widget from the widget library to the dashboard, or select a widget that is on the dashboard.
Select the MQL tab or the PromQL tab.
Enter your query and then click Run.

Write Monitoring-filter queries

When you want to do any of the following, you must use direct filter mode, which lets you enter a Monitoring filter:

Display a service-level objective (SLO).
Display the count of processes running on virtual machines (VMs).
Display a custom metric for which you don't yet have data.
Filter a time series based on a label for which you don't yet have data.

A Monitoring filter, or equivalently a metric filter, is an expression that Monitoring uses to identify the time series to chart. For example, the following expression results in a chart displaying a count of processes whose name includes nginx:

 select_process_count("monitoring.regex.full_match(\".*nginx.*\")")
 resource.type="gce_instance"

You can also use Monitoring filters to identify time series by their resource and metric type. The following expression results in a chart displaying the count of log entries for all Google Cloud virtual machine instances in the us-east1-b zone:

metric.type="logging.googleapis.com/log_entry_count"
resource.type="gce_instance"
resource.label."zone"="us-east1-b"

To enter a Monitoring filter, do the following:

In the Google Cloud console, select Monitoring or click the following button:
Go to Monitoring
In the navigation pane, select Dashboards, then select the dashboard that you want to view or edit.
If the Edit dashboard button is shown, then click it.
Drag a widget from the widget library to the dashboard, or select a widget that is on the dashboard.
Select Basic mode or Advanced mode.
Click Help on the Select a metric menu, and then select Direct Filter Mode.

On the page, a text box is shown. If you selected a resource type, metric, or filters before switching to Direct Filter Mode mode, then those settings are shown in the text box.
Enter a Monitoring filter expression in the text box. For syntax information, see the following documents:
When you use direct filter mode and no data is available that satisfies the filter, an error is shown. Common error messages include Chart definition invalid and No data is available for the selected timeframe.
Optional: Configure how the time series are grouped and aligned. For more information, see Choose how to display charted data.

To return to the menu-driven interface, click Standard mode.

Choose how to display charted data

After the time series data is selected, the next step is to determine how that data is displayed. For example, do you want to display each time series or do you want to combine time series together?

You specify how the data is displayed by configuring the aggregation options. Aggregation consists of aligning time series data and then combining different time series together. Combining time series is optional.

For a detailed explanation of aggregation, see Filtering and aggregation: manipulating time series.

Align data

Alignment is the process of converting the time series data received by Monitoring into a new time series which has data points at fixed intervals. The process of alignment consists of collecting all data points received in a fixed length of time, applying a function to combine those data points, and assigning a timestamp to the result. That function might compute the average of all samples or it might extract the maximum of all samples.

The Alignment period field specifies the minimum time interval to be used when aligning time series data. When there are too many data points to chart in the selected display period, then the alignment period is automatically increased so that every data point is represented. The default setting for this field is one minute.

The Alignment function field specifies the function used to combine all the data points in an alignment period. Most of the aligners perform common mathematical functions. For example, if you select min, then the aligned data point is the minimum of all data points within the alignment period.

For example, consider a metric with a sampling period of one minute. If a chart is configured to display 1 hour of data, then the chart can display all 60 data points. If the alignment period is set to 10 minutes, then the chart displays 6 data points. When the Alignment function field has a value of mean, then each point on the chart is the average of all points in an alignment period. However, if you now configure the chart to display one week of data, then there are too many points to display in the chart so the period is automatically modified. In this example, the modified alignment period is one hour.

While most of the aligners perform common mathematical functions, some perform more complicated actions:

next older: To retain only the most recent sample within an alignment period, use the next older aligner. This aligner is commonly used with uptime checks and is a good choice when you only care about the most recent value.

This aligner is valid only for gauge metrics.
percentile: To display a distribution metric on a plot type of line chart, stacked area chart, or stacked bar chart, you must select which percentile in the distribution to display. One way to specify this percentile is to select a percentile aligner. You can select the 5th, 50th, 95th, and the 99th percentiles. The aligned data point is determined by computing the specified percentile by using all data points in the alignment period.

This aligner is valid only for gauge and delta metrics when they have a distribution data type.
delta: To convert a cumulative metric or a delta metric into a delta metric with one sample per alignment period, use this aligner. This aligner might result in data interpolation. For an example, see Kinds, types, and conversions.

This aligner is valid only for cumulative and delta metrics.
rate: To convert a cumulative or delta metric into a gauge metric, use this aligner. If you choose this aligner, you can think of the time series being transformed as it was with a delta aligner and then divided by the alignment period. For example, if the unit of the original time series is MiB and the unit of the alignment period is second, then the chart has a unit of MiB/second. For more information, see Kinds, types, and conversions.

This aligner is valid only for cumulative and delta metrics.

For more information on the available aligners, see Aligner in the API reference.

To access the alignment fields, do the following:

Basic mode: All alignment fields are preconfigured.
Advanced mode: All alignment options are accessible after you select the resource type and metric. Default values for these fields are provided; however, you can modify the selections.
MQL mode: All alignment options are available.

The following screenshot illustrates the CPU utilization of the Compute Engine VM instances in a particular Google Cloud project. In this image, the alignment fields are at the default values: the alignment function is set to mean and the alignment period is set to 1 minute:

CPU utilization of VM instances using default alignment settings.

For comparison, the following screenshot illustrates the effect of changing the period from 1 minute to 5 minutes:

CPU utilization of VM instances using default with a 5 minute alignment period.

By increasing the period, the resulting chart has fewer points, decreasing from 60 points per time series to 10 points per time series. Each point on the chart is computed by averaging the time series points in an alignment period. By increasing the alignment period, more points are averaged together, which has a smoothing effect on the plotted data.

Combine time series

You can reduce the amount of data returned for a metric by combining different time series. To combine multiple time series, you typically specify a grouping and a function. Grouping is done by label values. The function defines how all time-series data within a group are combined into a new time series.

To group and combine time series, do one of the following:

Basic mode:
1. Determine how to group time series:
  - To display every time series, leave Grouped clear.
  - To group time series by label values, select Grouped and then use the Group by menu to select the labels for grouping. After you make your selections, click OK.
2. Determine how to combine data points:
  - If the radio buttons only list percentiles, then you've selected a metric that has a distribution value. Use the radio buttons to select which percentile from the distribution to view.
  - If the radio buttons list mean, min, and max, then you've selected a metric with a numerical value. Use these buttons to specify how data points are combined as part of the charting process. To understand how these combiners work, assume that a chart can display 60 data points and assume that the data rate is 1 point per minute. If you display one hour of data, then the chart can display every point. However, if you want to display 3 hours of data, you have to reduce 180 data points to 60, which is the number of points the chart can display. One way to reduce the data is to average three adjacent samples, another is to take the minimum.
Advanced mode:
- To combine all time-series data into a single time series, ensure Group by is empty and select how the time series are combined by using the Group by function menu.
  
  Note: You must use this option if you are displaying time series on a heatmap chart. For more information, see About distribution-valued metrics.
- To display all time-series data, do one of the following:
  - Ensure Group by is empty and select none for the Group by function.
  - Click the Group by menu and select Add all, and then select a Group by function. For metrics which store a numerical value, the choice of group-by function might not produce a visibly different chart.
- To group time series by specific label values, click Group by and select the labels for grouping. To specify how the time series are combined, select a function by using the Group by function menu.
MQL mode:

For information about Monitoring Query Language, see Using the Query Editor.

The following screenshot shows a grouping by user_labels.version with the group-by function set to the default value of sum. This selection results in one time series for each value of the user_labels.version. The data points in each time series are computed from the sum of all the values for individual time series for a specific version:

Showing time series' grouped by version.

You can group by multiple labels. When you have multiple grouping options, the group-by function is applied to the set of time series that have the same values for the selected labels.

The resulting chart displays one time series for each combination of label values. The order in which you specify the labels doesn't matter.

For example, the following screenshot illustrates grouping by user_labels.version and system_labels.machine_image:

Showing time series' grouped by version and machine image.

As illustrated, if you group by both the labels, you get one time series for each pair of values. The fact that you get a time series for each combination of labels means that this technique can easily create more data than you can usefully put on a single chart.

When you specify grouping or if you select a group-by function, the charted time series only contains required labels, such as the project identifier, and the labels specified by the grouping.

Remove group-by conditions

To remove all group-by conditions, do one of the following:

Basic mode: Clear Grouped.
Advanced mode: Do the following:
1. In the Group by menu, click Remove all and then click OK.
2. In the Group by function menu, select none.
Note: When using the Advanced mode, if you only delete the group-by labels, then your chart displays a single line that represents all data after the group-by function has been applied. To undo grouping, you must remove the group-by labels and reset the group-by function to none.
MQL mode: Delete grouping commands.

For information about Monitoring Query Language, see Using the Query Editor.

Secondary aggregation

The tab you select determines the aggregation options:

Basic and Advanced modes: The Dashboard editor determines how to map your aggregation selections to the primary and secondary aggregation fields specified by the Cloud Monitoring API by using the following information:
- Widget type
- Kind of metric
- Value type of the metric
- Which mode is being used to configure the widget
To determine this mapping for charts on custom dashboards, you can retrieve the dashboard configuration by using the Google Cloud CLI. For more information, see List dashboards.
MQL mode: These fields accessible.