View metric diagnostics

This document describes how to view information about the user-defined metrics that your Google Cloud project has access to. Your Google Cloud project has access to all metrics visible to its metrics scope.

User-defined metrics let you capture application-specific data or client-side system data. The built-in metrics collected by Cloud Monitoring can give you information on backend latency or disk usage, but they can't tell you how many background routines your application spawned.

User-defined metrics aren't free. For pricing information, see the Cloud Monitoring pricing document. Chargeable metrics are either charged by the number of bytes or by the number of samples that are ingested. This document describes how to view diagnostic information for the metrics described in the following table:

Metric prefix Monitored
Resources
Pricing
model
Meaning
custom.googleapis.com All Bytes Metrics that you define.
external.googleapis.com All Bytes Typically, open source libraries or third-party
providers define and write these metrics.
For more information, see External metrics.
prometheus.googleapis.com All Samples Metrics that Google Cloud Managed Service for Prometheus
defines and writes.
workload.googleapis.com k8s_* Samples Metrics that are written by the Google Kubernetes Engine
workload metrics pipeline. For more information,
see Managing Google Kubernetes Engine metrics define and write.
workload.googleapis.com Not k8s_* Bytes Metrics that third-party integrations with the
Ops agent define and write. For a list of these metrics, see
Third-party application metrics.

To help you manage your user-defined metric costs, and to help you diagnose and fix issues with these metrics, you can use the Diagnostics tab of the Google Cloud console.

Before you begin

The Diagnostics tab contains a table that provides detailed information about user-defined metrics also lets you view the audit logs for those metrics. When you don't have audit logging enabled, no audit logs are available.

To enable audit-logs for reads and writes of your user-defined metrics, do the following:

  1. In the Google Cloud console, select IAM & Admin and then Audit Logs, or use the following button:

    Go to the Audit Logs page

  2. Enter Stackdriver Monitoring API on the filter bar.

  3. Select Stackdriver Monitoring API.

  4. In the Log type tab select Data write and Data read, and then click Save.

For more information, see Configure Data Access audit logs.

View usage summary of user-defined metrics

The number of billable bytes or billable samples ingested determine the costs due to user-defined metrics. Some metrics are charged by bytes ingested while others by samples ingested. To predict your monthly costs due to user-defined metrics, you need to know the rate of data ingestion.

For summaries of the error rate and rates of metric ingestion, doing the following:

  1. In the Cloud console, select Monitoring or click the following button:
    Go to Monitoring
  2. In the navigation pane, select Metrics Explorer .
  3. In the toolbar, select the Diagnostics tab.

    The User-defined metrics pane provides summary information about the user-defined metrics in your metrics scope"

    Example of the summary pane.

    The summary includes the following information:

    • Number of metrics that received data within the selected time window.
    • Number of billable bytes ingested in the time window.
    • Number of billable samples ingested in the time window.
    • The error rate. The error rate is the percentage of metric writes that return an error status relative to the total number of metric writes.

View charts of user-defined metrics

To determine whether your applications are generating a consistent amount of data, which is expected behavior for stable applications, view the collection rates over time. By changing the time window over which you view metrics, you might see dips, peaks, or trends. For example, if you set the time window to a week and you see a constant increase in the number of bytes ingested over time, then that might be a design error. You might check to see if the metric's cardinality is also increasing.

The following screenshot illustrates the charts shown on the Diagnostics tab:

Example of the charts that display user-define metric information.

To view collection rates over time, do the following:

  1. In the Cloud console, select Monitoring or click the following button:
    Go to Monitoring
  2. In the navigation pane, select Metrics Explorer .
  3. In the toolbar, select the Diagnostics tab.

    The charts on the Diagnostics tab display collection rates. By default, chart legends are collapsed. To view the list of time series shown in a chart, click Legend. For information about how to set time references or expand the chart over a specific time window, see Explore charted data.

    • To view the rate of chargeable bytes and samples ingested into the current metrics scope, view the Billed bytes ingested and Billed samples ingested charts.

    • To view the rate of data writes, view the Per metric write rate by status chart.

      Each time series shows the rate of writes for a specific status and metric. For example, there might be a time series for successful data writes for the metric http.client.duration, and a time series for failed data writes of the same metric.

    • To view the volume of data written by metrics, view the Namespace volume ingestion chart.

      Each time series shows the rate of namespace writes for a specific metric.

    • To view the rate of writing metric data by the API, that is by calls to the Cloud Monitoring API method timeSeries.create method, view the API Metrics - CreateTimeSeries chart.

      Each time series shows the total number of API calls that resulted in a specific response code. For example, there is a time series that reports the number of API calls that returned a 2xx response code.

    • To view the rate of creating metric descriptors, view the API Metrics - CreateMetricDescriptor chart.

      It's not unusual for this chart to display "No data is available for the selected timeframe." When this chart has no data to display, that indicates that no metric types are being created.

      When this chart displays data, it indicates that metric descriptors are being created or updated. An update occurs when a label is added to an existing descriptor. For more information, see Create metric descriptors.

      Each time series shows the rate of creating metric descriptors for specific status.

Create alert based on the metric collection rates

To be notified of a spike, dip, or trend in the metric collection rates for your user-defined metrics, create an alerting policy. For example, a dip in collection of metrics might indicate that your application is performing poorly. Similarly, a spike might result in unexpected charges. Lastly, a trend might indicate that a metric has too many labels. In all situations, an alerting policy can notify you of the unusual behavior, and then you can resolve the situation.

To create an alerting policy that monitors a metric collection rate, do the following:

  1. In the Cloud console, select Monitoring or click the following button:
    Go to Monitoring
  2. In the navigation pane, select Metrics Explorer .
  3. In the toolbar, select the Diagnostics tab.
  4. In the chart whose data you want to monitor, click Create alerting policy.

    The alert policy dialog is opened with the condition fields populated. We recommend that you review all settings and make the following modifications:

    • Update the condition threshold value. The default value might not be satisfactory.
    • Add the notification channels to the policy.
    • Name the policy.

    For more information, see Create an alerting policy.

List your user-defined metrics

To manage your costs, you need to understand which user-defined metrics are generating the most traffic. It isn't sufficient to only know, for example, that 60 MiB of data are being ingested every hour. However, when you know that most of your user-defined data is due to one or two metrics, you can investigate the usage of those metrics.

To list your user-defined metrics, do the following:

  1. In the Cloud console, select Monitoring or click the following button:
    Go to Monitoring
  2. In the navigation pane, select Metrics Explorer .
  3. In the toolbar, select the Diagnostics tab.
  4. In the toolbar, select your time window. By default, this tab displays information about the metrics collected in the previous one hour.

    To find the list of your user-defined metrics, use the scroll bar to scroll to the end of the page. The table at the end of the Diagnostics tab lists the user-defined metrics that are in the current metrics scope. For each metric, the table displays the resource labels, the metric labels, and the metric cardinality. The metric cardinality defines the possible number of time series for a metric. For more information, see Cardinality.

    Example of table of user-defined metrics.

    Each row in the table is for a specific metric, and the row lists the monitored resource and the metric labels for the metric:

    • To identify metrics that have received data within the selected time window, add the Status: Active filter to the filter bar.
    • To sort the metrics by the number of bytes ingested, click Bytes ingested.
    • To find a specific metric, add a filter based on the metric description.

      For example, the filter Description: HTTP includes all metrics whose description field contains HTTP. The table might include metrics with descriptions like the following:

      The duration of the inbound HTTP request
      The duration of the outbound HTTP request
      The number of concurrent HTTP requests that are currently in-flight
      

      Similarly, the filter Metric:duration includes all metrics whose name includes duration. The table might include metrics like the following:

      http.server.duration
      http.client.duration
      

      By default, if you add multiple filters, then the table includes a row when the row meets all filters. However, you can insert an OR-filter between two other filter elements.

How to get more information about your user-defined metrics

While you can use the Diagnostics tab to view some information about a metric, you might want more information. For example, you might want to view a chart of a specific metric or create an alerting policy to notify you when that metrics ingestion rate is unexpected.

To get more details about a specific metric, do the following:

  1. In the Cloud console, select Monitoring or click the following button:
    Go to Monitoring
  2. In the navigation pane, select Metrics Explorer .
  3. In the toolbar, select the Diagnostics tab.
  4. Find the metric in the table, and then do any of the following:

    • Evaluate the metric cardinality. The metric cardinality defines the possible number of time series for a metric. Metrics with high cardinality might result in data sets that are too large to chart or query. High cardinality might also be the reason why one metric costs more than others. In general, a lower metric cardinality is better. We recommend that a metric's cardinality be less than 5000. For more information, see Cardinality.

    • To view a chart that displays the current metric, click More options and select View in Metrics Explorer.

      Metrics Explorer opens and is preconfigured to display the selected metric. You can modify the chart configuration, discard it, or you can add it to a custom dashboard.

    • To create an alerting policy that monitors the metric, click More options and select Create alert for metric.

      The alert policy dialog is opened with the condition fields populated. We recommend that you review all settings and make the following modifications:

      • Update the condition threshold value. The default value might not be satisfactory.
      • Add the notification channels to the policy.
      • Name the policy.

      For more information, see Create an alerting policy.

    • To view audit-logs associated with the metric, click More options and select View audit-logs for metric.

      Only audit-logs stored in the current Cloud project are shown. When the user-defined metric is stored in a monitored project, instead of in the current Cloud project, use the project picker to select the Cloud project that stores the metric.

      For information about audit-logs, see Cloud Audit Logs overview.

What's next