Using quota metrics

This page describes how to create alerting policies and charts to monitor or to view quota usage. For general information about quotas, including information on allocation quotas and rate quotas, see Working with quotas.

For all alerting policies and charts on this page, the monitored resource is consumer_quota. The metrics for this resource are a subset of the serviceruntime metrics.

  • For definitions of the terms gauge, delta, and cumulative, see Kinds of metrics.
  • For details about the fields used for combining time series, see Aligner and Reducer.

Alerting policy examples

An alerting policy is a collection of conditions and notification information:

  • A condition describes what is being monitored, how the time series data for that resource is to be combined, and when an alert must be generated. An alerting policy must specify at least one condition.
  • The notification channel specifies who is to be notified, and how they are to be notified, when an alert occurs. For example, you can configure the alerting policy to send an email to a specific person or to a group of people.

In the remainder of this section, each subsection includes a JSON representation of an alerting policy. If the policy can be created using the Google Cloud Console, two tables are also included:

  • The first table describes what is being monitored and how the data is to be combined.
  • The second table describes when an alert is to be generated.

The values in these tables can be used to configure an alerting policy condition.

Alerting on the ratio of quota-usage to quota-limit

You can create an alerting policy to notify you if the ratio of quota usage to the quota limit for any service in your Google Cloud project exceeds 80%. You can create this type of policy only by using the Cloud Monitoring API.

Using the Cloud Monitoring API

You can create this alerting policy using the API method alertPolicies.create. For more information on representing alerting policies in JSON or YAML, see Sample policies.

This example policy monitors the ratio of the Cloud Spanner API quota usage to quota limit. Because combining quota limits might be meaningless, in this example, a location is specified.

{
    "combiner": "OR",
    "conditions": [
      {
        "conditionThreshold": {
          "filter": "metric.type=\"serviceruntime.googleapis.com/quota/allocation/usage\" resource.type=\"consumer_quota\" resource.label.\"service\"=\"spanner.googleapis.com\" resource.label.\"location\"=\"asia-east1\"",
          "aggregations": [
            {
              "alignmentPeriod": "86400s",
              "crossSeriesReducer": "REDUCE_SUM",
              "groupByFields": [
                "metric.label.quota_metric"
              ],
              "perSeriesAligner": "ALIGN_NEXT_OLDER"
            }
          ],
          "denominatorFilter": "metric.type=\"serviceruntime.googleapis.com/quota/limit\" resource.type=\"consumer_quota\" resource.label.\"service\"=\"spanner.googleapis.com\" resource.label.\"location\"=\"asia-east1\"",
          "denominatorAggregations" : [
            {
              "alignmentPeriod": "86400s",
              "crossSeriesReducer": "REDUCE_SUM",
              "groupByFields": [
                "metric.label.quota_metric"
              ],
              "perSeriesAligner": "ALIGN_NEXT_OLDER"
            }
          ],
          "comparison": "COMPARISON_GT",
          "duration": "60s",
          "thresholdValue": 0.8,
          "trigger": {
            "count": 1
          }
        },
        "displayName": "Ratio quota usage to limit for spanner.googleapis.com by label.quota_metric SUM",
      }
    ],
    "displayName": "Ratio quota usage to limit",
    "enabled": true,
  }

Alerting on quota/exceeded errors

You can create an alerting policy to notify you if any service in your Google Cloud project is reporting a quota exceeded error. You can create this type of policy by using Google Cloud Console or by using the Cloud Monitoring API.

Using the Google Cloud Console

The remainder of the content in this subsection is for the conditions dialog of an alerting policy.

Complete the Target dialog by using the settings in the following table. These setting specify that you want to monitor the time series data for the serviceruntime metric /quota/exceeded for all services in your Google Cloud project, and that you want to group the data by the quota limit.

Target pane
Field

Value
Resource type Enter consumer_quota
Metric Select serviceruntime.googleapis.com/quota/exceeded
Filter

Leave empty.

By leaving this field empty, all available metric data is included in the chart. To restrict the data that is displayed, you can add a filter. For example, to restrict the data to the logging service, add the filter: service = logging.googleapis.com.

Group By

Select quota_metric.

This option groups the data by the type of name of the quota_metric.

Aggregator

Select sum.

The aggregation field is automatically set to sum when data is grouped. This setting defines how the data in the individual time series is combined.

Advanced Aggregation Aligner: count true
Alignment Period: 1 m

This metric is of type GAUGE, meaning each data point in the time series is an instantaneous measurement. The value type is BOOL. A value of true indicates that the quota was exceeded.

For this metric, the aligner settings of count true and count are equivalent.

Complete the Configuration dialog by using the following settings. These settings cause the alerting policy to create an alert if the number of quota-exceeded errors exceeds the value of 0 for 1 m. A value of 0 is selected here because quota exceeded errors are unexpected and an indication that a quota needs to be increased or that a service needs to be modified to reduce the API requests. You might want to use a higher threshold.

Configuration pane
Field

Value
Condition triggers if Any time series violates
Condition is above
Threshold 0
For 1 m

Using the Cloud Monitoring API

You can create this alerting policy using the API method alertPolicies.create. For more information on representing alerting policies in JSON or YAML, see Sample policies.

This example policy monitors the ratio of the Cloud Spanner API quota usage to quota limit. Because combining quota limits might be meaningless, in this example, a location is specified.

{
    "combiner": "OR",
    "conditions": [
      {
        "conditionThreshold": {
          "aggregations": [
            {
              "alignmentPeriod": "60s",
              "crossSeriesReducer": "REDUCE_SUM",
              "groupByFields": [
                "metric.label.quota_metric"
              ],
              "perSeriesAligner": "ALIGN_COUNT_TRUE"
            }
          ],
          "comparison": "COMPARISON_GT",
          "duration": "60s",
          "filter": "metric.type=\"serviceruntime.googleapis.com/quota/exceeded\" resource.type=\"consumer_quota\"",
          "trigger": {
            "count": 1
          }
        },
        "displayName": "Quota exceeded error by label.quota_metric SUM",
      }
    ],
    "displayName": "Quota exceeded policy",
  }

Alerting on the absolute quota/allocation/usage

You can create an alerting policy to notify you if the allocation quota usage for a specific service in your Google Cloud project is exceeding a user-specified threshold. You can create this type of policy by using Google Cloud Console or by using the Cloud Monitoring API.

Using the Google Cloud Console

The remainder of the content in this subsection is for the conditions dialog of an alerting policy.

Complete the Target dialog by using the settings in the following table. These setting specify that you want to monitor the time series data for the serviceruntime metric /quota/allocation/usage for one service in your Google Cloud project, and that you want to group the data by the quota limit.

Target pane
Field

Value
Resource type Enter consumer_quota
Metric Select serviceruntime.googleapis.com/quota/allocation/usage
Filter Add the filter service = iam.googleapis.com.

Replace iam.googleapis.com with the name of the service that you want to monitor. In this example, this common service was selected.

Group By

Select quota_metric.

This option groups the data by the type of name of the quota_metric.

Aggregator

Select sum.

The aggregation field is automatically set to sum when data is grouped. This setting defines how the data in the individual time series is combined.

Advanced Aggregation Aligner: next older
Alignment Period: 1440 m

The aligner is set to next older to include the measured value of this GAUGE metric. The alignment period matches is the sampling interval for this metric.

Complete the Configuration dialog by using the following settings. These settings cause the alerting policy to create an alert if the value of the time series exceeds 2.5 for 1440 m. The value of 1440 m matches the alignment period. The value of 2.5 was selected for this example, because it is slightly higher than the normal value for the test system. You should select the threshold based on the service you are monitoring in combination with an understanding of the expected range of values for the metric.

Configuration pane
Field

Value
Condition triggers if Any time series violates
Condition is above
Threshold 2.5
For 1440 m

Using the Cloud Monitoring API

You can create this alerting policy using the API method alertPolicies.create. For more information on representing alerting policies in JSON or YAML, see Sample policies.

This example policy monitors the ratio of the Cloud Spanner API quota usage to quota limit. Because combining quota limits might be meaningless, in this example, a location is specified.

{
    "combiner": "OR",
    "conditions": [
      {
        "conditionThreshold": {
          "aggregations": [
            {
              "alignmentPeriod": "86400s",
              "crossSeriesReducer": "REDUCE_SUM",
              "groupByFields": [
                "metric.label.quota_metric"
              ],
              "perSeriesAligner": "ALIGN_NEXT_OLDER"
            }
          ],
          "comparison": "COMPARISON_GT",
          "duration": "84600s",
          "filter": "metric.type=\"serviceruntime.googleapis.com/quota/allocation/usage\" resource.type=\"consumer_quota\" resource.label.\"service\"=\"iam.googleapis.com\"",
          "thresholdValue": 2.5,
          "trigger": {
            "count": 1
          }
        },
        "displayName": "Allocation quota usage for iam.googleapis.com by label.quota_metric SUM",
      }
    ],
    "displayName": "Absolute quota usage policy",
  }

Chart examples

Charts display time series data. You can use Metrics Explorer to create a chart. With Metrics Explorer you can either discard the chart when you no longer need it, or you can save it to a dashboard. In the dashboard view, you can add a chart to the dashboard.

To create a chart, you specify what you want to view and how you want to view that data. The remainder of this section contains settings that can be used to create charts. You can also use the settings for Target dialog of an alerting policy to create a chart.

Maximum usage in interval for quota/allocation/usage

To view the time series data for the serviceruntime metric quota/allocation/usage, for all services in your Google Cloud project, where the data is grouped by the name of the quota metric and the maximum value of the metric in a 25-hour interval is displayed, use the following settings:

Target pane
Field

Value
Resource type Enter consumer_quota
Metric Select serviceruntime.googleapis.com/quota/allocation/usage
Filter Add the filter service = iam.googleapis.com.

Replace iam.googleapis.com with the name of the service that you want to monitor. In this example, this common service was selected.

Group By

Select quota_metric.

This option groups the data by the type of name of the quota_metric.

Aggregator

Select sum.

The aggregation field is automatically set to sum when data is grouped. This setting defines how the data in the individual time series is combined.

Advanced Aggregation Aligner: max
Alignment Period: 1500 m
Secondary aggregator:sum

The aligner is set to display the maximum value of the metric measured over the alignment period. The alignment period, 25 hours, is slightly longer than the sampling interval for this metric. The secondary aggregator combines the time series for the different services into a single time series.

Time series for quota/rate/net_usage

To view the time series data for the serviceruntime metric quota/rate/net_usage, for all services in your Google Cloud project, where the data is grouped by the name of the quota metric and the rate of usage is displayed, use the following settings:

Target pane
Field

Value
Resource type Enter consumer_quota
Metric Select serviceruntime.googleapis.com/quota/rate/net_usage
Filter Add the filter service = iam.googleapis.com.

Replace iam.googleapis.com with the name of the service that you want to monitor. In this example, this common service was selected.

Group By

Select quota_metric.

This option groups the data by the type of name of the quota_metric.

Aggregator

Select sum.

The aggregation field is automatically set to sum when data is grouped. This setting defines how the data in the individual time series is combined.

Advanced Aggregation Aligner: rate
Alignment Period: 1 m

By setting the aligner to the value of rate, the values stored in the time series for this DELTA metric are converted into a new time series that stores rate data. The y-axis for the chart has the units of quota per second. The alignment period of 1 minute matches the sampling inteval for this metric.

Time series for quota/limit

To view the time series data for the serviceruntime metric quota/limit, for all services in your Google Cloud project, where the data is aggregated over 25 hours, use the following settings:

Target pane
Field

Value
Resource type Enter consumer_quota
Metric Select serviceruntime.googleapis.com/quota/limit
Filter Add the filter service = iam.googleapis.com.

Replace iam.googleapis.com with the name of the service that you want to monitor. In this example, this common service was selected.

Group By Leave empty.
Aggregator none
Advanced Aggregation Aligner: next older
Alignment Period: 1500 m

The aligner is set to next older to display the measured value of this GAUGE metric. The alignment period, 25 hours, is slightly longer than the sampling interval for this metric.