Creating an alerting policy

Alerting policies for the burn rate of your error budget are based on the time-series selector select_slo_burn_rate, described in Retrieving SLO data. There are other time-series selectors, and you can use some of them as the basis for alerting policies. For a discussion of SLO-based alerting policies, see Alerting on your burn rate.

You create alerting policies by using the alertPolicies.create method. The general use of this method is documented in Managing alerting policies.

Alerting policies for SLOs are similar to other metric-threshold alerting policies, but they differ in one specific way: the filter in the MetricThreshold specification of the condition uses a time-series selector instead of a pair of metric and monitored-resource types.

Conditions for SLO-based alerting policies

An alerting policy must have at least one condition. For a SLO-based condition, use a MetricThreshold-type condition.

A metric-threshold condition can contain two pairs of time-series configurations: filter and aggregations. Because SLO data is retrieved differently than other time-series data, the only field used in a condition for an SLO is the filter field.

A condition for an SLO does set the comparison, thresholdValue, duration, and trigger fields.

This example creates a condition that is violated when the burn rate exceeds 2 times the normal rate. The structure looks like this:

  "conditions": [
    {
      "displayName":"SLO burn rate alert for ${SLO_ID} exceeds 2",
      "conditionThreshold": {
        "filter": DATA_RETRIEVAL_FILTER_FOR_SLO,
        "comparison":"COMPARISON_GT",
        "thresholdValue": 2,
        "duration": {
          "seconds":"0",
        },
      },
    }
  ],

To set the filter field, you need the resource name of a specific SLO. This value is of the form projects/${PROJECT}/services/${SERVICE_ID}/serviceLevelObjectives/${SLO_ID}. For information on finding the SLO ID, see Listing SLOs.

To create an alert on burn rate, use the time-series selector select_slo_burn_rate. This selector takes two values, the target SLO and the lookback period. For more information, see select_slo_burn_rate.

For example, the following filter gets the burn rate of the target SLO with a 1-hour lookback period:

"filter":"select_slo_burn_rate(\"projects/${PROJECT}/services/${SERVICE_ID}/serviceLevelObjectives/${SLO_ID}\", \"60m\")"

The rest of the alerting policy

To complete the alerting policy, specify values for the remaining fields:

• displayName: A description of the alerting policy.
• combiner: Describes the logic for combining conditions. This policy has only one condition, so either AND or OR works.
• notificationChannels: An array of existing notification channels to use when the alerting policy is triggered. For information on finding and creating notification channels, see Notification channels.
• documentation: Information that is sent when the condition is violated to help recipients diagnose the problem. For details, see Documentation.

Creating the alerting policy

The following example uses the API to create a burn-rate alerting policy. For information about listing, modifying, and deleting alerting policies, see Managing alerting policies by API.

Protocol

To create the alerting policy by using curl, send a POST message to the https://monitoring.googleapis.com/v3/projects/${PROJECT_ID}/alertPolicies endpoint, and provide the alerting policy in the request body. The JSON in the request body describes an alerting policy that uses a threshold condition based on the select_slo_burn_rate time-series selector with a one-hour lookback period.

1. Create a variable to hold the request body:

   CREATE_ALERT_POST_BODY=$(cat <<EOF
   {
     "displayName":"SLO burn-rate alert for ${SLO_ID} with a threshold of 2",
     "combiner":"AND",
     "conditions": [
       {
         "displayName":"SLO burn rate alert for ${SLO_ID} exceeds 2",
         "conditionThreshold": {
           "filter":"select_slo_burn_rate(\"projects/${PROJECT}/services/${SERVICE_ID}/serviceLevelObjectives/${SLO_ID}\", \"60m\")",
           "comparison":"COMPARISON_GT",
           "thresholdValue": 2,
           "duration": {
             "seconds":"0",
           },
         },
       }
     ],
     "notificationChannels": ["${NOTIFICATION_CHANNEL}", ],
     "documentation": {
        "content": "SLO burn for the past 60m exceeded twice the acceptable budget burn rate.",
        "mime_type": "text/markdown",
     },
   }
   EOF
   )
   

2. Post the request to the endpoint:

   curl  --http1.1 --header "Authorization: Bearer ${ACCESS_TOKEN}" --header "Content-Type: application/json" -X POST -d "${CREATE_ALERT_POST_BODY}" https://monitoring.googleapis.com/v3/projects/${PROJECT_ID}/alertPolicies