Alerting policies for the burn rate of your error budget are based on the
time-series selector select_slo_burn_rate
, described in Retrieving SLO
data. There are other time-series selectors, and you can use some of
them as the basis for alerting policies. For a discussion of SLO-based
alerting policies, see Alerting on your burn rate.
You create alerting policies by using the
alertPolicies.create
method. The general use of
this method is documented in Managing alerting policies.
Alerting policies for SLOs are similar to other metric-threshold alerting
policies, but they differ in one specific way: the filter
in the MetricThreshold
specification of the
condition uses a time-series selector instead of a pair of metric and
monitored-resource types.
Conditions for SLO-based alerting policies
An alerting policy must have at least one condition. For a SLO-based
condition, use a MetricThreshold
-type condition.
A metric-threshold condition can contain two pairs of time-series
configurations: filter
and aggregations
. Because SLO data
is retrieved differently than other time-series data, the only field used
in a condition for an SLO is the filter
field.
A condition for an SLO does set the comparison
, thresholdValue
, duration
,
and trigger
fields.
This example creates a condition that is violated when the burn rate exceeds 2 times the normal rate. The structure looks like this:
"conditions": [ { "displayName":"SLO burn rate alert for ${SLO_ID} exceeds 2", "conditionThreshold": { "filter": DATA_RETRIEVAL_FILTER_FOR_SLO, "comparison":"COMPARISON_GT", "thresholdValue": 2, "duration": { "seconds":"0", }, }, } ],
To set the filter
field, you need the resource name of a specific SLO. This
value is of the form
projects/${PROJECT}/services/${SERVICE_ID}/serviceLevelObjectives/${SLO_ID}
.
For information on finding the SLO ID, see Listing SLOs.
To create an alert on burn rate, use the time-series selector
select_slo_burn_rate
. This selector takes two values, the target SLO
and the lookback period. For more information, see
select_slo_burn_rate
.
For example, the following filter gets the burn rate of the target SLO with a 1-hour lookback period:
"filter":"select_slo_burn_rate(\"projects/${PROJECT}/services/${SERVICE_ID}/serviceLevelObjectives/${SLO_ID}\", \"60m\")"
The rest of the alerting policy
To complete the alerting policy, specify values for the remaining fields:
displayName
: A description of the alerting policy.combiner
: Describes the logic for combining conditions. This policy has only one condition, so eitherAND
orOR
works.notificationChannels
: An array of existing notification channels to use when the alerting policy is triggered. For information on finding and creating notification channels, see Notification channels.documentation
: Information that is sent when the condition is violated to help recipients diagnose the problem. For details, seeDocumentation
.
Creating the alerting policy
The following example uses the API to create a burn-rate alerting policy. For information about listing, modifying, and deleting alerting policies, see Managing alerting policies by API.
Protocol
To create the alerting policy by usingcurl
, send a POST
message
to the https://monitoring.googleapis.com/v3/projects/${PROJECT_ID}/alertPolicies
endpoint, and provide the
alerting policy in the request body. The JSON in the request body describes
an alerting policy that uses a threshold condition based on the
select_slo_burn_rate
time-series selector with a one-hour lookback period.
Create a variable to hold the request body:
CREATE_ALERT_POST_BODY=$(cat <<EOF { "displayName":"SLO burn-rate alert for ${SLO_ID} with a threshold of 2", "combiner":"AND", "conditions": [ { "displayName":"SLO burn rate alert for ${SLO_ID} exceeds 2", "conditionThreshold": { "filter":"select_slo_burn_rate(\"projects/${PROJECT}/services/${SERVICE_ID}/serviceLevelObjectives/${SLO_ID}\", \"60m\")", "comparison":"COMPARISON_GT", "thresholdValue": 2, "duration": { "seconds":"0", }, }, } ], "notificationChannels": ["${NOTIFICATION_CHANNEL}", ], "documentation": { "content": "SLO burn for the past 60m exceeded twice the acceptable budget burn rate.", "mime_type": "text/markdown", }, } EOF )
Post the request to the endpoint:
curl --http1.1 --header "Authorization: Bearer ${ACCESS_TOKEN}" --header "Content-Type: application/json" -X POST -d "${CREATE_ALERT_POST_BODY}" https://monitoring.googleapis.com/v3/projects/${PROJECT_ID}/alertPolicies