Example alerting policies

This document shows examples of alerting policies that customize the content of alert notifications.

Before you begin

Be familiar with setting up alerts for Personalized Service Health.

See the Service Health log schema reference for the fields you can set in the alert notifications and conditions.

Example alert conditions

The following sections show examples of fine-grained alert conditions that you can use for Cloud Logging or Terraform.

Alert on a new incident for a specific Google Cloud product

Set ALERT_CONDITION to the following:

labels."servicehealth.googleapis.com/new_event"=true AND jsonPayload.detailedCategory = \"CONFIRMED_INCIDENT\" AND jsonPayload.impactedProductIds =~ \"hV87iK5DcEXKgWU2kDri\" AND jsonPayload.@type = \"type.googleapis.com/google.cloud.servicehealth.logging.v1.EventLog\"

The product name is in the common Google Cloud format. See Google Cloud Service Health for an example of the format.

Alert on a new incident for a specific region

Set ALERT_CONDITION to the following:

labels."servicehealth.googleapis.com/new_event"=true AND jsonPayload.detailedCategory = \"CONFIRMED_INCIDENT\" AND jsonPayload.impactedLocations =~ \"us-central1\" AND jsonPayload.@type = \"type.googleapis.com/google.cloud.servicehealth.logging.v1.EventLog\"

The location name is in the common Google Cloud format. See Google Cloud Service Health for an example of the format.

Alert on a new emerging incident for a specific product

Set ALERT_CONDITION to the following:

labels."servicehealth.googleapis.com/new_event"=true AND jsonPayload.detailedCategory = \"EMERGING_INCIDENT\" AND jsonPayload.impactedProductIds =~ \"ix7u9beT8ivBdjApTif3\" AND jsonPayload.@type = \"type.googleapis.com/google.cloud.servicehealth.logging.v1.EventLog\"

Alert on any update or change to an incident

Set ALERT_CONDITION to the following:

jsonPayload.detailedCategory = \"CONFIRMED_INCIDENT\" AND jsonPayload.@type = \"type.googleapis.com/google.cloud.servicehealth.logging.v1.EventLog\"

Alert on any update on confirmed incidents

Set ALERT_CONDITION to the following:

jsonPayload.state =~ \"ACTIVE\" AND jsonPayload.detailedCategory = \"CONFIRMED_INCIDENT\" AND jsonPayload.detailedState != \"EMERGING\" AND jsonPayload.@type = \"type.googleapis.com/google.cloud.servicehealth.logging.v1.EventLog\"

Alert on state or relevance updates

Set ALERT_CONDITION to the following:

jsonPayload.detailedCategory = \"CONFIRMED_INCIDENT\" AND "labels.servicehealth.googleapis.com/updated_fields" =~ \"'state'\" OR labels."servicehealth.googleapis.com/updated_fields" =~ \"'relevance'\" AND jsonPayload.@type = \"type.googleapis.com/google.cloud.servicehealth.logging.v1.EventLog\"

Alert notification with full incident details

The following alert notification provides full incident details.

{
  "displayName": "ALERT_NAME",
  "combiner": "OR",
  "conditions": [ {
    "displayName": "Log match condition",
    "conditionMatchedLog": {
      "filter": "ALERT_CONDITION",
      "labelExtractors": {
        "description": "EXTRACT(jsonPayload.description)",
        "impactedLocations": "EXTRACT(jsonPayload.impactedLocations)",
        "impactedProducts": "EXTRACT(jsonPayload.impactedProducts)",
        "startTime": "EXTRACT(jsonPayload.startTime)",
        "state": "EXTRACT(jsonPayload.state)"
        "detailedState": "EXTRACT(jsonPayload.detailedState)"
      } } } ],
  "documentation": {
  "content": "# View details at [Service Health dashboard](https://console.cloud.google.com/servicehealth/incidentDetails/projects%2F${resource.labels.resource_container}%2Flocations%2F${resource.labels.location}%2Fevents%2F${resource.labels.event_id}?project=${resource.labels.resource_container}) \n## Description \n${log.extracted_label.description} \n## Impacted locations \n${log.extracted_label.impactedLocations} \n## Impacted products \n${log.extracted_label.impactedProducts} \n## Incident start time \n${log.extracted_label.startTime} \n## Event state \n${log.extracted_label.state} \n## Event detailed state \n ${log.extracted_label.detailedState}",
    "mimeType": "text/markdown"
  },
  "notificationChannels": [ "NOTIFICATION_CHANNEL" ],
  "enabled": true,
  "alertStrategy": { "notificationRateLimit": { "period": "300s" }, "autoClose": "1800s" }
}

The labelExtractors and content fields in the example include all information available for the incident.

The following example provides a link to the Service Health dashboard.

{
  "displayName": "ALERT_NAME",
  "combiner": "OR",
  "conditions": [ {
    "displayName": "Log match condition",
    "conditionMatchedLog": {
      "filter": "ALERT_CONDITION",
      "labelExtractors": {
        "description": "EXTRACT(jsonPayload.description)",
        "impactedLocations": "EXTRACT(jsonPayload.impactedLocations)",
        "impactedProducts": "EXTRACT(jsonPayload.impactedProducts)",
        "startTime": "EXTRACT(jsonPayload.startTime)",
        "state": "EXTRACT(jsonPayload.state)",
        "detailedState": "EXTRACT(jsonPayload.detailedState)"
      } } } ],
  "documentation": {
  "content": "# View details at [Service Health dashboard](https://console.cloud.google.com/servicehealth/eventDetails/projects%2F${resource.labels.resource_container}%2Flocations%2F${resource.labels.location}%2Fevents%2F${resource.labels.event_id})",
    "mimeType": "text/markdown"
  },
  "notificationChannels": [ "NOTIFICATION_CHANNEL" ],
  "enabled": true,
  "alertStrategy": { "notificationRateLimit": { "period": "300s" }, "autoClose": "1800s" }
}