Configure alerts through Cloud Logging

This document explains how to set up, customize, and test alerts for service health events.

Personalized Service Health integrates with Cloud Monitoring log alerts so you can configure and receive alerts. You can create notifications for the following conditions:

When new incidents are reported.
When existing incidents are updated, such as when Google Cloud Support sends a new message.
When incidents for specific Google Cloud products or locations such as regions are created or updated.
When specific details of existing incidents are updated, such as state and relevance.

Since these notifications use log alerts, each notification is associated with a Monitoring incident. Each notification includes links to the incident's page and a direct link to the Service Health dashboard. Learn more about investigating incidents.

Once you have configured an alert, you will receive a notification each time your alerting policy conditions are met. These notifications are subject to Monitoring limits; an example is 20 alerts per policy per day per project.

The following sections show how to configure a log alerting policy and provide examples for common alert scenarios. Log alerts rely on notification channels, which support a variety of integrations including email, SMS, Pub/Sub, webhooks, Slack, and PagerDuty.

To route an alert to a destination other than the options listed earlier, consider using a webhook. For example, you can configure a webhook to ServiceNow. For more information, see Configure webhook in Google Cloud in the ServiceNow documentation.

Alert notification formats

Alert notification formats vary by notification channel.

Email

If you configure the alert to send an email, when the alerting condition is met, a message using the address alerting-noreply@google.com will be delivered to the email address of your choosing.

Slack

If you configure your custom notification to post to Slack, a message from the Cloud Monitoring application will appear in the channel you selected when the event occurs.

Set up an alerting policy

An alert uses an alerting policy, which describes the circumstances under which you want to be alerted and how you want to be notified.

You can configure alerting policies within the Service Health dashboard or using the Google Cloud CLI.

Alerting policies require a Cloud Monitoring notification channel, which defines where an alert is sent. If you don't have a notification channel, you can create one in the Google Cloud console or using the Monitoring API.

Within the Service Health dashboard

See the quickstart.

Using the gcloud CLI

To set up an alerting policy using the gcloud CLI:

Get your notification channel ID.

a. List the notification channel IDs.
```
gcloud config set project PROJECT_ID gcloud beta monitoring channels list
```
b. Look for entries with projects/PROJECT_ID/notificationChannels/. These entries are the notification channel IDs.

Create a policy.json file with the following content:

ALERT_NAME appears on the notification. Example: "Google Cloud SQL incident"

ALERT_CONDITION defines when to alert.

When defining conditions, use the values found at Google Cloud products and locations.

Example alert conditions:

Alert condition	`ALERT_CONDITION` value
Receive alerts for any Cloud SQL event	`jsonPayload.impactedProductIds =~ \"hV87iK5DcEXKgWU2kDri\"`
Receive alerts for any event occurring in zone `us-central1-a`	`jsonPayload.impactedLocations =~ \"us-central1-a\"`

See more examples of alerting conditions.

NOTIFICATION_CHANNEL is the notification channel ID, which you fetched using the list command in the previous step. Example: projects/PROJECT_ID/notificationChannels/885798905074.

The following snippet shows an example of the policy.json file.

{
  "displayName": "ALERT_NAME",
  "combiner": "OR",
  "conditions": [ {
    "displayName": "Log match condition",
    "conditionMatchedLog": {
      "filter": "ALERT_CONDITION",
     } } ],
  "notificationChannels": [ "NOTIFICATION_CHANNEL" ],
}

Create the Cloud Monitoring policy.

gcloud config set project PROJECT_ID
gcloud alpha monitoring policies create --policy-from-file="policy.json"

Customize the content of the alert

You can set the fields inside the notification through the following fields in the JSON file:

labelExtractors: The properties to include in the header of the notification. To customize these properties, see the LabelExtractors guide.
content: The formatting of the email notification. You can use markdown variable substitution, which lets you use labelExtractors as variables. To customize these variables, see this guide.

See example alerting policies and conditions for ways to customize your alerts.

Test your alert

To test the alert, you'll use a sample test log using Cloud Logging. The test log shows the alert logic you previously configured.

Using the Google Cloud console

Do the following:

Go to the reference page for the entries:write method, which lets you write log entries to Cloud Logging. A window that lets you try the method appears.
In the request body, change PROJECT_ID to your own project.

Change any jsonPayload fields in the request body, depending on the alert condition you're testing. You can also check the event_log.proto schema and tweak the values in the log to test specific scenarios that you are interested in.

{
  "entries": [
    {
      "logName": "projects/PROJECT_ID/logs/servicehealth.googleapis.com%2Factivity",
      "resource": {
        "type": "servicehealth.googleapis.com/Event",
        "labels": {
          "resource_container": "PROJECT_ID",
          "location": "global",
          "event_id": "U4AqrjwFQYi5fFBmyAX-Gg"
        }
      },
      "labels": {
        "servicehealth.googleapis.com/new_event": "true",
        "servicehealth.googleapis.com/updated_fields": "[]"
      },
      "jsonPayload": {
        "@type": "type.googleapis.com/google.cloud.servicehealth.logging.v1.EventLog",
        "category": "INCIDENT",
        "title": "EXAMPLE... NOT REAL INCIDENT",
        "description": "EXAMPLE FOR TESTING, NOT REAL INCIDENT. We are experiencing an issue with Google Cloud infrastructure components at us-east1, australia-southeast2. Our engineering team continues to investigate the issue. We apologize to all who are affected by the disruption.",
        "updateTime": "2022-07-15T22:26:40Z",
        "endTime": "2022-07-16T22:13:20Z",
        "impactedLocations": "['us-east1', 'australia-southeast2']",
        "impactedProducts": "['CloudSQL']",
        "impactedProductIds": "['hV87iK5DcEXKgWU2kDri']",
        "startTime": "2022-07-13T12:26:40Z",
        "nextUpdateTime": "2022-07-13T16:26:40Z",
        "state": "ACTIVE",
        "detailedState": "CONFIRMED",
        "relevance": "RELATED"
      }
    }
  ]
}

Click Execute. A window where Google APIs Explorer requests access to your account appears.
Allow Google APIs Explorer access. The window with the request body shows the following screenshot, which indicates success.
Wait a few minutes, then verify that the alert fired.
1. In the Google Cloud console, go to Cloud Monitoring > Incidents and look for the alert.
2. Check if you received an alert on the notification channel used by the alerting policy.

If you need to test the alert again, wait at least 5 minutes before doing so.

Using gcloud

You can also create a test log entry by calling the Cloud Logging API using the gcloud command.

(Optional) Check your current project.
```
gcloud config list
```

Set your current project.

gcloud_name config set project PROJECT_ID

Write the test log entry.

gcloud logging write --payload-type=json
LOG_NAME
'{ "category": "INCIDENT", "relevance": "IMPACTED", "@type": "type.googleapis.com/google.cloud.servicehealth.logging.v1.EventLog", "description": "This is a test log entry"}'

Wait a few minutes, then verify that the alert fired.