Create and manage notification channels

This document describes how to configure notification channels by using the Google Cloud console. Cloud Monitoring uses these channels to notify you, or your on-call team, when an alerting policy fires. When you create an alerting policy, you select who is notified by making selections from the list of configured notification channels. For example, you might configure alerts that monitor Compute Engine instances to publish a Pub/Sub topic and to notify the Slack channel for the on-call team.

If your preferred notification channel isn't supported, consider creating a pipeline that relies on sending your notifications to Pub/Sub. For a Python example that uses Flask, see Creating custom notifications with Cloud Monitoring and Cloud Run. For other examples, see the cloud-alerting-notification-forwarding Git repository.

To configure notification channels by using the Cloud Monitoring API, see Create and manage notification channels by API.

For information about alerting-policy notifications, see the following pages:

By default, an alerting policy sends a notification only when an incident is created. To receive a notification when the incident is opened and when it is closed, edit the alerting policy and, in the notifications section, select Notify on incident closure.

Before you begin

To get the permissions that you need to view and configure notification channels by using the Google Cloud console, ask your administrator to grant you the Monitoring Editor (roles/monitoring.editor) IAM role on your project. For more information about granting roles, see Manage access.

You might also be able to get the required permissions through custom roles or other predefined roles.

For more information about Cloud Monitoring roles, see Control access with Identity and Access Management.

Create a notification channel

When you are creating an alerting policy, you can select any configured notification channel and add it to your policy. You can pre-configure your notification channels, or you can configure them as part of the process of creating an alerting policy. For more information, see Create a channel on demand.

To create a notification channel by using the Google Cloud console, follow the channel-specific instructions contained in the following table:

Email

To add an email notification channel, do the following:

  1. In the Google Cloud console, select Monitoring

    Go to Monitoring

  2. Click Alerting and then click Edit notification channels.
  3. In the Email section, click Add new.
  4. Enter a single email address and a description.
  5. Click Save.

If you use a group email address as the notification channel for an alerting policy, then configure the group to accept mail from alerting-noreply@google.com.

You can create email channels during the creation of an alerting policy. For more information, see Create a channel on demand.

Mobile App

Use the Google Cloud console Mobile App to monitor your Google Cloud console resources and Monitoring information from anywhere. Google Cloud console Mobile App notifications are either sent to a specific device or to a specific user:

List of Google Cloud console Mobile App notification channels.

  • Device indicates that notifications are sent only to the specific device that created the notification channel. For device-scoped notification channels, the Display name field includes device information.
  • User indicates that the notifications are sent to all of your devices that have the Google Cloud console Mobile App installed.

Cloud Monitoring determines the notification scope when the channel is created. You can't select or change the scope.

To configure a Google Cloud console Mobile App notification channel for a specific Google Cloud project, do the following:

  1. Install the Google Cloud console Mobile App from your mobile device's app store.

  2. Select a project for viewing in the Google Cloud console Mobile App.

    After you select a project, a data exchange between the app and the selected Google Cloud project occurs. A notification channel is created when one doesn't exist, and after a few minutes, this channel is listed under the Mobile Devices section of the Notification channels page.

To add your mobile device as a notification channel for an alerting policy, in the alerting Notifications section, select Google Cloud console (mobile) and then choose your mobile device from the list.

PagerDuty

Integration with PagerDuty allows for one-way or two-way synchronization with Monitoring. Independent of your configuration, the following are true:

  • If an incident is created in Monitoring, then an incident is opened in PagerDuty.
  • You can't use PagerDuty to close an incident in Monitoring.

If you use one-way synchronization and if you resolve the incident in PagerDuty, then the state of the incident in PagerDuty is decoupled from the state of the incident in Monitoring. In effect, if you resolve an incident in PagerDuty, then the incident is permanently closed in PagerDuty and can't be reopened.

If you use two-way synchronization, then Monitoring controls the state shown by PagerDuty. If you resolve the incident in PagerDuty and if Monitoring has the incident open, then the incident is reopened in PagerDuty.

To set up PagerDuty notifications, do the following:

  1. In PagerDuty: Create a PagerDuty account at the PagerDuty site.
  2. Add the PagerDuty notification channel:
    1. In the Google Cloud console, select Monitoring
      Go to Monitoring
    2. Click Alerting and then click Edit notification channels.
    3. In the PagerDuty section, click Add new.
    4. Enter the Display Name. This name should match the name provided to PagerDuty when you added the integration.
    5. Enter the Integration Service Key generated by PagerDuty into the Service Key field.
    6. Click Save.
  3. (Optional) If you want to configure two-way synchronization, then do the following:
    1. Open PagerDuty.
    2. Select Configuration, select Services, and then select the service name you entered when configuring the integration.
    3. Click Edit Settings, select Create incidents, and then clear Create alerts and incidents.

When you create an alerting policy, select PagerDuty in the Notifications section and choose your PagerDuty configuration.

The JSON packet for PagerDuty has the following format:

 {
     "description": A string of various fields in the incident,
     "details": JSON payload with schema version 1.2.
 }

To view an example and the schema for the details field, expand the following sections.

Deprecation policy

The payload schema is subject to the Google Cloud deprecation policy outlined in Section 1.4(d) of the Google Cloud Platform Terms of Service. Note that the schema does not control the formats of generated field values, and these formats can change without notice. For example, incident.summary, incident.documentation.content, and incident.url are meant to include data pertaining to their fields, but the schema doesn't have constraints to guarantee accurate parsing of these fields. You can consume the value as a whole and expect that it adheres to the deprecation policy, but don't rely on parsing the generated fields.

SMS

To configure SMS notifications, do the following:

  1. In the Google Cloud console, select Monitoring

    Go to Monitoring

  2. Click Alerting and then click Edit notification channels.
  3. In the SMS section, click Add new.
  4. Complete the dialog and click Save.

When you set up your alerting policy, select the SMS notification type and choose a verified phone number from the list.

Slack

The Monitoring Slack integration allows your alerting policies to post to a Slack channel when a new incident is created. To set up Slack notifications, do the following:

  1. In Slack: Create a Slack workspace and channel at the Slack site. Record the channel URL.

  2. In the Google Cloud console, select Monitoring:

    Go to Monitoring

  3. Click Alerting and then click Edit notification channels.

  4. In the Slack section, click Add new to open the Slack sign-in page:

    1. Select your Slack workspace.
    2. Click Allow to enable Cloud Monitoring access to your Slack workspace. This action takes you back to the Monitoring configuration page for your notification channel.
    3. In the Slack Channel Name field, enter the name of the Slack channel you want to use for notifications.
    4. In the Cloud Alerting Display Name field, enter a short descriptive statement. Monitoring displays the value of this field on the Notifications channel page.
    5. (Optional) To test the connection between Cloud Monitoring and your Slack workspace, click Send test notification. If the connection is successful, then you see a message This is a test alert notification... in the Slack notification channel that you specified. Check the notification channel to confirm receipt.

  5. If the Slack channel you want to use for notifications is a private channel, then you must manually invite the Monitoring app to the channel:

    1. Open Slack.

    2. Go to the channel you specified as your Monitoring notification channel.

    3. Invite the Monitoring app to the channel by entering and sending the following message in the channel:

      /invite @Google Cloud Monitoring

      Be sure you invite the Monitoring app to the private channel you specified when creating the notification channel in Monitoring. Inviting the Monitoring app to public channels is optional.

    When you create an alerting policy, select Slack in the Notifications section and choose your Slack configuration.

Webhooks

To configure Webhooks notifications, do the following:

  1. The webhook handler: Identify the public endpoint URL to receive webhook data from Monitoring.
  2. In the Google Cloud console, select Monitoring

    Go to Monitoring

  3. Click Alerting and then click Edit notification channels.
  4. In the Webhook section, click Add new.
  5. Complete the dialog.
  6. Click Test Connection to send a test payload to the Webhook endpoint. You can go to the receiving endpoint to verify delivery.
  7. Click Save.

When you create an alerting policy, select Webhook in the Notifications section and choose your webhook configuration. The notifications sent by Error Reporting follow the 1.0 schema while notifications sent by Monitoring follow the 1.2 schema:

Basic authentication

In addition to the webhook request sent by Cloud Monitoring, basic authentication utilizes the HTTP specification for the username and password. Cloud Monitoring requires your server to return a 401 response with the proper WWW-Authenticate header. For more information about basic authentication, see the following:

Token authentication

Token Authentication requires a query string parameter in the endpoint URL and a key that the server expects to be secret between itself and Monitoring. The following is a sample URL that includes a token:

https://www.myserver.com/stackdriver-hook?auth_token=1234-abcd

If Monitoring posts an incident to the endpoint URL, your server can validate the attached token. This method of authentication is most effective when used with SSL/TLS to encrypt the HTTP request preventing snoopers from learning the token.

For an example server in Python, see this sample server.

Deprecation policy

The payload schema is subject to the Google Cloud deprecation policy outlined in Section 1.4(d) of the Google Cloud Platform Terms of Service. Note that the schema does not control the formats of generated field values, and these formats can change without notice. For example, incident.summary, incident.documentation.content, and incident.url are meant to include data pertaining to their fields, but the schema doesn't have constraints to guarantee accurate parsing of these fields. You can consume the value as a whole and expect that it adheres to the deprecation policy, but don't rely on parsing the generated fields.

Pub/Sub

This section describes how to configure Pub/Sub notification channels. For redundancy, we recommend that you create multiple notification channels. Pairing Pub/Sub with Google Cloud console Mobile App, PagerDuty, Webhooks, or Slack is recommended, because Pub/Sub uses a different delivery mechanism.

To configure a Pub/Sub notification channel:

  1. Enable the Pub/Sub API and create a topic
  2. Configure the notification channel for a topic
  3. Authorize a service account
  4. Set the notification channel in an alerting policy

To understand the data schema, see Schema example.

To receive the alert notifications, you must also create a Pub/Sub subscription. When the subscription is in a different project than the topic, create a service account in the subscriber project, and grant it the role of roles/pubsub.subscriber for the topic.

Before you begin

When you add the first notification channel for a Google Cloud project, Cloud Monitoring creates a service account for that project. It also grants the Identity and Access Management role Monitoring Notification Service Agent to the service account. This service account lets Monitoring send notifications to Pub/Sub-based notification channels in this project.

The service account has the following format:

service-PROJECT_NUMBER@gcp-sa-monitoring-notification.iam.gserviceaccount.com

The name of the notifications service account includes a unique Google Cloud project number. You can find the project name, ID, and number on the project's dashboard in the Google Cloud console, or you can retrieve it with the following command:

gcloud projects describe PROJECT_ID --format="value(project_number)"

You can view and edit the service account by using the Google Cloud console or the Google Cloud CLI.

Enable the Pub/Sub API and create a topic

  1. Enable the Pub/Sub API for your Google Cloud project:
    Enable Pub/Sub API

    1. Select the Google Cloud project in which you plan to create the Pub/Sub topic.
    2. Click Enable when shown. No action is required when API enabled is shown.

  2. Create a Pub/Sub topic. For information about how to create a topic, see Create a topic. For example, to create a topic called notificationTopic by using the Google Cloud CLI, run the following command: 

    gcloud pubsub topics create notificationTopic

Next, configure your Pub/Sub notification channels.

Configure notifications for a topic

To create a Pub/Sub notification channel, you can use the Monitoring API, the Google Cloud CLI, or the Google Cloud console. After you create the notification channel, authorize the notifications service account to publish each topic that you are using as a notification channel.

For information about using the Monitoring API or the Google Cloud CLI to create the notification channel, see Creating channels.

To use the Google Cloud console to create the notification channel, do the following:

  1. In the Google Cloud console, select Monitoring

    Go to Monitoring

  2. Select the Google Cloud project that contains the Pub/Sub topic you created.
  3. Click Alerting and then click Edit notification channels.

  4. In the Pub/Sub section, click Add new.

    The Created Pub/Sub Channel dialog displays the name of the service account that Monitoring created.

  5. Authorize the service account. Your service account can publish all topics or specific topics:

    • To publish all topics, select Manage roles, and add the role of Pub/Sub Publisher.

      After you complete these steps that configure the notification channel, skip the next section, which is titled Authorize service account, and proceed to Set the notification channel in an alerting policy.

    • To publish specific topics, proceed to the next step and authorize the service account to publish specific topics after you complete configuring the notification channel. For instructions to authorize the service account, see Authorize service account.
  6. Enter a display name for your channel and the Pub/Sub topic name.
  7. (Optional) To verify that the channel is correctly configured, click Send test notification.
  8. Select Add channel.

Next, authorize the service account.

Authorize service account

Authorization lets the notifications service account publish each Pub/Sub topic that you are using as a notification channel. This section describes how you can do the following:

  • Authorize a service account for a specific topic.
  • Authorize a service account for all topics.

Authorize a service account for a specific topic

You can authorize a service account to publish a specific topic by using the Google Cloud console and the Google Cloud CLI. This section describes both approaches.

To authorize your service account for a specific topic by using the Google Cloud console, do the following:

  1. Go to the Topics page for Pub/Sub:
    Go to Topics
  2. Select the topic.
  3. In the Permissions tab, select Add principal.
  4. In the New principal field, enter the name of the notifications service account. The service account has the following naming convention: 
    service-PROJECT_NUMBER@gcp-sa-monitoring-notification.iam.gserviceaccount.com
  5. Select the role Pub/Sub Publisher, and then select Save.

To authorize your service account for a specific topic by using the Google Cloud CLI, grant the pubsub.publisher IAM role for the topic to the service account. For example, the following command configures the IAM role for the notificationTopic topic:

gcloud pubsub topics add-iam-policy-binding \
projects/PROJECT_NUMBER/topics/notificationTopic --role=roles/pubsub.publisher \
--member=serviceAccount:service-PROJECT_NUMBER@gcp-sa-monitoring-notification.iam.gserviceaccount.com

A response to a successful execution of the add-iam-policy-binding command is like the following:

 Updated IAM policy for topic [notificationTopic].
 bindings:
 ‐ members:
    ‐ serviceAccount:service-PROJECT_NUMBER@gcp-sa-monitoring-notification.iam.gserviceaccount.com
    role: roles/pubsub.publisher
 etag: BwWcDOIw1Pc=
 version: 1

For more information, see the pubsub topics add-iam-policy-binding reference page.

Authorize a service account for all topics

To authorize your service account for all topics, do the following:

  1. Go to the IAM page:
    Go to IAM
  2. Select Include Google-provided role grants as shown in the following image:
    Select the Include Google-provided role grants option.
  3. Search for the service account that has the following format: 
    service-PROJECT_NUMBER@gcp-sa-monitoring-notification.iam.gserviceaccount.com

    Reload the page when the service account isn't listed.

  4. Select Edit for the notifications service account, select Add another role, and then add the role Pub/Sub Publisher.

Next, set the notification channel in an alerting policy.

Set the notification channel in an alerting policy

To use a Pub/Sub notification channel in an alerting policy, select Pub/Sub as the channel type, and then select the topic.

Schema example

To view an example JSON packet and the schema, expand the following sections.

Deprecation policy

The payload schema is subject to the Google Cloud deprecation policy outlined in Section 1.4(d) of the Google Cloud Platform Terms of Service. Note that the schema does not control the formats of generated field values, and these formats can change without notice. For example, incident.summary, incident.documentation.content, and incident.url are meant to include data pertaining to their fields, but the schema doesn't have constraints to guarantee accurate parsing of these fields. You can consume the value as a whole and expect that it adheres to the deprecation policy, but don't rely on parsing the generated fields.

Create a channel on demand

When adding a notification channel to an alerting policy, you must select a channel from a list. To update the list of options when you are creating an alerting policy, use the following process:

  1. In the notification dialog, click Manage Notification Channels. You're taken to the Notification channels window in a new browser tab.
  2. To add a new notification channel, locate the channel type, click Add new, and then follow the channel-specific instructions contained in the previous table.

  3. Return to the original tab, and in the notification dialog, click Refresh .

    Notification dialog displaying the refresh and manage channels buttons.

  4. Select the notification channel from the updated list.

Test a notification channel

Cloud Monitoring doesn't support a test option for notification channels. However, you can verify that a notification channel is properly configured by creating an alerting policy and configuring the condition to trigger.

For example, you can verify your notification channel configurations by using a procedure similar to the following:

  1. If you don't have a Compute Engine instance in your Google Cloud project, then create one.

  2. Create an alerting policy to monitor the CPU utilization of that instance.

    If your Google Cloud project contains multiple instances, then add a filter to select one instance.

  3. Configure the Condition trigger fields as follows:

    1. Select Threshold for the Condition type field.
    2. Select Below threshold for the Threshold position field.
    3. Select a value that is larger than the CPU utilization for your instance. For example, if the chart displays the CPU utilization to be about 5%, then set the threshold to 10%.
    4. Expand Advanced Options and select No retest for the Restest window field.

    These settings configure the policy to trigger when the CPU utilization of a Compute Engine instance is less than the specified threshold. Because you selected the threshold to be larger than the CPU utilization of your instance, the condition triggers.

  4. Add the notification channels to your policy.

  5. Expand Incident autoclose duration and select 30 min.

  6. Enter a policy name and click Create policy.

  7. After the condition to triggers, verify that you received a notification on each notification channel.

  8. Delete the alerting policy and the Compute Engine instance that you created. The incident is closed automatically after the autoclose duration expires.

  9. If you created a VM for this test, then delete it.

Edit and delete notification channels

To edit or delete a notification channel by using the Google Cloud console, do the following:

  1. In the Google Cloud console, select Monitoring, or click the following button:

    Go to Monitoring

  2. In the Monitoring navigation pane, click Alerting.

  3. Click Edit notification channels.

    The Notification channels dashboard contains a section for each type of notification channel. Each section lists all configurations for that type:

    • To modify an entry, click Edit . Click Save after your changes are complete.
    • To delete an entry, click Delete . Click Delete in the confirmation dialog.

View notification channel logs

You can use the Logs Explorer to view notification channel errors:

  1. To begin using the Google Cloud console to view the errors, navigate to the Logs Explorer:

    Go to the Logs Explorer

  2. Enter and run your query. For queries specific to notification channel errors, see Cloud Monitoring queries queries.