Create and manage notification channels

This document describes how to configure notification channels by using the Google Cloud console. Cloud Monitoring uses these channels to notify you, or your on-call team, when an alerting policy fires. When you create an alerting policy, you select who is notified by making selections from the list of configured notification channels. For example, you might configure alerting policies that monitor Compute Engine instances to publish a Pub/Sub topic and to notify the Slack channel for the on-call team.

For more information, see Notification latency and Notifications per incident.

This document describes how to use the Google Cloud console. If you use the Cloud Monitoring API, then see Create and manage notification channels by API.

Before you begin

To get the permissions that you need to view and configure notification channels by using the Google Cloud console, ask your administrator to grant you the Monitoring Editor (roles/monitoring.editor) IAM role on your project. For more information about granting roles, see Manage access.

You might also be able to get the required permissions through custom roles or other predefined roles.

For more information about Cloud Monitoring roles, see Control access with Identity and Access Management.

Create a notification channel

When you are creating an alerting policy, you can select any configured notification channel and add it to your policy. You can pre-configure your notification channels, or you can configure them as part of the process of creating an alerting policy. For more information, see Create a channel on demand.

To create a notification channel by using the Google Cloud console, follow the channel-specific instructions contained in the following table:

Email

To add an email notification channel, do the following:

  1. In the navigation panel of the Google Cloud console, select Monitoring, and then select  Alerting:

    Go to Alerting

  2. Click Edit notification channels.
  3. In the Email section, click Add new.
  4. Enter a single email address and a description.
  5. Click Save.

If you use a group email address as the notification channel for an alerting policy, then configure the group to accept mail from alerting-noreply@google.com.

You can create email channels during the creation of an alerting policy. For more information, see Create a channel on demand.

Mobile App

Use the Google Cloud console Mobile App to monitor your Google Cloud console resources and Monitoring information from anywhere. Google Cloud console Mobile App notifications are either sent to a specific device or to a specific user:

List of Google Cloud console Mobile App notification channels.

  • Device indicates that notifications are sent only to the specific device that created the notification channel. For device-scoped notification channels, the Display name field includes device information.
  • User indicates that the notifications are sent to all of your devices that have the Google Cloud console Mobile App installed.

Cloud Monitoring determines the notification scope when the channel is created. You can't select or change the scope.

To configure a Google Cloud console Mobile App notification channel for a specific Google Cloud project, do the following:

  1. Install the Google Cloud console Mobile App from your mobile device's app store.
  2. Select a project for viewing in the Google Cloud console Mobile App.

    After you select a project, a data exchange between the app and the selected Google Cloud project occurs. A notification channel is created when one doesn't exist, and after a few minutes, this channel is listed under the Mobile Devices section of the Notification channels page.

To add your mobile device as a notification channel for an alerting policy, in the alerting Notifications section, select Google Cloud console (mobile) and then choose your mobile device from the list.

PagerDuty

Integration with PagerDuty allows for one-way or two-way synchronization with Monitoring. Independent of your configuration, the following are true:

  • If an incident is created in Monitoring, then an incident is opened in PagerDuty.
  • You can't use PagerDuty to close an incident in Monitoring.

If you use one-way synchronization and if you resolve the incident in PagerDuty, then the state of the incident in PagerDuty is decoupled from the state of the incident in Monitoring. In effect, if you resolve an incident in PagerDuty, then the incident is permanently closed in PagerDuty and can't be reopened.

If you use two-way synchronization, then Monitoring controls the state shown by PagerDuty. If you resolve the incident in PagerDuty and if Monitoring has the incident open, then the incident is reopened in PagerDuty.

To set up PagerDuty notifications, do the following:

  1. In your PagerDuty account, configure the integration in a new service or add the integration to an existing service.
    1. Create an account or sign in to an existing account.
    2. Go to the Services page.
      • To create a service, do the following:
        1. Click + New Service.
        2. Enter a service name and description, and then click Next.
        3. Configure an escalation policy, and then click Next.
        4. If your PagerDuty account supports alert grouping and transient alerts, configure them, and then click Next.
        5. On the integrations setup page, in the Search for an integration(s) toolbar, enter Events API v1, and then click Create Service. You are then returned to the Integrations panel of your new service.
      • To edit an existing service, do the following:
        1. Select a service from your service directory.
        2. On your service page, select the Integrations panel.
        3. Click + Add another integration to go to the integrations setup page.
        4. On the integrations setup page, in the Search for an integration(s) toolbar, enter Events API v1, and then click Add. You are then returned to the Integrations panel of your edited service.
    3. On your service's Integrations panel, find the listing for your new integration, expand its menu, and then copy the integration key.
  2. In Cloud Monitoring, add the PagerDuty notification channel:
    1. In the navigation panel of the Google Cloud console, select Monitoring, and then select  Alerting:

      Go to Alerting

    2. Click Edit notification channels.
    3. In the PagerDuty section, click Add new.
    4. Enter the Display Name. This name should match the name of the PagerDuty service where you configured your new integration in Step 1.
    5. Enter the integration key that you copied in Step 1 into the Service Key field.
    6. Click Save.
  3. Optional: If you want to configure two-way synchronization, then do the following:
    1. Go to your PagerDuty account.
    2. Select Services, and then select the service name you entered when configuring the integration.
    3. Select your service's Integrations panel.
    4. Go to the Alert and Incident Settings section and then click Edit.
    5. Select Create both incidents and alerts and then save your changes.

When you create an alerting policy, select PagerDuty in the Notifications section and choose your PagerDuty configuration.

The JSON packet for PagerDuty has the following format:

 {
     "description": A string of various fields in the incident,
     "details": JSON payload with schema version 1.2.
 }

To view an example and the schema for the details field, expand the following sections.

Deprecation policy

The payload schema is subject to the Google Cloud deprecation policy outlined in Section 1.4(d) of the Google Cloud Platform Terms of Service. Note that the schema does not control the formats of generated field values, and these formats can change without notice. For example, incident.summary, incident.documentation.content, and incident.url are meant to include data pertaining to their fields, but the schema doesn't have constraints to guarantee accurate parsing of these fields. You can consume the value as a whole and expect that it adheres to the deprecation policy, but don't rely on parsing the generated fields.

SMS

To configure SMS notifications, do the following:

  1. In the navigation panel of the Google Cloud console, select Monitoring, and then select  Alerting:

    Go to Alerting

  2. Click Edit notification channels.
  3. In the SMS section, click Add new.
  4. Complete the dialog and click Save.

When you set up your alerting policy, select the SMS notification type and choose a verified phone number from the list.

Slack

The Monitoring Slack integration allows your alerting policies to post to a Slack channel when a new incident is created. To set up Slack notifications, do the following:

  1. In Slack: Create a Slack workspace and channel at the Slack site. Record the channel URL.

  2. In the navigation panel of the Google Cloud console, select Monitoring, and then select  Alerting:

    Go to Alerting

  3. Click Edit notification channels.

  4. In the Slack section, click Add new to open the Slack sign-in page:

    1. Select your Slack workspace.
    2. Click Allow to enable Cloud Monitoring access to your Slack workspace. This action takes you back to the Monitoring configuration page for your notification channel.
    3. In the Slack Channel Name field, enter the name of the Slack channel you want to use for notifications.
    4. In the Cloud Alerting Display Name field, enter a short descriptive statement. Monitoring displays the value of this field on the Notifications channel page.
    5. (Optional) To test the connection between Cloud Monitoring and your Slack workspace, click Send test notification. If the connection is successful, then you see a message This is a test alert notification... in the Slack notification channel that you specified. Check the notification channel to confirm receipt.
  5. If the Slack channel you want to use for notifications is a private channel, then you must manually invite the Monitoring app to the channel:

    1. Open Slack.
    2. Go to the channel you specified as your Monitoring notification channel.

    3. Invite the Monitoring app to the channel by entering and sending the following message in the channel:

      /invite @Google Cloud Monitoring

      Be sure you invite the Monitoring app to the private channel you specified when creating the notification channel in Monitoring. Inviting the Monitoring app to public channels is optional.

    When you create an alerting policy, select Slack in the Notifications section and choose your Slack configuration.

Webhooks

To configure Webhooks notifications, do the following:

  1. The webhook handler: Identify the public endpoint URL to receive webhook data from Monitoring.
  2. In the navigation panel of the Google Cloud console, select Monitoring, and then select  Alerting:

    Go to Alerting

  3. Click Edit notification channels.
  4. In the Webhook section, click Add new.
  5. Complete the dialog.
  6. Click Test Connection to send a test payload to the Webhook endpoint. You can go to the receiving endpoint to verify delivery.
  7. Click Save.

When you create an alerting policy, select Webhook in the Notifications section and choose your webhook configuration. The notifications sent by Error Reporting follow the 1.0 schema while notifications sent by Monitoring follow the 1.2 schema:

Basic authentication

In addition to the webhook request sent by Cloud Monitoring, basic authentication utilizes the HTTP specification for the username and password. Cloud Monitoring requires your server to return a 401 response with the proper WWW-Authenticate header. For more information about basic authentication, see the following:

Token authentication

Token Authentication requires a query string parameter in the endpoint URL and a key that the server expects to be secret between itself and Monitoring. The following is a sample URL that includes a token:

https://www.myserver.com/stackdriver-hook?auth_token=1234-abcd

If Monitoring posts an incident to the endpoint URL, your server can validate the attached token. This method of authentication is most effective when used with SSL/TLS to encrypt the HTTP request preventing snoopers from learning the token.

For an example server in Python, see this sample server.

Deprecation policy

The payload schema is subject to the Google Cloud deprecation policy outlined in Section 1.4(d) of the Google Cloud Platform Terms of Service. Note that the schema does not control the formats of generated field values, and these formats can change without notice. For example, incident.summary, incident.documentation.content, and incident.url are meant to include data pertaining to their fields, but the schema doesn't have constraints to guarantee accurate parsing of these fields. You can consume the value as a whole and expect that it adheres to the deprecation policy, but don't rely on parsing the generated fields.

Pub/Sub

This section describes how to configure Pub/Sub notification channels. For redundancy, we recommend that you create multiple notification channels. Pairing Pub/Sub with Google Cloud console Mobile App, PagerDuty, Webhooks, or Slack is recommended, because Pub/Sub uses a different delivery mechanism.

To configure a Pub/Sub notification channel:

  1. Enable required APIs and create a topic
  2. Configure the notification channel for a topic
  3. Authorize a service account
  4. Set the notification channel in an alerting policy

To understand the data schema, see Schema example.

To receive the alert notifications, you must also create one of the following subscriptions: pull, push, or BigQuery.

When the subscription is in a different project than the topic, create a service account in the subscriber project, and grant it the role of roles/pubsub.subscriber.

Before you begin

When you add the first notification channel for a Google Cloud project, Cloud Monitoring creates a service account for that project. It also grants the Identity and Access Management role Monitoring Notification Service Agent to the service account. This service account lets Monitoring send notifications to Pub/Sub-based notification channels in this project.

The service account has the following format:

service-PROJECT_NUMBER@gcp-sa-monitoring-notification.iam.gserviceaccount.com

The name of the notifications service account includes a unique Google Cloud project number. You can find the project name, ID, and number on the project's dashboard in the Google Cloud console, or you can retrieve it with the following command:

gcloud projects describe PROJECT_ID --format="value(project_number)"

You can view and edit the service account by using the Google Cloud console or the Google Cloud CLI.

Enable required APIs and create a topic

In the Google Cloud project where you plan to create the notification channel and Pub/Sub topic, do the following:

  1. Sign in to your Google Cloud account. If you're new to Google Cloud, create an account to evaluate how our products perform in real-world scenarios. New customers also get $300 in free credits to run, test, and deploy workloads.
  2. In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

    Go to project selector

  3. Make sure that billing is enabled for your Google Cloud project.

  4. Enable the Pub/Sub and Cloud Monitoring APIs.

    Enable the APIs

  5. In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

    Go to project selector

  6. Make sure that billing is enabled for your Google Cloud project.

  7. Enable the Pub/Sub and Cloud Monitoring APIs.

    Enable the APIs

  8. Create a Pub/Sub topic. For information about how to create a topic, see Create a topic. For example, to create a topic called notificationTopic by using the Google Cloud CLI, run the following command:

    gcloud pubsub topics create notificationTopic

Next, configure your Pub/Sub notification channels.

Configure notifications for a topic

To create a Pub/Sub notification channel, you can use the Monitoring API, the Google Cloud CLI, or the Google Cloud console. After you create the notification channel, authorize the notifications service account to publish each topic that you are using as a notification channel.

For information about using the Monitoring API or the Google Cloud CLI to create the notification channel, see Creating channels.

To use the Google Cloud console to create the notification channel, do the following:

  1. In the navigation panel of the Google Cloud console, select Monitoring, and then select  Alerting:

    Go to Alerting

  2. Select the Google Cloud project that contains the Pub/Sub topic you created.
  3. Click Edit notification channels.
  4. In the Pub/Sub section, click Add new.

    The Created Pub/Sub Channel dialog displays the name of the service account that Monitoring created.

  5. Authorize the service account. Your service account can publish all topics or specific topics:

    • To publish all topics, select Manage roles, and add the role of Pub/Sub Publisher.

      After you complete these steps that configure the notification channel, skip the next section, which is titled Authorize service account, and proceed to Set the notification channel in an alerting policy.

    • To publish specific topics, proceed to the next step and authorize the service account to publish specific topics after you complete configuring the notification channel. For instructions to authorize the service account, see Authorize service account.
  6. Enter a display name for your channel and the Pub/Sub topic name.
  7. (Optional) To verify that the channel is correctly configured, click Send test notification.
  8. Select Add channel.

Next, authorize the service account.

Authorize service account

Authorization lets the notifications service account publish each Pub/Sub topic that you are using as a notification channel. This section describes how you can do the following:

  • Authorize a service account for a specific topic.
  • Authorize a service account for all topics.

Authorize a service account for a specific topic

You can authorize a service account to publish a specific topic by using the Google Cloud console and the Google Cloud CLI. This section describes both approaches.

To authorize your service account for a specific topic by using the Google Cloud console, do the following:

  1. In the navigation panel of the Google Cloud console, select Pub/Sub, and then select Topics:

    Go to Topics

  2. Select the topic.
  3. In the Permissions tab, select Add principal.
  4. In the New principal field, enter the name of the notifications service account. The service account has the following naming convention:
    service-PROJECT_NUMBER@gcp-sa-monitoring-notification.iam.gserviceaccount.com
  5. Select the role Pub/Sub Publisher, and then select Save.

To authorize your service account for a specific topic by using the Google Cloud CLI, grant the pubsub.publisher IAM role for the topic to the service account. For example, the following command configures the IAM role for the notificationTopic topic:

gcloud pubsub topics add-iam-policy-binding \
projects/PROJECT_NUMBER/topics/notificationTopic --role=roles/pubsub.publisher \
--member=serviceAccount:service-PROJECT_NUMBER@gcp-sa-monitoring-notification.iam.gserviceaccount.com
 

A response to a successful execution of the add-iam-policy-binding command is like the following:

 Updated IAM policy for topic [notificationTopic].
 bindings:
 ‐ members:
    ‐ serviceAccount:service-PROJECT_NUMBER@gcp-sa-monitoring-notification.iam.gserviceaccount.com
    role: roles/pubsub.publisher
 etag: BwWcDOIw1Pc=
 version: 1
 

For more information, see the pubsub topics add-iam-policy-binding reference page.

Authorize a service account for all topics

To authorize your service account for all topics, do the following:

  1. In the navigation panel of the Google Cloud console, select IAM:

    Go to IAM

  2. Select Include Google-provided role grants as shown in the following image:
    Select the Include Google-provided role grants option.
  3. Search for the service account that has the following format:
    service-PROJECT_NUMBER@gcp-sa-monitoring-notification.iam.gserviceaccount.com

    Reload the page when the service account isn't listed.

  4. Select Edit for the notifications service account, select Add another role, and then add the role Pub/Sub Publisher.

Next, set the notification channel in an alerting policy.

Set the notification channel in an alerting policy

To use a Pub/Sub notification channel in an alerting policy, select Pub/Sub as the channel type, and then select the topic.

Schema example

To view an example JSON packet and the schema, expand the following sections.

Deprecation policy

The payload schema is subject to the Google Cloud deprecation policy outlined in Section 1.4(d) of the Google Cloud Platform Terms of Service. Note that the schema does not control the formats of generated field values, and these formats can change without notice. For example, incident.summary, incident.documentation.content, and incident.url are meant to include data pertaining to their fields, but the schema doesn't have constraints to guarantee accurate parsing of these fields. You can consume the value as a whole and expect that it adheres to the deprecation policy, but don't rely on parsing the generated fields.

Google Chat

The Google Cloud Monitoring app allows your Google Chat spaces to receive notifications.

Before you begin

If you are in an organization, then to install the Google Cloud Monitoring app, you must have access to the Google Cloud Monitoring app. If you don't have access, then request that a Google Workspace administrator do the following:

  1. Add the app to the Marketplace apps allowlist and enable access at the top organizational unit.
  2. Configure Marketplace app user install settings so that users can install the app.

Install the Google Cloud Monitoring app

You must use the following procedure to install the Google Cloud Monitoring app. Installation from the Google Workspace Marketplace listing isn't supported.

To install the app and configure a space as a notification channel, do the following:

  1. In Google Chat, open or create the space where you want to receive notifications.

  2. Install the Google Cloud Monitoring app:

    1. Select your space name and then select Apps and Integrations.

    2. If the Google Cloud Monitoring app doesn't appear in the Apps list, then return to your space, enter @Google Cloud Monitoring, and then press Enter.

    3. Click Add to space.

  3. Copy the chat space ID to your clipboard.

    If you installed the Google Cloud Monitoring app, then the chat space ID is included in the installation confirmation. Otherwise, you can find the chat space ID as the series of numbers and letters following space/ at the end of your chat space URL.

  4. Open a new browser tab and then configure the notification channel:

    1. In the navigation panel of the Google Cloud console, select Monitoring, and then select  Alerting:

      Go to Alerting

    2. Click Edit notification channels.

    3. Go to the Google Chat section and click Add New.

    4. Enter your chat space ID in the Chat Space ID field and then click Save.

Create a channel on demand

When adding a notification channel to an alerting policy, you must select a channel from a list. To update the list of options when you are creating an alerting policy, use the following process:

  1. In the notification dialog, click Manage Notification Channels. You're taken to the Notification channels window in a new browser tab.
  2. To add a new notification channel, locate the channel type, click Add new, and then follow the channel-specific instructions contained in the previous table.
  3. Return to the original tab, and in the notification dialog, click Refresh.

    Notification dialog displaying the refresh and manage channels buttons.

  4. Select the notification channel from the updated list.

Get notified on an unsupported channel

If your preferred notification channel isn't supported, consider creating a pipeline that relies on sending your notifications to Pub/Sub. For a Python example that uses Flask, see Creating custom notifications with Cloud Monitoring and Cloud Run. For other examples, see the cloud-alerting-notification-forwarding Git repository.

Test a notification channel

Cloud Monitoring doesn't support a test option for notification channels. However, you can verify that a notification channel is properly configured by creating an alerting policy and configuring the condition to trigger.

To manually test your notification channels, you could create an alerting policy that monitors the CPU utilization of your virtual machine (VM) instances:

  1. In the navigation panel of the Google Cloud console, select Compute Engine, and then select VM instances:

    Go to VM instances

  2. Either select a VM instance or create an instance.
  3. Create an alerting policy to monitor the CPU utilization of that instance.

    If your Google Cloud project contains multiple instances, then add a filter to select one instance.

  4. Configure the Condition trigger fields as follows:

    1. Select Threshold for the Condition type field.
    2. Select Below threshold for the Threshold position field.
    3. Select a value that is larger than the CPU utilization for your instance. For example, if the chart displays the CPU utilization to be about 5%, then set the threshold to 10%.
    4. Expand Advanced Options and select No retest for the Retest window field.

    These settings configure the condition to trigger when the CPU utilization of a VM instance is less than the specified threshold. Because you selected the threshold to be larger than the CPU utilization of your VM instance, the condition triggers.

  5. Add the notification channels to your policy.

  6. Expand Incident autoclose duration and select 30 min.

  7. Enter a policy name and click Create policy.

  8. After the condition triggers, verify that you received a notification on each notification channel.

  9. Delete the alerting policy. The incident is closed automatically after the autoclose duration expires.

  10. If you created a VM instance for this test, then delete it.

Edit and delete notification channels

To edit or delete a notification channel by using the Google Cloud console, do the following:

  1. In the navigation panel of the Google Cloud console, select Monitoring, and then select  Alerting:

    Go to Alerting

  2. Click Edit notification channels.

    The Notification channels dashboard contains a section for each type of notification channel. Each section lists all configurations for that type:

    • To modify an entry, click Edit. Click Save after your changes are complete.
    • To delete an entry, click Delete. Click Delete in the confirmation dialog.

View notification channel logs

You can use the Logs Explorer to view notification channel errors:

  1. In the navigation panel of the Google Cloud console, select Logging, and then select Logs Explorer:

    Go to Logs Explorer

  2. Enter and run your query. For queries specific to notification channel errors, see Cloud Monitoring queries.

What's next

For information about incidents, see the following documents: