Troubleshoot missing notifications

This page explains why you might not receive notifications as expected, and offers possible remedies for those situations.

Notifications aren't received

If you don't receive any notifications through any configured notification channels, then do the following:

In the Google Cloud console, go to the Logs Explorer page:
Go to Logs Explorer

If you use the search bar to find this page, then select the result whose subheading is Logging.
Select the appropriate Google Cloud project.
Query the logs for notification channel events:
1. Expand the Log name menu, and select notification_channel_events.
2. Expand the Severity menu and select Error.
3. Optional: To select a custom time range, use the time-range selector.
4. Click Run query.
The previous steps create the following query:
```
resource.type:"stackdriver_notification_channel"
logName="projects/PROJECT_ID/logs/monitoring.googleapis.com%2Fnotification_channel_events"
severity=ERROR
```
The summary line and the jsonPayload field typically contain failure information. For example, when a gateway error occurs, the summary line includes "failed with 502 Bad Gateway".

Webhook notifications aren't received

This section applies when you don't receive notifications through a configured webhook notification channel.

Private endpoint

If you have a private endpoint, then use Pub/Sub notifications combined with a pull subscription to that notification topic. You can't use webhooks for notifications to private endpoints.

When you configure a Pub/Sub notification channel, incident notifications are sent to a Pub/Sub queue that has Identity and Access Management controls. Any service that can query for, or listen to, a Pub/Sub topic can consume these notifications. For example, applications running on App Engine, Cloud Run, or Compute Engine virtual machines can consume these notifications.

If you use a pull subscription, then a request is sent to Google that waits for a message to arrive. These subscriptions require access to Google but they don't require rules for firewalls or inbound access.

Public endpoint

To identify why the delivery failed, examine your Cloud Logging log entries for failure information.

For example, you can search for log entries for the notification channel resource by using the Logs Explorer, with a filter like the following:

resource.type="stackdriver_notification_channel"

Pub/Sub notifications aren't received

If you don't receive notifications through a configured Pub/Sub notification channel, then try the following:

Ensure that the notifications service account exists. Notifications aren't sent when the service account has been deleted.

To verify that the service account exists, do the following:
1. In the Google Cloud console, go to the IAM page:
  Go to IAM
  
  If you use the search bar to find this page, then select the result whose subheading is IAM & Admin.
2. Search for a service account that has the following naming convention:
```
service-PROJECT_NUMBER@gcp-sa-monitoring-notification.iam.gserviceaccount.com
```
  If this service account isn't listed, then select Include Google-provided role grants.
If the notifications service account doesn't exist, then you must begin the process of creating the Pub/Sub notification channel for Monitoring to create the service account:
1. In the Google Cloud console, go to the Alerting page:
  Go to Alerting
  
  If you use the search bar to find this page, then select the result whose subheading is Monitoring.
2. Click Edit notification channels.
3. In the Pub/Sub section, click Add new.
  
  Monitoring creates the notifications service account when one doesn't exist. The Create Pub/Sub Channel dialog shows the name of the notifications service account.
  
  Note: You don't need to finish creating the Pub/Sub notification channel for Monitoring to create the notifications service account.
4. If you don't want to add a notification channel, click Cancel. Otherwise, finish creating the notification channel and click Add channel.
5. Grant the service account permissions to publish your Pub/Sub topics:
  1. In a new browser tab, open the Create a notification channel document.
  2. Select the Pub/Sub tab, and then follow the steps in the Authorize service account section of the page.
Ensure that the notifications service account has been authorized to send notifications for the Pub/Sub topics of interest.

To view the permissions for a service account, you can use the Google Cloud console or the Google Cloud CLI command:
- The IAM page in the Google Cloud console lists the roles for each service account.
- The Pub/Sub Topics page in the Google Cloud console, lists each topic. When you select a topic, the Permissions tab lists the roles granted to service accounts.
- To list all service accounts and their roles, run the following Google Cloud CLI command:
```
gcloud projects get-iam-policy PROJECT_ID
```
  The following is a partial response for this command:
```
    serviceAccount:service-PROJECT_NUMBER@gcp-sa-monitoring-notification.iam.gserviceaccount.com
       role: roles/monitoring.notificationServiceAgent
     - members:
       [...]
       role: roles/owner
     - members:
       - serviceAccount:service-PROJECT_NUMBER@gcp-sa-monitoring-notification.iam.gserviceaccount.com
       role: roles/pubsub.publisher
```
  The command response includes only roles, it doesn't include per-topic authorization.
- To list the IAM bindings for a specific topic, run the following command:
```
gcloud pubsub topics get-iam-policy TOPIC
```
  The following is a sample response for this command:
```
    bindings:
    - members:
      - serviceAccount:service-PROJECT_NUMBER@gcp-sa-monitoring-notification.iam.gserviceaccount.com
      role: roles/pubsub.publisher
    etag: BwXPRb5WDPI=
    version: 1
```
For information about how to authorize the notifications service account, see Authorize service account.

You aren't notified when a VM shuts down

To get notified when a virtual machine (VM) shuts down, create an uptime check to periodically interrogate the VM and then create an alerting policy to monitor that uptime check. If you are using a Virtual Private Cloud (VPC), then you might need to create a private uptime check.

An alerting policy that monitors the compute.googleapis.com/instance/uptime metric or won't notify you when the VM shuts down. For this metric, alerting policies only monitor time series for VM instances that are in the RUNNING state. If a VM is in any other state, such as STOPPED or DELETED, then they aren't monitored. For information on VM instance states, see VM instance life cycle.

Notifications for request-count alerting policies aren't received

If you don't receive notifications for an alerting policy that monitors the serviceruntime.googleapis.com/api/request_count metric, then make sure that the policy's alignment period is no more than 7 hours 30 minutes.

SMS notification messages or verification codes aren't received

If you don't receive SMS notifications or verification codes, then make sure that you haven't reached the SMS message limit. There might be logs that confirm this error. Check your logs for Denied quota token.

As a general recommendation, we discourage relying solely on SMS channels for notifications. SMS notifications are offered on a "best effort" basis.