This page explains why you might not receive notifications as expected, and offers possible remedies for those situations.
Notifications aren't received
If you don't receive any notifications through any configured notification channels, then do the following:
-
In the Google Cloud console, go to the Logs Explorer page:
If you use the search bar to find this page, then select the result whose subheading is Logging.
- Select the appropriate Google Cloud project.
Query the logs for notification channel events:
- Expand the Log name menu, and select notification_channel_events.
- Expand the Severity menu and select Error.
- Optional: To select a custom time range, use the time-range selector.
- Click Run query.
The previous steps create the following query:
resource.type:"stackdriver_notification_channel" logName="projects/PROJECT_ID/logs/monitoring.googleapis.com%2Fnotification_channel_events" severity=ERROR
The summary line and the
jsonPayload
field typically contain failure information. For example, when a gateway error occurs, the summary line includes "failed with 502 Bad Gateway".
Webhook notifications aren't received
This section applies when you don't receive notifications through a configured webhook notification channel.
Private endpoint
If you have a private endpoint, then use Pub/Sub notifications combined with a pull subscription to that notification topic. You can't use webhooks for notifications to private endpoints.
When you configure a Pub/Sub notification channel, incident notifications are sent to a Pub/Sub queue that has Identity and Access Management controls. Any service that can query for, or listen to, a Pub/Sub topic can consume these notifications. For example, applications running on App Engine, Cloud Run, or Compute Engine virtual machines can consume these notifications.
If you use a pull subscription, then a request is sent to Google that waits for a message to arrive. These subscriptions require access to Google but they don't require rules for firewalls or inbound access.
Public endpoint
To identify why the delivery failed, examine your Cloud Logging log entries for failure information.
For example, you can search for log entries for the notification channel resource by using the Logs Explorer, with a filter like the following:
resource.type="stackdriver_notification_channel"
Pub/Sub notifications aren't received
If you don't receive notifications through a configured Pub/Sub notification channel, then try the following:
Ensure that the notifications service account exists. Notifications aren't sent when the service account has been deleted.
To verify that the service account exists, do the following:
-
In the Google Cloud console, go to the IAM page:
If you use the search bar to find this page, then select the result whose subheading is IAM & Admin.
Search for a service account that has the following naming convention:
service-PROJECT_NUMBER@gcp-sa-monitoring-notification.iam.gserviceaccount.com
If this service account isn't listed, then select Include Google-provided role grants.
If the notifications service account doesn't exist, then you must begin the process of creating the Pub/Sub notification channel for Monitoring to create the service account:
-
In the Google Cloud console, go to the notifications Alerting page:
If you use the search bar to find this page, then select the result whose subheading is Monitoring.
- Click Edit notification channels.
In the Pub/Sub section, click Add new.
Monitoring creates the notifications service account when one doesn't exist. The Create Pub/Sub Channel dialog shows the name of the notifications service account.
If you don't want to add a notification channel, click Cancel. Otherwise, finish creating the notification channel and click Add channel.
Grant the service account permissions to publish your Pub/Sub topics:
- In a new browser tab, open the Create a notification channel document.
- Select the Pub/Sub tab, and then follow the steps in the Authorize service account section of the page.
-
Ensure that the notifications service account has been authorized to send notifications for the Pub/Sub topics of interest.
To view the permissions for a service account, you can use the Google Cloud console or the Google Cloud CLI command:
- The IAM page in the Google Cloud console lists the roles for each service account.
- The Pub/Sub Topics page in the Google Cloud console, lists each topic. When you select a topic, the Permissions tab lists the roles granted to service accounts.
To list all service accounts and their roles, run the following Google Cloud CLI command:
gcloud projects get-iam-policy PROJECT_ID
The following is a partial response for this command:
serviceAccount:service-PROJECT_NUMBER@gcp-sa-monitoring-notification.iam.gserviceaccount.com role: roles/monitoring.notificationServiceAgent - members: [...] role: roles/owner - members: - serviceAccount:service-PROJECT_NUMBER@gcp-sa-monitoring-notification.iam.gserviceaccount.com role: roles/pubsub.publisher
The command response includes only roles, it doesn't include per-topic authorization.
To list the IAM bindings for a specific topic, run the following command:
gcloud pubsub topics get-iam-policy TOPIC
The following is a sample response for this command:
bindings: - members: - serviceAccount:service-PROJECT_NUMBER@gcp-sa-monitoring-notification.iam.gserviceaccount.com role: roles/pubsub.publisher etag: BwXPRb5WDPI= version: 1
For information about how to authorize the notifications service account, see Authorize service account.
You aren't notified when a VM shuts down
To get notified when a virtual machine (VM) shuts down, create an uptime check to periodically interrogate the VM and then create an alerting policy to monitor that uptime check. If you are using a Virtual Private Cloud (VPC), then you might need to create a private uptime check.
An alerting policy that monitors the
compute.googleapis.com/instance/uptime
metric or
won't notify you when the VM shuts down. For this metric,
alerting policies only monitor time series for VM instances that are in the
RUNNING
state. If a VM is in any other state, such as STOPPED
or DELETED
,
then they aren't monitored. For information on VM instance states, see
VM instance life cycle.
Notifications for request-count alerting policies aren't received
If you don't receive notifications for an alerting policy that monitors the
serviceruntime.googleapis.com/api/request_count
metric,
then make sure that the policy's alignment period is no more
than 7 hours 30 minutes.
SMS notification messages or verification codes aren't received
If you don't receive SMS notifications, then make sure that you haven't
reached the SMS message limit. There might be logs that
confirm this error. Check your logs for Denied quota token
.
As a general recommendation, we discourage relying solely on SMS channels for notifications. SMS notifications are offered on a "best effort" basis.