This document describes how you can view, investigate, and manage incidents for log-based alerting policies.
Before you begin
Ensure that you have the permissions that you need:
-
To get the permissions that you need to view incidents by using the Google Cloud console, ask your administrator to grant you the following IAM roles on your project:
-
Monitoring Cloud Console Incident Viewer (
roles/monitoring.cloudConsoleIncidentViewer
) -
Stackdriver Accounts Viewer (
roles/stackdriver.accounts.viewer
)
For more information about granting roles, see Manage access.
You might also be able to get the required permissions through custom roles or other predefined roles.
-
Monitoring Cloud Console Incident Viewer (
-
To get the permissions that you need to manage incidents by using the Google Cloud console, ask your administrator to grant you the following IAM roles on your project:
-
Monitoring Cloud Console Incident Editor (
roles/monitoring.cloudConsoleIncidentEditor
) -
Stackdriver Accounts Viewer (
roles/stackdriver.accounts.viewer
)
For more information about granting roles, see Manage access.
You might also be able to get the required permissions through custom roles or other predefined roles.
-
Monitoring Cloud Console Incident Editor (
For more information about Cloud Monitoring roles, see Control access with Identity and Access Management.
Find incidents
To see a list of incidents, do the following:
-
In the Google Cloud console, select Monitoring, and then select notifications Alerting, or click the following button:
- The Summary pane lists the number of open incidents.
- The Incidents pane displays the most recent open incidents. To list the most recent incidents in the table, including those that are closed, click Show closed incidents.
Optional: To view the details of a specific incident, select the incident in the list. The Incident details page opens. For information about this page, see the Investigating incidents section of this page.
Find older incidents
The Incidents pane on the Alerting page shows the most recent open incidents. To locate older incidents, do one of the following:
To page through the entries in the Incidents table, click arrow_back_ios Newer or arrow_forward_ios Older.
To navigate to the Incidents page, click See all incidents. From the Incidents page, you can do all the following:
- Show closed incidents: To list all incidents in the table, click Show closed incidents.
- Filter incidents: For information about adding filters, see Filtering incidents.
- Acknowledge or close an incident, or snooze its alerting policy. To access these options, click more_vert More options in the incident's row, and make a selection from the menu. For more information, see Managing incidents.
Filter incidents
When you enter a value on the filter bar, only incidents that match the filter are listed in the Incidents table. If you add multiple filters, then an incident is displayed only if it satisfies all the filters.
To add a filter the table of incidents, do the following:
On the Incidents page, click filter_list Filter table and then select a filter property. Filter properties include all the following:
- State of the incident
- Name of the alerting policy
- When the incident was opened or closed
Select a value from the secondary menu or enter a value in the filter bar.
Investigate incidents
After you have found the incident you want to investigate, go to the Incident details page for that incident. To view the details, click on the incident summary in the table of incidents on either the Alerting page or the Incidents page.
Alternately, if you received a notification that includes a link to the incident, then you can use that link to view the incident details.
The following screenshot shows the details page for an incident:

The Incident details page provides the following information:
Status information, including:
- Name: The name of the alerting policy that caused this incident.
- Status: The status of the incident: open, acknowledged, or closed.
- Duration: The length of time for which the incident was open.
A Logs pane, which displays log entries that match the alert query. The pane lets you filter these entries as part of your investigation.
To refresh the log-entries list, click refresh Refresh. To view the logs in the Logs Explorer, click open_in_new View in Logs Explorer.
Information about the alerting policy that caused the incident:
Condition pane: identifies the condition in the alerting policy that caused the incident. For log-based alerting policies creating by using the Logs Explorer, the condition name is always "Log match condition."
This pane also reports the time between notifications, and auto-close duration from the alerting policy.
Message pane: provides a brief explanation of the cause based on the configuration of the condition in the alerting policy. This pane is always populated.
Documentation pane: shows the documentation template for notifications that you provided when creating the alerting policy. This information might include a description of what the alerting policy monitors and include tips for mitigation.
If you skipped this field when creating the alerting policy, then this pane reports "No documentation is configured."
- Labels: reports the following:
- The labels and values for the monitored resource included in the log entry that triggered the alerting policy. This information can help you identify the specific monitored resource that caused the incident. These labels are also reported in the Message string.
- Any user-specified labels and values that you defined on the alerting policy. You can use these labels for organizing and identifying alerting policies. Labels associated with a policy are listed in the Policy Labels section, while labels defined as part of a condition are listed in the Metric labels section. Metadata labels are only displayed when there is a filter or grouping that depends on the label. For example usage, see Add severity levels to an alerting policy.
The Incidents details page also provides tools for investigating the incident:
- Links to other troubleshooting tools. The configuration of your project
and alerting policy and the age of the incident determine
which links are available.
- To see the details page for the alerting policy, click View policy.
- To edit the definition of the alerting policy, click Edit policy.
- To see related log entries in Logs Explorer, click View logs. For more information, see Using Logs Explorer.
- Annotations: Provides a log of your findings, results, suggestions,
or other comments from your investigation of the incident.
- To add an annotation, enter text in the field and click Add comment.
- To discard the comment, click Cancel.
Manage incidents
Incidents are in one of the following states:
error Open: The log-based alerting policy was triggered, and the incident is still open. If the same alert is triggered again and there is already an incident open, then a new incident isn't opened.
warning Acknowledged: The incident is open and has manually been marked as acknowledged. Typically, this status indicates that the incident is being investigated.
check_circle Closed: You have manually closed the incident, or it was automatically closed after the auto-close period expired.
Acknowledging incidents
We recommend that you mark an incident as acknowledged when you begin investigating the cause of the incident.
To mark an incident as acknowledged, do the following:
- In the Incidents pane of the Alerting page, click See all incidents.
On the Incidents page, find the incident that you want to acknowledge, and then do one of the following:
- Click more_vert More options and then select Acknowledge.
- Open the details page for the incident and then click Acknowledge incident.
Snooze an alerting policy
To prevent Monitoring from creating incidents and sending notifications during a specific time period, snooze the related alerting policy. When you snooze an alerting policy, incidents related to the alerting policy remain open but don't cause further notifications. The incidents close based on the alerting policy auto-close duration.
To create a snooze for an incident that you are viewing, do the following:
On the Incident details page, click Snooze.
Select the snooze duration. After you select the snooze duration, the snooze begins immediately.
When you view an incident's details page, you can create a snooze for the related alerting policy by clicking Snooze and then choosing a duration. The snooze begins immediately. You can also snooze an alerting policy from the Incidents page by finding the incident that you want to snooze, clicking more_vert More options, and then selecting Snooze. You can snooze alerting policies during outages to prevent further notifications during the troubleshooting process.
Close incidents
You can let Monitoring close an incident for you, or you can close the incident.
Monitoring automatically closes an incident when the auto-close duration for the alerting policy expires. By default, the auto-close duration is 7 days. The minimum auto-close duration is 30 minutes.
To close an incident, do the following:
- In the Incidents pane of the Alerting page, click See all incidents.
On the Incidents page, find the incident that you want to close, and then do one of the following:
- Click more_vert View more and then select Close incident.
- Open the details page for the incident and then click Close incident.
Unable to close incident
, try again in a few
minutes. You can't close a new incident immediately because the triggering
log entry is still considered active by the alerting system.
Data retention and limits
For information about limits and about the retention period of incidents, see Limits for alerting and uptime checks.
What's next
- To create and manage alerting policies with the Cloud Logging API or from the command line, see Managing alerting policies by API.