Monitoring your logs

This page describes how you can use Cloud Monitoring to observe trends in your logs and to notify you when conditions you describe occur. To provide Cloud Monitoring with data from your logs, Logging offers you the following:

  • Log-based metrics, which you can use as follows:
    • To create alerting policies that notify you of changes over time.
    • To create charts that display changes over time.
  • Log-based alerts, which notify you any time a specific event appears in a log.

If you want to evaluate your logs over a period of time, use charts or alerting policies based on log-based metrics.

If you want a notification in near real time when a message has appeared in your logs, use a log-based alert.

The rest of this page describes how to choose between log-based metrics and log-based alerts, and discusses the differences between them.

Log-based metrics

If you want to monitor recurring events in your logs over a period of time, use log-based metrics. Log-based metrics generate numeric data from your logs. Log-based metrics are suitable if you want to do any of the following:

  • Count the occurrences of a message, like a warning or error, in your logs and receive a notification when the number of occurrences crosses a threshold.
  • Observe trends in your data, like latency values in your logs, and receive a notification if the values change in an unacceptable way.
  • Create charts to display the numeric data extracted from your logs.

Because log-based metrics generate numeric data from your logs, you can use log-based metrics in alerting policies and display them in charts. For information about creating alerting policies and charts for log-based metrics, see Creating charts and alerts.

Cloud Monitoring provides a set of predefined log-based metrics, and you can define your own. To see a list of the system-defined log-based metrics, click the following button:

User-defined log-based metrics

For information about defining and using your own log-based metrics, see Using log-based metrics. If you define your own log-based metrics, you might incur charges. For more information about costs associated with metric ingestion, see Chargeable metrics.

Log-based alerting

When you want to be notified any time a specific message occurs in a log, use log-based alerts. Log-based alerts are well suited for catching security-related events in logs, like the following:

  • You want to be notified if an event appears in an audit log; for example, a human user accesses the security key of a service account.
  • Your application writes deployment messages to logs, and you want to be notified when a deployment change is logged.

Log-based alerts are well suited for events that you expect to be both rare and important. You don't want to know about a trend or pattern; you want to know that something occurred.

For information about creating log-based alerts, see Using log-based alerts.

You can simulate a log-based alert by defining a log-based metric and creating an alerting policy with a threshold of 1, but log-based alerts give you that behavior without the need to create a log-based metric and configure a metric-based alerting policy.

Comparison of alerting options

This section describes the primary ways in which alerting policies built on log-based metrics differ from log-based alerts.

Available logs

User-defined log-based metrics are calculated from all logs received by the Logging API for the Cloud project, regardless of any inclusion filters or exclusion filters that may apply to the Cloud project. If you create an alerting policy based on a user-defined log-based metric, then the policy monitors data from all logs.

System-defined log-based metrics are calculated only from logs that have been ingested by Logging for the Cloud project. If a log has been explicitly excluded from ingestion by Logging, then it isn't included in these metrics. If you create an alerting policy based on a system-defined log-based metric, then the policy monitors data only from included logs.

Log-based alerts operate only on included logs; you can't use log-based alerts to notify you about messages in excluded logs.

Both log-based metrics and log-based alerts operate on the Cloud project scope, not on individual buckets.

Creating and managing alerting policies

You create, modify, and delete alerting policies based on log-based metrics in Cloud Monitoring, like any other metric-based alerting policy. For more information, see Managing policies.

You can create log-based alerts by using the Logs Explorer or the Cloud Monitoring API. You modify and delete log-based alerts in Monitoring. For more information see Managing log-based alerts.

Alerting across multiple projects

You can monitor metrics from several projects by configuring a metrics scopes. A metrics scope lists all of the projects and accounts that it monitors, and it is hosted by a scoping project. The scoping project stores the alerting policies and other configurations that you create for the metrics scope. The scoping project for a metrics scope is the project selected by the Cloud Console project picker.

Alerting policies based on log-based metrics, like alerting policies based on other metrics, work across all projects in the metrics scope of the scoping project.

Log-based alerts do not operate on metrics scopes; the logs in projects are not part of a metrics scope. A log-based alert evaluates only the logs that reside in the scoping project.

Viewing alerting policies

The Policies page in Monitoring lists all of the alerting policies in your Google Cloud project. This list includes policies that use log-based metrics and log-based alerts.

Log-based alerts appear in the list with the value Logs in the Type column. Alerts based on metrics, including log-based metrics, appear in the list with the value Metrics in the Type column. The following screenshot shows an excerpt of a policy list:

Use the **Type** column in the list of alerting policies to distinguish log-based alerts and metric-based alerts.

Notification channels

You can send notifications from both metric- and log-based alerts to any of the notification channels supported by Monitoring. You must configure these channels before you can use them in alerting policies.

For more information, see Managing notification channels.

Incidents and notifications

When an alerting policy is triggered, it opens an incident in Monitoring and sends notifications to any channels you've selected. To see the details of the incident, you can click View incident in the notification message, or you can navigate directly to the Incidents page in Monitoring.

Alerting policies based on log-based metrics create incidents and notifications like all other metric-based alerting policies in Monitoring, as described in Alerting behavior. For more information about managing incidents for metric-based alerting policies, see Incidents for metric-based alerts in the Monitoring documentation.

Log-based alerts are not metric-based alerts. Log-based alerts create incidents and notifications as follows:

  • The first time Cloud Logging ingests a log entry that matches your alert query, an incident is created and a notification is sent. If another matching log entry is then ingested, a new incident is created only if the previous incident has been closed. However, it might take up to three minutes for a closed incident to be purged. If a matching log entry is received in the three minutes after you closed an incident, the system might reopen that incident rather than creating a new incident.

  • There is a limit of 20 notifications a day for each log-based alert. If you reach this limit, your notification includes a message that you have reached the limit for the day.

  • When you create a log-based alert, you can specify the minimum time between notifications to reduce repeated notifications. For example, if you select 10 minutes as the time between notifications, and your log-based alert is triggered twice within that period, you receive only one notification.

    There is a maxiumum rate of 1 notification every 5 minutes for each log-based alert. You can configure the notification frequency to be much lower, though.

  • Incidents are automatically closed after 7 days, unless you close them manually.

For more information about managing these incidents, see Managing incidents for log-based alerts.

Authorization requirements

Using log-based metrics or log-based alerts requires authorization for both Cloud Logging and Cloud Monitoring.

Costs and limits

If you define your own log-based metrics, the following apply:

  • There are limits on the number and structure of user-defined log-based metrics. For more information about these limits, see limits for log-based metrics.
  • You might incur charges for user-defined log-based metrics. For more information about costs associated with metric ingestion, see Chargeable metrics.

There are no charges associated with using alerting policies based on log-based metrics.

The following Monitoring limits related to alerting policies apply:

Category Value Policy type1
Alerting policies (sum of metric and log) per metrics scope 2 500 Metric, Log
Conditions per alerting policy 6 Metric
Maximum time period that a
metric-absence condition evaluates3
1 day Metric
Maximum time period that a
metric-threshold condition evaluates3
23 hours 30 minutes Metric
Notification channels per alerting policy 16 Metric, Log
Maximum rate of notifications 1 notification every 5 minutes for each log-based alert Log
Maximum number of notifications 20 notifications a day for each log-based alert Log
Maximum number of simultaneously open incidents
per alerting policy
5000 Metric
Period after which an incident with no new data is
automatically closed
7 days Metric
Maximum duration of an incident if not manually closed 7 days Log
Retention of closed incidents 13 months Not applicable
Retention of open incidents Indefinite Not applicable
Notification channels per metrics scope 4000 Not applicable
Uptime checks per metrics scope 4 100 Not applicable
1Metric: an alerting policy based on metric data; Log: an alerting policy based on log messages (log-based alerts)
2Apigee and Apigee hybrid are deeply integrated with Cloud Monitoring. The alerting limit for all Apigee subscription levels—Standard, Enterprise, and Enterprise Plus—is the same as for Cloud Monitoring: 500 per metrics scope .
3The maximum time period that a condition evaluates is the sum of the alignment period and the duration window values. For example, if the alignment period is set to 15 hours, and the duration window is set 15 hours, then 30 hours of data is required to evaluate the condition.
4This limit applies to the number of uptime-check configurations. Each uptime-check configuration includes the time interval between testing the status of the specified resource. See Managing uptime checks for more information.