Overview of logs-based metrics

This page provides a conceptual overview of logs-based metrics.

Logs-based metrics are based on the content of log entries. For example, the metrics can record the number of log entries containing particular messages, or they can extract latency information reported in log entries. You can use logs-based metrics in Cloud Monitoring charts and alerting policies.

There are two kinds of logs-based metrics:

  • System-defined logs-based metrics, provided by Cloud Logging for use by all Google Cloud projects.
  • User-defined logs-based metrics, created by you to track things in your Google Cloud project that are of particular interest to you. For example, you might create a logs-based metric to count the number of log entries that match a given filter.

Logs-based metrics apply only to a single Google Cloud project. You cannot create them for logs buckets or for other Google Cloud resources such as Cloud Billing accounts or organizations.

Getting started

To verify that you have the correct permissions to use logs-based metrics in a Cloud project, see Access control on this page.

If you're using Cloud Monitoring for the first time, see the Cloud Monitoring section on this page.

Viewing your logs-based metrics

The Logs-based metrics page in the Google Cloud Console lets you create a logs-based metric, and it contains lists of all the logs-based metrics for your Cloud project.

To view a list of logs-based metrics for your Google Cloud project, do the following:

  1. Go to the Logs-based metrics page in the Cloud Console:

    Go to Logs-based metrics

  2. Select an existing Cloud project. The lists of logs-based metrics in your Cloud project appear.

The logs-based metrics interface is divided into two metric-type panes: System metrics and User-defined metrics.

Each pane contains a table summary of the metrics. Each metric's row has a menu that features the following options:

  • View in Metrics Explorer lets you view the data for a system logs-based metric by opening Metrics Explorer in Cloud Monitoring.

    You can use Metrics Explorer to specify a target metric for an alerting policy. The chart next to the Target region gives you visual feedback on the data being captured by the target metric.

  • Create alert from metric lets you create an alerting policy based on the logs-based metric.

    Selecting this option opens the Cloud Monitoring console, where you can create, edit, and manage alerting policies. For details on creating alerting policies for your logs-based metrics, read Creating an alerting policy.

User-defined metrics pane

The User-defined metrics pane of the logs-based metrics interface has several features to help you manage the user-defined metrics on your Cloud project:

  • The user-defined metrics table includes Name, Description, Type, and Filter columns. These are specified when you create the metric.

  • The Filter user-defined metrics box lets you filter your metric list by text search or metric Name, Description, and Filter.

  • The user-defined metrics table includes columns for Previous month usage and Month-to-date usage (MTD). This usage data is useful, for example, if you want to determine which metrics ingest the most data or to estimate your bills.

Clicking on any of the column names lets you sort data in ascending or descending order. At the bottom of the table, you can also select the number of rows that you want to display.

The menu for each metric in your user-defined metrics pane contains additional features for managing your metrics:

The user-defined logs-based metrics pane overflow menu options.

  • View metric details: Shows you the metric's Name, Type, Description, Filter, Units, and Labels, if defined.
  • Edit metric: Lets you edit certain fields for the metric.
  • Disable metric: Lets you stops the metric from being calculated. You can re-enable a disabled metric from the same menu.
  • Delete metric: Lets you delete the metric.
  • View logs for metric: Takes you to the Logs Explorer and populates the metric's filter in the Query builder and runs the query.

Overview of logs-based metric types

System logs-based metrics are calculated from included logs only. User-defined logs-based metrics are calculated from both included and excluded logs.

The data for a logs-based metric comes from log entries received after a metric is created. A metric isn't retroactively populated with data from log entries that are already in Logging.

Logging accumulates information for a logs-based metric each time it receives a matching log entry. Logging writes a new data point to the metric's time series at the rate of 1 datapoint per minute, making the data available to Cloud Monitoring.

Each data point in a logs-based metric's time series represents only the additional information (the delta) received since the previous data point.

User-defined logs-based metrics can be of the counter or distribution metric types. Most system-defined logs-based metrics are counters, but some are of the Boolean type.

The following sections describe the characteristics of counter-type and distribution-type metrics.

Counter metrics

Counter metrics count the number of log entries matching a given filter. For example, you can do the following:

  • Count the log entries that contain a certain specific error message.
  • Count the number of times each user invokes an operation, by looking for log messages that match this pattern:

    ... user [USERNAME] called  [OPERATION] ...
    

    By extracting [USERNAME] and OPERATION and using them as values for two labels, you can later ask, "How many times did sally call the UPDATE operation?", "How many people called the READ operation?", "How many times did george call an operation?", and so on.

For more information, go to Creating counter metrics.

Distribution metrics

Distribution metrics accumulate numeric data from log entries matching a filter. The metrics contain a time series of distribution objects, each of which contains the following:

  • A count of the number of values in the distribution.
  • The mean of the values.
  • The sum of squared deviations: Sumi=1..n(xi–mean)2
  • A set of histogram buckets with the count of values in each bucket. You can use the default bucket layout or choose your own.

A common use for distribution metrics is to track latencies. As each log entry is received, a latency value is extracted from somewhere in the log entry and is added to the distribution. At regular intervals, the accumulated distribution is written to Cloud Monitoring.

For information on distributions, including their format within a time series and how they are visualized, see Charting distribution metrics.

For information on creating distribution logs-based metrics, see Creating distribution metrics.

Labels

Logs-based metrics can have labels, which allow multiple time series to be collected for the metric. Values for the labels are extracted from fields in the matching log entries. Logging records separate time series for each combination of label values.

The system logs-based metrics have predefined labels. You can define the labels for user-defined metrics. For more information, see Logs-based metric labels.

User-defined logs-based metrics

User-defined logs-based metrics are created by a user on a Google Cloud project. These metrics count the number of log entries that match a given filter or record particular values within the matching log entries.

For information on creating and managing your user-defined metrics using the Cloud Console, see Creating counter metrics and Creating distribution metrics.

System logs-based metrics

Logging provides a set of metrics that includes counters. The counter metrics record the number of logging events that occurred within a specific time period. The metrics have labels that record the counts by log name and severity level.

For a list of these metrics, see Google Cloud metrics: logging.

Cloud Monitoring

You can use both system and user-defined logs-based metrics in Cloud Monitoring to create charts and alerting policies. For more information, go to Creating charts and alerts.

In Cloud Monitoring, logs-based metrics use the following naming patterns:

  • System: logging.googleapis.com/<var>SYSTEM_METRIC_NAME</var>
  • User-defined: logging.googleapis.com/user/<var>...</var>

Note that user-defined metrics include the string user.

Configuration

To configure a Cloud Monitoring Workspace for your project, do the following:
  1. In the Cloud Console, select your Google Cloud project.
    Go to Cloud Console
  2. In the navigation pane, select Monitoring.

    If you have never used Cloud Monitoring, then on your first access of Monitoring in the Google Cloud Console, a Workspace is automatically created and your project is associated with that Workspace. Otherwise, if your project isn't associated with a Workspace, then a dialog appears and you can either create a Workspace or add your project to an existing Workspace. We recommend that you create a Workspace. After you make your selection, click Add.

Access control

To view your Cloud project logs-based metrics, verify you have the correct role and permissions.

Identity and Access Management roles and permissions govern access to Google Cloud data. Following is a summary of the common roles and permissions a Cloud project member needs to access logs-based metrics:

  • Logging/Logs Configuration Writer (roles/logging.configWriter) lets you list, create, get, update, and delete logs-based metrics.

  • Logging/Logs Viewer (roles/logging.viewer) lets you view existing metrics. You can also add the logging.logMetrics.get and logging.logMetrics.list permissions to a custom role.

  • Monitoring Viewer (roles/monitoring.viewer) lets you read the time series that the logs-based metric contains. You can also add the monitoring.timeSeries.list permission to a custom role.

  • Logging Admin (roles/logging.admin), Project Editor (roles/editor), and Project Owner (roles/owner) are broad-level roles that contain the permission to create logs-based metrics (logging.logMetrics.create).

For more information about permissions and roles, see Access control guide.

Pricing

User-defined logs-based metrics are a class of Cloud Monitoring custom metrics and are chargeable.

For pricing information, see Cloud Logging pricing: Logs-based metrics.

Troubleshooting

If you encounter issues when using logs-based metrics, see Troubleshooting.