Logs-based metrics are Stackdriver Monitoring metrics that are based on the content of log entries. For example, the metrics can record the number of log entries containing particular messages, or they can extract latency information reported in log entries. You can use logs-based metrics in Stackdriver Monitoring charts and alerting policies.
System (logs-based) metrics are predefined by Stackdriver Logging. These metrics record the number of logging events that occurred within a specific time period.
User-defined (logs-based) metrics are created by a user on a project. They count the number of log entries that match a given filter, or keep track of particular values within the matching log entries.
Stackdriver Logging accumulates information for logs-based metrics every time a matching log entry is received. On a regular schedule, Stackdriver Logging writes a new data point to the metric's time series at the rate of 1 datapoint per minute, making the data available to Stackdriver Monitoring.
The data for logs-based metrics comes from log entries received after the metrics are created. The metrics are not populated with data from log entries that are already in Stackdriver Logging.
Go to the Stackdriver Logging > Logs-based metrics page in the GCP Console:
Select an existing GCP project at the top of the page. You see the list of logs-based metrics in your GCP project:
Logs-based metrics interface
The logs-based metrics interface is divided into two metric-type areas: System Metrics and User-defined metrics.
Both areas contain a table summary of the metrics. Each table row has an overflow menu on the far right and features the following options:
View in Metrics Explorer lets you see the data for a system logs-based metric by opening the Stackdriver Monitoring Metrics Explorer.
You can use the Metrics Explorer to specify a target metric for an alerting policy. The chart next to the Target region gives you visual feedback on the data being captured by the target metric.
Create alert from metric lets you create an alerting policy based on your logs-based metric. Note that this option is available only if a metric is a counter metric.
Selecting this option opens the Stackdriver Monitoring console, where you can create, edit, and manage alerting policies. For details, see Creating an alerting policy.
User-defined metrics interface
The User-defined metrics area of the Logs-based metrics interface has several additional features to help you manage the user-defined metrics on a project:
The user-defined metrics table includes Name, Description, Type, and Filter columns. These are specified when the metric is created.
The Filter Metrics box, above the user-defined metrics table, lets you filter your metric list by text search or metric Name, Description, and Filter:
The user-defined metrics table includes columns for Previous Month Usage and Usage (MTD). These usage metrics are useful, for example, if you want to see which metrics ingest the most data or to estimate your bills.
You can Edit metric and Delete metric using the overflow menus at the end of a table row.
In addition, clicking on any of the column names lets you sort data in ascending or descending order. At the bottom of the table, you can also select the number of rows that you wish to display.
Overview of logs-based metric types
Stackdriver Logging logs-based metrics can be one of two metric types: counter or distribution. All system logs-based metrics are the counter type. User-defined logs-based metrics can be either the counter type or the distribution type.
Each data point in a logs-based metric's time series represents only the additional information (the delta) received since the previous data point.
The following sections describe the characteristics of counter-type and distribution-type metrics.
Counter metrics count the number of log entries matching an advanced logs filter. For example, you can do the following:
- Create a metric that counts log entries containing a certain specific error message.
Count the number of times each user invokes an operation, by looking for log messages like
... user [USERNAME] called [OPERATION] ...
By extracting [USERNAME] and [OPERATION] and using them as values for two labels, you can later ask, "How many times did
UPDATEoperation?", "How many people called the
READoperation?", "How many times did
georgecall an operation?", and so on.
For more information, see Creating Counter Metrics.
Distribution metrics accumulate numeric data from log entries matching a filter. The metrics contain a time series of Distribution objects, each of which contains the following:
- A count of the number of values in the distribution.
- The mean of the values.
- The sum of squared deviations: Sumi=1..n(xi–mean)2
- A set of histogram buckets with the count of values in each bucket. You can use the default bucket layout or choose your own.
A common use for distribution metrics is to track latencies. As each log entry is received, a latency value is extracted from somewhere in the log entry and is added to the distribution. At regular intervals, the accumulated distribution is written to Stackdriver Monitoring.
For more information, see Creating Distribution Metrics.
Logs-based metrics can optionally have labels, which allow a single metric to hold multiple time series. Values for the labels are extracted from fields in the matching log entries. Stackdriver Logging records separate time series for each different value of your label.
System logs-based metrics
Stackdriver Logging provides some predefined counter metrics that track the number and volume of log entries received. The metrics have labels that record the counts by log name and severity level. The following table lists the metrics:
|Metric name and description||Type||Value||Labels|
The total number of log entries received.
The total number of bytes received in log entries.
The total number of log entries that were excluded.
The total number of bytes in log entries that were excluded.
The total number of log entries that were exported using sinks.
The total number of bytes in log entries that were exported using sinks.
The total number of log entries that were not exported due to export configuration errors.
The number of late-arriving log entries.1
1 The late-arriving log entries are not included in the
See Logging metrics for a full list of system logs-based metrics.
You can use both system and user-defined logs-based metrics in
Stackdriver Monitoring to create charts and alerting policies. Your user-defined
logs-based metric names are prefixed by
user/; the system logs-based metrics
If you are using the Stackdriver Monitoring API, the logs-based metrics names are shown below:
For more information, see Creating Charts and Alerts.
Metric is missing logs data
There are several possible reasons for missing data in logs-based metrics:
New log entries might not match your metric's logs filter. A logs-based metric gets data from matching log entries that are received after the metric is created. Stackdriver Logging does not backfill the metric from previous log entries.
New log entries might not contain the correct field, or the data might not be in the right format for extraction by your distribution metric. Check that your field names and regular expressions are correct.
Your metric counts might be delayed. Even though you see countable log entries in the Logs Viewer, it takes up to a minute to update the logs-based metrics in Stackdriver Monitoring.
The log entries you see might be counted late or might not be counted at all, because they are time-stamped too far in the past or future. If a log entry is received more than 10 minutes in the past or future, then the log entry will not be counted in the logs-based metric.
The number of late-arriving entries is recorded for each log in the system logs-based metric,
Example: A log entry matching a logs-based metric arrives late. It has a
timestampof 2:30 PM and a
receivedTimestampof 2:45 PM. This entry will not be counted in the logs-based metric.
Metric has too many time series
You might have a user-defined label in a logs-based metric that has a large number of unique values. The number of time series in a metric depends on the number of different combinations of label values. When the cardinality of the label values is very high, the metric can get throttled and some data points might not be written to the metric. The metric could be slow to load in charts due to the large number of time series it has to process. You might also incur costs for API calls to query time series data; see Stackdriver Monitoring costs for details.
To avoid creating high cardinality metrics:
Check that your label fields and extractor regular expressions match values that have a limited cardinality.
Avoid extracting long text messages that can change, without bounds, as label values.
Avoid extracting numerical values with unbounded cardinality.
Only extract values from labels of known cardinality; for instance, status codes with a set of known values.
Metric name is invalid
When you create a counter or distribution metric, choose a metric name that is unique among the logs-based metrics in your project.
Metric-name strings must not exceed 100 characters and can include only the following characters:
The special characters
The forward slash character
/denotes a hierarchy of pieces within the metric name and cannot be the first character of the name.
Label values are truncated
Values for user-defined labels must not exceed 1,024 bytes.