Log-based metrics on log buckets

Stay organized with collections Save and categorize content based on your preferences.

This document provides information about bucket-level log-based metrics. Bucket-level log-based metrics are user-defined log-based metrics that evaluate log entries routed to a specific log bucket. With bucket-level log-based metrics, you can create log-based metrics that extract data from logs in the following cases:

  • Logs that are routed from one project to a bucket in another project.
  • Logs that are routed into a bucket through an aggregated sink.

Like project-level log-based metrics, bucket-level log-based metrics are defined within a Google Cloud project, not at the folder or organization level. If you want to track the folder or organization from which a log entry was routed to the bucket, then you can define labels in the log-based metric to record that information.

Bucket-level log-based metrics differ from project-level log-based metrics in some ways, and this document describes the differences.

Projects and metrics scopes

Bucket-level log-based metrics, like all log-based metrics, are defined in a specific Cloud project. For bucket-level log-based metrics, you must define the metric in the project that includes the log bucket.

By default, only the project that defines a bucket-level log-based metric can see the metric data for the log bucket. However, if your project appears in the metrics scope of other projects, those projects can also see your metrics. For more information about log-based-metrics and metrics scopes, see Visibility to metrics scopes.

The Cloud projects that route logs to the bucket do not need to be in the metrics scope of any project. When the logs are received at the bucket, the metric data derived from the routed logs is visible to any project that includes the bucket's project in its metrics scope. If a project routing logs is not in the metrics scope of any project, then other information in the routing project is not visible.

When using bucket-level log-based metrics, the following statements apply:

  • You define the log-based metric in the project with the destination bucket.
  • Your bucket-level log-based metrics evaluates all log entries routed to the bucket. The evaluated set of log entries includes any log entries that are produced in the same project as the bucket and routed to the bucket.
  • The metric data is visible to any other project that includes the project with the bucket in its metrics scope.
  • Projects routing logs don't need to be in any project's metrics scope.

Create or modify log-based metrics on log buckets

During the public preview, you can create or modify log-based metrics on buckets only by using the Google Cloud CLI, specifically the gcloud alpha logging metrics command group. The create and update commands in this alpha group support the --bucket-name flag.

When you create a filter for the log entries that you want to count in your metric, you can use regular expressions. The length of a filter can't exceed 20,000 characters.

When creating log-based metrics, you must specify a metric name. In Cloud Monitoring, your metric name is prefixed with logging.googleapis.com/user when it is created. Choose a metric name that is unique among log-based metrics in your Cloud project. Some naming restrictions apply; see Troubleshoot log-based metrics for details.

Create a counter metric

To create a counter-type log-based metric with only the default labels, use the following command:

gcloud alpha logging metrics create METRIC_NAME \
      --description="METRIC_DESCRIPTION" \
      --log-filter="FILTER" \
      --bucket-name=BUCKET

For example, the following command creates a log-based counter metric called error_count on the log bucket my-test-bucket.

gcloud alpha logging metrics create error_count \
     --description="Errors in syslog." \
     --log-filter="resource.type=gce_instance AND logName:logs/syslog AND severity>=ERROR" \
     --bucket-name=projects/my-project/locations/global/buckets/my-test-bucket

To create a counter-type log-based metric with additional user-specified labels, create a file that contains a specification of a LogMetric in JSON or YAML format. Then use the following command to read the configuration from your file:

gcloud alpha logging metrics create METRIC_NAME --config-from-file FILENAME

Create a distribution metric

To create a distribution-type log-based metric, create a file that contains a specification of a LogMetric in JSON or YAML format. Then use the following command to read the configuration from your file:

gcloud alpha logging metrics create METRIC_NAME --config-from-file FILENAME

For information about describing the histogram buckets for a distribution, see Histogram buckets.

Update a log-based metric

To update the description, filter, or bucket associated with a log-based metric, use the following command:

gcloud alpha logging metrics update METRIC_NAME \
  [ --description="METRIC_DESCRIPTION" | --log-filter="FILTER" ] \
  [ --bucket-name=BUCKET ]

When you use the gcloud alpha logging metrics update METRIC_NAME command:

  • You can specify either the --description flag or the --log-filter flag but not both, and you can specify either in conjunction with the --bucket-name flag.

  • If you change the bucket associated with your log-based metric, then any metric data collected before the change no longer reflects the current configuration. The metric data collected for the previous bucket is not removed.

  • The METRIC_NAME value must match an existing log-based metric in your Google Cloud project, or this command creates a new metric.

To update any fields except the METRIC_NAME, create a file that contains the revised specification of your LogMetric in JSON or YAML format. Then use the following command:

gcloud alpha logging metrics update METRIC_NAME --config-from-file FILENAME

Replace the FILENAME variable with the name of the file that you created.

View your bucket-level log-based metrics

During the public preview, you can view your bucket-level log-based metrics only in Cloud Monitoring, for example, by using Metrics Explorer.

To view the metrics for a monitored resource by using Metrics Explorer, do the following:

  1. In the Google Cloud console, go to Monitoring or use the following button:
    Go to Monitoring
  2. In the Monitoring navigation pane, click Metrics Explorer.
  3. In the toolbar, select the Explorer tab.
  4. Select the Configuration tab.
  5. Expand the Select a metric menu, and then use the submenus to select a resource type and metric. For example, to chart the CPU utilization of a virtual machine, do the following:
    1. (Optional) To reduce the menu's options, enter part of the metric name in the Filter bar. For this example, enter utilization.
    2. In the Active resources menu, select VM instance.
    3. In the Active metric categories menu, select Instance.
    4. In the Active metrics menu, select CPU utilization.

Bucket-level log-based metrics write time-series data against the logging_bucket monitored-resource type.

When user-defined log-based metrics are ingested into Monitoring, the metric name you specified is prefixed with logging.googleapis.com/user.

For more information about using Metrics Explorer, see Create charts with Metrics Explorer.

Labels for bucket-level log-based metrics

Bucket-level log-based metrics provide a default log label. This label holds the value of the LOG_ID portion of the logName field in the log entry, for example, cloudresourcemanager.googleapis.com%2Factivity.

You can customize the default labels for bucket-level log-based metrics just as you can the labels of project-level log-based metrics. For example, you can use labels in the log-based metric to record the folder or organization from which a log entry originated, even though bucket-level log-based metrics can't be created for folders or organizations.

For more information about labels for log-based metrics, see Log-based metrics labels.

Monitored-resource type: logging_bucket

Bucket-level log-based metrics use a new monitored-resource type, logging_bucket, when writing time-series data to Cloud Monitoring. The logging_bucket resource type includes the following labels:

  • project_id: The identifier of your Google Cloud project, a string like my-test-project.
  • bucket_id: The name of the log bucket associated with the log-based metric, a string like my-test-bucket.
  • location: The location of the log bucket.
  • monitored_resource_type: The type of the monitored resource that produced this log entry; the value comes from the type field from the MonitoredResource structure contained in the resource field of the log entry.
  • source_resource_container: The identifier for the project, folder, or organization from which the log entry was sent, a string in one of the following forms:
    • projects/PROJECT_ID
    • organizations/ORGANIZATION_ID
    • folders/FOLDER_ID