Regionalize your logs

This page describes how to store your logs in a Cloud Logging bucket in a designated region. For a list of supported regions, see Locations.

This guide walks through this process using the example of redirecting all logs to the europe-west1 region. This process involves the following steps:

  1. Create a log bucket in the designated region for storing the logs.

  2. Redirect the _Default sink to route the logs to the new log bucket.

  3. Search for logs in the Logs Explorer.

  4. (Optional) Update the log retention period.

Overview

In Logging, log buckets are regional resources: the infrastructure that stores, indexes, and searches your logs is located in a specific geographical location. Google manages that infrastructure so that your applications are available redundantly across the zones within that region.

Your organization might be required to store its logs data in specific regions. The primary factors in selecting the region where your logs are stored include meeting your organization's latency, availability, or compliance requirements. When selecting a region for logs storage, consider the locations of the other Google Cloud products and services that your application uses.

Key concepts

The following key concepts apply to data regionality for Logging.

Log Router locations

The Log Router processes all log entries written to the Cloud Logging API. It checks each log entry against existing rules to determine which log entries to store in Logging buckets and which log entries to route to supported destinations using sinks. To reliably route logs, the Log Router also stores the logs temporarily, which buffers against temporary disruptions on any sink.

The Log Router processes logs in the region in which they are received. The Log Router might send logs to a different region based on a sink's definition or if you've opted to share log data with another Google service such as the Security Command Center Threat Detection. Sinks apply to logs equally and regardless of region.

Log bucket locations

Log buckets are the containers in your Google Cloud project, billing account, folder, and organization that store and organize your logs data.

For each Google Cloud project, billing account, folder, and organization, Logging automatically creates two log buckets: _Required and _Default, which are set to the global location. You can't change the location of existing buckets. However, your organization can create a policy that sets a different default location for these buckets. For more information, see Configure default settings for organizations.

You can also create user-defined buckets for any Google Cloud project. When you create a user-defined bucket, you can specify a location region for storing its logs data. After you create the bucket, the location can't be changed, but you can create a new bucket and direct logs to that bucket using sinks. To learn how to set the region for your buckets, see Regionalize your logs.

Logging supports querying logs from multiple regions together, in which case queries are processed in the same locations as the buckets being queried and then aggregated in the region the query was received from to return the results.

Before you begin

To complete the steps in this guide, you need to know the following:

  • In which Google Cloud project do you want to store the logs? In this guide, we use a Google Cloud project called logs-test-project.

  • What is the name and location of the log bucket in which you want to store the logs? In this guide, the bucket name is region-1-logs-bucket, and the location is europe-west1.

  • Which logs do you want to include? In this guide, we include all logs routed by the _Default sink.

Supported regions

When you create your log bucket, you can choose where to store your logs. For a list of supported regions, see Locations.

Create the log bucket

Log buckets store the logs that are routed from other Google Cloud projects, folders, or organizations. For more information, see Configure log buckets.

To create the bucket in the Google Cloud project that you want to store logs in, complete the following steps:

  1. Navigate to the Google Cloud console, or click the following button:

    Go to Google Cloud console

  2. In a terminal, run the following command to create a bucket, replacing the parts in bold with your own information:

     gcloud logging buckets create region-1-logs-bucket \
       --location=europe-west1 \
       --project=logs-test-project
    
  3. Verify that the bucket was created:

    gcloud logging buckets list --project=logs-test-project
    

Redirect the _Default logs sink

You route logs to a log bucket by creating a sink. A sink includes a filter, which selects which log entries to export through the sink, and a destination. In this guide, we update the existing _Default sink to route logs to our bucket, region-1-logs-bucket.

To update the sink, run the following command, replacing the parts in bold with your own information:

gcloud logging sinks update _Default \
  logging.googleapis.com/projects/logs-test-project/locations/europe-west1/buckets/region-1-logs-bucket \
  --log-filter='NOT LOG_ID("cloudaudit.googleapis.com/activity") AND NOT
    LOG_ID("externalaudit.googleapis.com/activity") AND NOT
    LOG_ID("cloudaudit.googleapis.com/system_event") AND NOT
    LOG_ID("externalaudit.googleapis.com/system_event") AND NOT
    LOG_ID("cloudaudit.googleapis.com/access_transparency") AND NOT
    LOG_ID("externalaudit.googleapis.com/access_transparency")' \
  --description="Updated the _Default sink to route logs to the europe-west1 region"

Create a log entry to test your sink

To verify that you updated the sink properly, complete the following steps:

  1. Send a test log message to your regionalized bucket using the gcloud logging write command, and then wait for a few minutes. For example:

    gcloud logging write TEST_LOG_NAME "Test to route logs to region-1-logs-bucket" --project=logs-test-project
    
  2. In the navigation panel of the Google Cloud console, select Logging, and then select Logs Explorer:

    Go to Logs Explorer

  3. In the Log field pane, select the Global resource type.

  4. Your test log entry displays in the Query results panel.

Search logs in the Google Cloud console

After setting the permissions in the previous section, go to the Google Cloud console and complete the following steps:

  1. In the navigation panel of the Google Cloud console, select Logging, and then select Logs Explorer:

    Go to Logs Explorer

  2. Select Refine Scope.

  3. On the Refine scope panel, select Scope by storage.

  4. Select region-1-logs-bucket.

  5. Click Apply.

  6. The Logs Explorer refreshes to show logs from your bucket.

    For information on using the Logs Explorer, refer to Using the Logs Explorer.

[Optional] Updating the bucket's log retention period

To change the retention period for your logs in your bucket, run the following command:

gcloud logging buckets update region-1-logs-bucket \
  --location=europe-west1 --project=logs-test-project \
  --retention-days=14

Next steps