Managing logs buckets

This page describes how to create and manage logs buckets. Logs buckets are Cloud Logging storage containers in your Google Cloud projects that hold your logs data. You can create logs sinks to route all, or just a subset, of your logs to any logs bucket. This flexibility allows you to choose the Cloud project in which your logs are stored and what other logs are stored with them.

To create a logs bucket, you must have a Google Cloud project with billing enabled.

Overview

For each Google Cloud project, Logging automatically creates two logs buckets: _Required and _Default. All logs generated in the project are stored in the _Required and _Default logs buckets, which live in the project that the logs are generated in. The following describes the role and purpose of the _Required and _Default buckets:

• _Required: This bucket holds Admin Activity audit logs, System Event audit logs, and Access Transparency logs, and retains them for 400 days. You aren't charged for the logs stored in _Required, and the retention period of the logs stored here cannot be modified. You cannot delete this bucket.

• _Default: This bucket holds all other ingested logs in a Google Cloud project except for the logs held in the _Required bucket. Standard Cloud Logging pricing applies to these logs. Log entries held in the _Default bucket are retained for 30 days, unless you apply custom retention rules. You can't delete this bucket, but you can disable the _Default log sink that routes logs to this bucket.

For these buckets, Logging automatically creates log sinks named _Required and _Default that route logs to the corresponding buckets.

Logs buckets only have regional availability, including those created in the global region. Setting location to global means that Logging doesn't specify where it physically stores the logs.

Logging also creates some default views that can be used to access logs in a bucket:

• The _AllLogs view is available on all buckets and shows all logs in the bucket.

• The _Default view is only available for the _Default bucket and shows all logs except Data Access audit logs.

For more information on how Cloud Logging routes and stores your logs data, see Logs Router overview. For information on the logs bucket API methods, refer to the LogBucket reference documentation.

Limitations

While logs buckets are generally available, other features are only available in the Preview stages. Be aware of the following limitations:

• Custom views on a logs bucket are currently in Preview.

• You cannot create logs-based metrics for logs buckets; these metrics apply only to a single Cloud project.

Managing buckets

Using the gcloud command-line tool and the Google Cloud Console, you can create, update, and delete your custom logs buckets.

Creating a logs bucket

To create a custom logs bucket for your project, complete the following steps. You can create a maximum of 10 buckets per project.

gcloud

To create a bucket in your project, run the gcloud beta logging buckets create command:

gcloud beta logging buckets create BUCKET_ID --location=LOCATION OPTIONAL_FLAGS

For example:

gcloud beta logging buckets create my-bucket --location global --description "My first bucket"

Note: Once you create your bucket, you can't change your bucket's region. If you need a bucket in a different region, you must create a new bucket in that region, redirect the appropriate sinks to the new bucket, and then delete the old bucket.

Console

To create a bucket in your project, complete the following steps:

1. From the Logging menu, select Logs Storage.

   Go to Logs Storage

2. Click Create Logs Bucket.

3. Enter a Name and Description for your bucket.

4. Optionally, to set a custom retention period or bucket region, select Next.

5. In the Retention field, enter the number of days, between 1 to 3650 days, that you want Cloud Logging to retain your logs.

6. Select your bucket's region by clicking the Select Logs Bucket Region drop-down menu and selecting the region in which you want your bucket.

   Note: Once you create your bucket, you can't change your bucket's region. If you need a bucket in a different region, you must create a new bucket in that region, redirect the appropriate sinks to the new bucket, and then delete the old bucket.

7. Click Create bucket. Your new bucket appears in the Logs bucket list.

After creating a bucket, you can configure Logs Views to control who can access the logs in your new bucket and which logs are accessible to them.

Updating a logs bucket

To update the attributes of your bucket, complete the following steps.

gcloud

To update your bucket's attributes, run the gcloud beta logging buckets update command:

gcloud beta logging buckets update BUCKET_ID --location=LOCATION UPDATED_ATTRIBUTES

For example:

gcloud beta logging buckets update my-bucket --location=global --description "Updated description"

Console

To update your bucket's attributes, complete the following steps:

1. From the Logging menu, select Logs Storage.

   Go to Logs Storage

2. For the bucket you want to update, click More .

3. Select Edit bucket.

4. Edit your bucket as needed.

5. Click Update bucket.

Locking a logs bucket

You can lock a bucket to prevent anyone from updating or immediately deleting it. To lock a bucket, complete the following steps.

Warning: You can't delete a locked bucket until all of the logs routed to the bucket are no longer retained in accordance with the bucket's retention period.

GCLOUD

To lock your bucket, run the gcloud beta logging buckets update command with the --locked flag:

gcloud beta logging buckets update BUCKET_ID --location=LOCATION --locked

For example:

gcloud beta logging buckets update my-bucket --location=global --locked

Viewing logs buckets

To list the logs buckets associated with a Google Cloud project, do the following.

gcloud

To list your project's logs buckets, run the gcloud beta logging buckets list command:

gcloud beta logging buckets list

Logs buckets have the following attributes:

• LOCATION
• BUCKET_ID
• RETENTION_DAYS
• LIFECYCLE_STATE
• LOCKED
• CREATE_TIME
• UPDATE_TIME

To view the details for a specific bucket, run the gcloud beta logging buckets describe command. For example, to view the details for the _Default logs bucket, run this command:

gcloud beta logging buckets describe _Default --location=global

Console

The Logs Storage page displays your project's logs buckets.

Go to Logs Storage

Logs buckets have the following attributes:

• Name
• Description
• Retention period
• Region
• Status

To view the details for a specific bucket, click More and select View bucket details.

Deleting a logs bucket

To delete a logs bucket, complete the following steps.

gcloud

To delete a logs bucket, run the gcloud beta logging buckets delete command:

gcloud beta logging buckets delete BUCKET_ID --location=LOCATION

Console

To delete a logs bucket, complete the following steps:

1. From the Logging menu, select Logs Storage.

   Go to Logs Storage

2. For the bucket you want to delete, click More .

3. Select Delete bucket.

4. On the confirmation panel, click Delete.

5. On the Logs Storage page, your bucket has an indicator that it's pending deletion. The bucket, including all the logs in it, is deleted after 7 days.

Note: A deleted bucket stays in this pending state for 7 days, and Logging continues to route logs to the bucket during that time. To stop routing logs to a deleted bucket, you can delete the log sinks that have that bucket as a destination, or you can modify the filter for the sinks to stop routing logs to the deleted bucket.

Restoring a deleted logs bucket

You can restore, or undelete, a bucket that's in the pending deletion state.

GCLOUD

To restore a logs bucket that is pending deletion, run the gcloud beta logging buckets undelete command:

gcloud beta logging buckets undelete BUCKET_ID --location=LOCATION

CONSOLE

To restore a logs bucket that is pending deletion, complete the following steps:

1. From the Logging menu, select Logs Storage.

   Go to Logs Storage

2. For the bucket you want to restore, click More .

3. Select Restore deleted bucket.

4. On the confirmation panel, click Restore.

5. On the Logs Storage page, the pending-deletion indicator is removed from your bucket.

Troubleshooting and common questions

If you encounter problems when using logs buckets, refer to the following troubleshooting steps and answers to common questions.

Why do I see logs for a project even though I excluded them from my _Default sink?

If you're accessing logs in a centralized project and see logs that you excluded from the _Default sink, you might be viewing the logs under one of the following conditions:

• Viewing the logs using the Legacy Logs Viewer, which doesn't support viewing centralized logs.

• Viewing the logs using the Logs Explorer with Scope by project selected in the Refine scope panel, which shows you logs generated by the project regardless of where you store them.

To verify that you correctly excluded the logs, you can select Scope by storage in the Refine scope panel for the Logs Explorer and select the _Default bucket in your project. You shouldn't see the excluded logs anymore.

Why can't I create logs-based metrics in the logs bucket?

You cannot create logs-based metrics for logs buckets; these metrics apply only to a single Cloud project.

What's next

For information on addressing common use cases with logs buckets, refer to the following documentation:

• Aggregating your organization's logs into a central logs bucket.

• Configuring multi-tenant logging for Google Kubernetes Engine (GKE) clusters.

• Regionalizing your project's logs using logs buckets.

• You can configure custom retention periods as needed on each of your logs buckets and the _Default bucket.