Managing logs buckets

This page describes how to create and manage logs buckets.

Logs buckets are Cloud Logging storage containers in your Google Cloud projects that hold your logs data. You can create logs sinks to route all, or just a subset, of your logs to any logs bucket. This flexibility allows you to choose the Cloud project in which your logs are stored and what other logs are stored with them.

From their name, logs buckets might sound like Cloud Storage buckets, but logs buckets are a feature of Cloud Logging storage. Unlike logs data that is stored in Cloud Storage, the logs that you store in Cloud Logging are indexed, optimized, and delivered to let you analyze your logs in real time.

Overview

For each Cloud project, Logging automatically creates two logs buckets: _Required and _Default. All logs generated in the project are stored in the _Required and _Default logs buckets, which live in the project that the logs are generated in. The following describes the role and purpose of the _Required and _Default buckets:

  • _Required: This bucket holds Admin Activity audit logs, System Event audit logs, and Access Transparency logs, and retains them for 400 days. You aren't charged for the logs stored in _Required, and the retention period of the logs stored here cannot be modified. You cannot delete this bucket.

  • _Default: This bucket holds all other ingested logs in a Google Cloud project except for the logs held in the _Required bucket. Standard Cloud Logging pricing applies to these logs. Log entries held in the _Default bucket are retained for 30 days, unless you apply custom retention rules. You can't delete this bucket, but you can disable the _Default log sink that routes logs to this bucket.

For these buckets, Logging automatically creates log sinks named _Required and _Default that route logs to the corresponding buckets.

Logs buckets only have regional availability, including those created in the global region. Setting location to global means that Logging doesn't specify where it physically stores the logs.

Logging also creates some default views that can be used to access logs in a bucket:

  • The _AllLogs view is available on all buckets and shows all logs in the bucket.

  • The _Default view is only available for the _Default bucket and shows all logs except Data Access audit logs.

For more information on Logs Views, see Managing Logs Views on your Logs Buckets.

For more information on how Cloud Logging routes and stores your logs data, see Logs Router overview. For information on the logs bucket API methods, refer to the LogBucket reference documentation.

Access control

For managing logs buckets, you need to consider the following IAM roles and permissions. For the full list of Logging access controls, see Access controls.

Logs buckets activity User access IAM permissions IAM roles and recommended access control settings
Managing logs bucket configurations Who can create, list, update, delete, undelete, and view details of logs buckets. logging.buckets.{create,list,get,update,delete,undelete} These permissions are available as part of the Logging Configuration Writer or Logging Admin roles. You can also create a custom role with more limited permissions.
Writing log entries to a logs bucket Who can write log entries to a specific logs bucket. logging.buckets.write

If a log sink routes log entries to a logs bucket in the same Cloud project, the log sink doesn't require permissions.

If a log sink routes logs to a log bucket in a different Cloud project, you must grant the log sink the logging.buckets.write permission. Use the Logs Bucket Writer role to grant this permission in the Cloud project that contains the logs bucket. This role should be granted for a log sink's service account using an IAM condition that matches a specific logs bucket.

For an example on setting an IAM condition on a logs sink using the gcloud tool, see Routing logs from one project to a bucket in a different project.

Reading log entries from a logs bucket Who can view logs entries from a specific logs bucket using a logs view. logging.views.{access,listLogs,listResourceKeys,listResourceValues} Use the Logs View Accessor role to grant this permission. This role should be granted using an IAM condition that matches a specific logs view.

For information on setting this IAM condition on a Logs View, see Adding users to a Logs View.

Cloud Logging recommends providing the least privileged access when configuring Identity and Access Management roles and permissions. For more information on using least privilege access, see Using IAM securely.

Limitations

While logs buckets are generally available, other features are only available in the Preview stages. Be aware of the following limitations:

Managing buckets

Using the gcloud command-line tool and the Google Cloud Console, you can create, update, and delete your custom logs buckets.

Creating a logs bucket

You need the logging.buckets.create permission to create a logs bucket. This permission is available as part of the Logging Configuration Writer role. You can also create a custom role with more limited permissions. For the full list of Logging access controls, see Access controls.

To create a custom logs bucket for your project, complete the following steps:

gcloud

To create a bucket in your project, run the gcloud logging buckets create command:

gcloud logging buckets create BUCKET_ID --location=LOCATION OPTIONAL_FLAGS

For example:

gcloud logging buckets create my-bucket --location global --description "My first bucket"

Console

To create a bucket in your project, complete the following steps:

  1. From the Logging menu, select Logs Storage.

    Go to Logs Storage

  2. Click Create Logs Bucket.

  3. Enter a Name and Description for your bucket.

  4. Optionally, to set a custom retention period or bucket region, select Next.

  5. In the Retention field, enter the number of days, between 1 to 3650 days, that you want Cloud Logging to retain your logs.

  6. Select your bucket's region by clicking the Select Logs Bucket Region drop-down menu and selecting the region in which you want your bucket.

  7. Click Create bucket. Your new bucket appears in the Logs bucket list.

After creating a bucket, you can configure Logs Views to control who can access the logs in your new bucket and which logs are accessible to them.

Updating a logs bucket

The logging.buckets.update permission is required to update a logs bucket. This permission is available as part of the Logging Configuration Writer role. You can also create a custom role with more limited permissions. For the full list of Logging access controls, see Access controls.

To update the attributes of your bucket, complete the following steps:

gcloud

To update your bucket's attributes, run the gcloud logging buckets update command:

gcloud logging buckets update BUCKET_ID --location=LOCATION UPDATED_ATTRIBUTES

For example:

gcloud logging buckets update my-bucket --location=global --description "Updated description"

Console

To update your bucket's attributes, complete the following steps:

  1. From the Logging menu, select Logs Storage.

    Go to Logs Storage

  2. For the bucket you want to update, click More .

  3. Select Edit bucket.

  4. Edit your bucket as needed.

  5. Click Update bucket.

Locking a logs bucket

The logging.buckets.update permission is required to lock a logs bucket. This permission is available as part of the Logging Configuration Writer role. You can also create a custom role with more limited permissions. For the full list of Logging access controls, see Access controls.

You can lock a bucket to prevent anyone from updating or immediately deleting it. To lock a bucket, do the following:

gcloud

To lock your bucket, run the gcloud logging buckets update command with the --locked flag:

gcloud logging buckets update BUCKET_ID --location=LOCATION --locked

For example:

gcloud logging buckets update my-bucket --location=global --locked

Listing logs buckets

The logging.buckets.list permission is required to list logs bucket details. This permission is available as part of the Logging Configuration Writer role. You can also create a custom role with more limited permissions. For the full list of Logging access controls, see Access controls.

To list the logs buckets associated with a Cloud project, and to see details such as retention settings, do the following:

gcloud

Run the gcloud logging buckets list command:

gcloud logging buckets list

You see the following attributes for the logs buckets:

  • LOCATION: The region in which the bucket's data is stored.
  • BUCKET_ID: The name given to the bucket when it was created.
  • RETENTION_DAYS: The number of days that the bucket's data will be stored by Cloud Logging.
  • LIFECYCLE_STATE: Indicates whether the bucket is pending deletion by Cloud Logging.
  • LOCKED: Whether the bucket is locked or unlocked.
  • CREATE_TIME: A timestamp that indicates when the bucket was created.
  • UPDATE_TIME: A timestamp that indicates when the bucket was last modified.

You can also view the attributes for just one bucket. For example, to view the details for the _Default logs bucket, run the gcloud logging buckets describe command:

gcloud logging buckets describe _Default --location=global

Console

Go to the Logs Storage page:

Go to Logs Storage

You see a table Logs buckets that lists the buckets associated with the current Cloud project.

The table lists the following attributes for each logs bucket:

  • Name: The name given to the bucket when it was created.
  • Description: The description given to to the bucket when it was created.
  • Retention period: The number of days that the bucket's data will be stored by Cloud Logging.
  • Region: The geographic location in which the bucket's data is stored.
  • Status: Whether the bucket is locked or unlocked.

If a bucket is pending deletion by Cloud Logging, its table entry is annotated with a warning .

Viewing logs buckets details

The logging.buckets.get permission is required to view the details of a logs bucket. This permission is available as part of the Logging Configuration Writer role. You can also create a custom role with more limited permissions. For the full list of Logging access controls, see Access controls.

To view the details of a single logs bucket, do the following:

gcloud

Run thegcloud beta logging buckets describe command:

gcloud beta logging buckets describe _Default --location=global

You see the following attributes for the logs bucket:

  • createTime: A timestamp that indicates when the bucket was created.
  • description: The description given to the bucket when it was created.
  • lifecycleState: Indicates whether the bucket is pending deletion by Cloud Logging.
  • name: The name given to the bucket when it was created.
  • retentionDays: The number of days that the bucket's data will be stored by Cloud Logging.
  • updateTime: A timestamp that indicates when the bucket was last modified.

Console

Go to the Logs Storage page:

Go to Logs Storage

On the logs bucket, click More > View bucket details.

The dialog box lists the following attributes for the logs bucket:

  • Name: The name given to the bucket when it was created.
  • Description: The description given to to the bucket when it was created.
  • Retention period: The number of days that the bucket's data will be stored by Cloud Logging.
  • Region: The geographic location in which the bucket's data is stored.

Deleting a logs bucket

The logging.buckets.delete permission is required to delete a logs bucket. This permission is available as part of the Logging Configuration Writer role. You can also create a custom role with more limited permissions. For the full list of Logging access controls, see Access controls.

To delete a logs bucket, do the following:

gcloud

To delete a logs bucket, run the gcloud logging buckets delete command:

gcloud logging buckets delete BUCKET_ID --location=LOCATION

Console

To delete a logs bucket, complete the following steps:

  1. From the Logging menu, select Logs Storage.

    Go to Logs Storage

  2. For the bucket you want to delete, click More .

  3. Select Delete bucket.

  4. On the confirmation panel, click Delete.

  5. On the Logs Storage page, your bucket has an indicator that it's pending deletion. The bucket, including all the logs in it, is deleted after 7 days.

Restoring a deleted logs bucket

The logging.buckets.undelete permission is required to restore a logs bucket. This permission is available as part of the Logging Configuration Writer role. You can also create a custom role with more limited permissions. For the full list of Logging access controls, see Access controls.

You can restore, or undelete, a logs bucket that's in the pending deletion state. To restore a logs bucket, do the following:

gcloud

To restore a logs bucket that is pending deletion, run the gcloud logging buckets undelete command:

gcloud logging buckets undelete BUCKET_ID --location=LOCATION

Console

To restore a logs bucket that is pending deletion, complete the following steps:

  1. From the Logging menu, select Logs Storage.

    Go to Logs Storage

  2. For the bucket you want to restore, click More .

  3. Select Restore deleted bucket.

  4. On the confirmation panel, click Restore.

  5. On the Logs Storage page, the pending-deletion indicator is removed from your bucket.

Writing to and reading from a logs bucket

Writing to a logs bucket

The logging.logEntries.create permission is required to write log entries to a Cloud project, folder, or organization. This permission is available as part of the Logs Writer and Logging Admin roles. For the full list of Logging access controls, see Access controls.

You don't directly write logs to a logs bucket. Rather, you write logs to a Cloud project, folder, or organization. The sinks in the parent resource then route the logs to destinations, including logs buckets. A sink routes logs to a logs bucket destination when the logs match the sink's filter and the sink has permission to route the logs to the logs bucket.

If a log sink routes logs to a logs bucket in the same Cloud project, the log sink doesn't require permissions.

If a log sink routes logs to a logs bucket in a different Cloud project, you must grant the log sink the logging.buckets.write permission. Use the Logs Bucket Writer role to grant this permission in the Cloud project that contains the logs bucket. This role should be granted for a log sink's service account using an IAM condition that matches a specific logs bucket.

For instructions on granting permissions for a service account to write to a logs bucket in a different Cloud project, see Destination permissions.

Reading from a logs bucket

The logging.views.listLogs permission is required to read logs from a logs bucket. This permission is available as part of the Logs View Accessor role. For the full list of Logging access controls, see Access controls.

Each logs bucket has a set of logs views. To read logs from a logs bucket, you need access to a logs view on the logs bucket. For more information on logs views, see Managing Logs Views.

We recommend setting these permissions using an IAM condition. For more information on adding users to a logs view using an IAM condition, see Adding users to a Logs View.

To read logs from a logs bucket, do the following:

gcloud

To read logs from a logs bucket, run the gcloud beta logging read command:

gcloud beta logging read --bucket=BUCKET_ID --location=LOCATION --view=VIEW_ID

Console

For instructions on reading logs from a logs bucket, see Refine scope.

Troubleshooting and common questions

If you encounter problems when using logs buckets, refer to the following troubleshooting steps and answers to common questions.

Why do I see logs for a project even though I excluded them from my _Default sink?

If you're accessing logs in a centralized project and see logs that you excluded from the _Default sink, you might be viewing the logs under one of the following conditions:

  • Viewing the logs using the Legacy Logs Viewer, which doesn't support viewing centralized logs.

  • Viewing the logs using the Logs Explorer with Scope by project selected in the Refine scope panel, which shows you logs generated by the project regardless of where you store them.

To verify that you correctly excluded the logs, you can select Scope by storage in the Refine scope panel for the Logs Explorer and select the _Default bucket in your project. You shouldn't see the excluded logs anymore.

Why can't I create logs-based metrics in the logs bucket?

You cannot create logs-based metrics for logs buckets; these metrics apply only to a single Cloud project.

What's next

For information on addressing common use cases with logs buckets, refer to the following documentation: