Managing log buckets

This page describes how to create and manage log buckets.

Before you begin

To get started with log buckets, do the following:

• Ensure you're familiar with log buckets.
• Ensure that you've enabled billing for your Google Cloud project.
• Verify that you and your Cloud project members have the correct role or permissions to create and manage your Cloud project log buckets. See Access control on this page for more information.

Managing buckets

This section describes how to manage your log buckets using the gcloud command-line tool or the Google Cloud Console.

Creating a log bucket

You need the logging.buckets.create permission to create a user-defined log bucket. This permission is available as part of the Logging Configuration Writer role.

You can create a maximum of 10 buckets per Cloud project.

To create a user-defined log bucket for your Cloud project, complete the following steps:

gcloud

To create a log bucket in your Cloud project, run the gcloud logging buckets create command:

gcloud logging buckets create BUCKET_ID --location=LOCATION OPTIONAL_FLAGS

For example:

gcloud logging buckets create my-bucket --location global --description "My first bucket"

Note: After you create your log bucket, you can't change your bucket's region. If you need a bucket in a different region, you must create a new bucket in that region, redirect the appropriate sinks to the new bucket, and then delete the old bucket.

Console

To create a log bucket in your Cloud project, complete the following steps:

1. From the Logging menu, select Logs Storage.

   Go to Logs Storage

2. Click Create Logs Bucket.

3. Enter a Name and Description for your bucket.

4. (Optional) To set a bucket region, click the Select logs bucket region drop-down menu and select the region in which you want your bucket. If you don't select a region, this value sets to global, which means that the logs could be physically stored in any of the regions.

   Note: After you create your log bucket, you can't change your bucket's region. If you need a bucket in a different region, you must create a new bucket in that region, redirect the appropriate sinks to the new bucket, and then delete the old bucket.

5. (Optional) To set a custom retention period for the logs in the bucket, click Next.

   In the Retention field, enter the number of days, between 1 day and 3650 days, that you want Cloud Logging to retain your logs. If you don't customize the retention period, the default is 30 days.

   You can also configure custom retention at a later time.

6. Click Create bucket. Your new bucket appears in the Logs bucket list.

After creating a bucket, you can configure log views to control who can access the logs in your new bucket and which logs are accessible to them.

Updating a log bucket

The logging.buckets.update permission is required to update a log bucket. This permission is available as part of the Logging Configuration Writer role. You can also create a custom role with more limited permissions. For the full list of Logging access controls, see Access controls.

To update the attributes of your bucket, complete the following steps:

gcloud

To update your bucket's attributes, run the gcloud logging buckets update command:

gcloud logging buckets update BUCKET_ID --location=LOCATION UPDATED_ATTRIBUTES

For example:

gcloud logging buckets update my-bucket --location=global --description "Updated description"

Console

To update your bucket's attributes, complete the following steps:

1. From the Logging menu, select Logs Storage.

   Go to Logs Storage

2. For the bucket you want to update, click More .

3. Select Edit bucket.

4. Edit your bucket as needed.

5. Click Update bucket.

Locking a log bucket

Warning: Locking a log bucket is irreversible.

When you lock a bucket against updates, it includes locking the bucket's retention policy. After a retention policy is locked, you can't delete the bucket until every log in the bucket has fulfilled the bucket's retention period.

The logging.buckets.update permission is required to lock a log bucket against updates. This permission is available as part of the Logging Configuration Writer role. You can also create a custom role with more limited permissions. For the full list of Logging access controls, see Access controls.

To prevent anyone from updating or deleting a log bucket, lock the bucket. To lock the bucket, do the following:

gcloud

To lock your bucket, run the gcloud logging buckets update command with the --locked flag:

gcloud logging buckets update BUCKET_ID --location=LOCATION --locked

For example:

gcloud logging buckets update my-bucket --location=global --locked

Console

The Cloud Console doesn't support locking a log bucket.

Listing log buckets

The logging.buckets.list permission is required to list log bucket details. This permission is available as part of the Logging Configuration Writer role. You can also create a custom role with more limited permissions. For the full list of Logging access controls, see Access controls.

To list the log buckets associated with a Cloud project, and to see details such as retention settings, do the following:

gcloud

Run the gcloud logging buckets list command:

gcloud logging buckets list

You see the following attributes for the log buckets:

• LOCATION: The region in which the bucket's data is stored.
• BUCKET_ID: The name given to the bucket when it was created.
• RETENTION_DAYS: The number of days that the bucket's data will be stored by Cloud Logging.
• LIFECYCLE_STATE: Indicates whether the bucket is pending deletion by Cloud Logging.
• LOCKED: Whether the bucket is locked or unlocked.
• CREATE_TIME: A timestamp that indicates when the bucket was created.
• UPDATE_TIME: A timestamp that indicates when the bucket was last modified.

You can also view the attributes for just one bucket. For example, to view the details for the _Default log bucket, run the gcloud logging buckets describe command:

gcloud logging buckets describe _Default --location=global

Console

Go to the Logs Storage page:

Go to Logs Storage

You see a table Logs buckets that lists the buckets associated with the current Cloud project.

The table lists the following attributes for each log bucket:

• Name: The name given to the bucket when it was created.
• Description: The description given to to the bucket when it was created.
• Retention period: The number of days that the bucket's data will be stored by Cloud Logging.
• Region: The geographic location in which the bucket's data is stored.
• Status: Whether the bucket is locked or unlocked.

If a bucket is pending deletion by Cloud Logging, its table entry is annotated with a warning .

Viewing log buckets details

The logging.buckets.get permission is required to view the details of a log bucket. This permission is available as part of the Logging Configuration Writer role. You can also create a custom role with more limited permissions. For the full list of Logging access controls, see Access controls.

To view the details of a single log bucket, do the following:

gcloud

Run the gcloud logging buckets describe command:

gcloud logging buckets describe _Default --location=global

You see the following attributes for the logs bucket:

• createTime: A timestamp that indicates when the bucket was created.
• description: The description given to the bucket when it was created.
• lifecycleState: Indicates whether the bucket is pending deletion by Cloud Logging.
• name: The name given to the bucket when it was created.
• retentionDays: The number of days that the bucket's data will be stored by Cloud Logging.
• updateTime: A timestamp that indicates when the bucket was last modified.

Console

Go to the Logs Storage page:

Go to Logs Storage

On the log bucket, click More > View bucket details.

The dialog box lists the following attributes for the log bucket:

• Name: The name given to the bucket when it was created.
• Description: The description given to to the bucket when it was created.
• Retention period: The number of days that the bucket's data will be stored by Cloud Logging.
• Region: The geographic location in which the bucket's data is stored.

Deleting a log bucket

The logging.buckets.delete permission is required to delete a log bucket. This permission is available as part of the Logging Configuration Writer role. You can also create a custom role with more limited permissions. For the full list of Logging access controls, see Access controls.

To delete a log bucket, do the following:

gcloud

To delete a log bucket, run the gcloud logging buckets delete command:

gcloud logging buckets delete BUCKET_ID --location=LOCATION

Console

To delete a log bucket, complete the following steps:

1. From the Logging menu, select Logs Storage.

   Go to Logs Storage

2. For the bucket you want to delete, click More .

3. Select Delete bucket.

4. On the confirmation panel, click Delete.

5. On the Logs Storage page, your bucket has an indicator that it's pending deletion. The bucket, including all the logs in it, is deleted after 7 days.

Note: A deleted bucket stays in this pending state for 7 days, and Logging continues to route logs to the bucket during that time. To stop routing logs to a deleted bucket, you can delete the log sinks that have that bucket as a destination, or you can modify the filter for the sinks to stop routing logs to the deleted bucket.

Restoring a deleted log bucket

The logging.buckets.undelete permission is required to restore a log bucket. This permission is available as part of the Logging Configuration Writer role. You can also create a custom role with more limited permissions. For the full list of Logging access controls, see Access controls.

You can restore, or undelete, a log bucket that's in the pending deletion state. To restore a log bucket, do the following:

gcloud

To restore a log bucket that is pending deletion, run the gcloud logging buckets undelete command:

gcloud logging buckets undelete BUCKET_ID --location=LOCATION

Console

To restore a log bucket that is pending deletion, complete the following steps:

1. From the Logging menu, select Logs Storage.

   Go to Logs Storage

2. For the bucket you want to restore, click More .

3. Select Restore deleted bucket.

4. On the confirmation panel, click Restore.

5. On the Logs Storage page, the pending-deletion indicator is removed from your bucket.

Writing to a log bucket

The logging.logEntries.create permission is required to write log entries to a Cloud project, folder, or organization. This permission is available as part of the Logs Writer and Logging Admin roles. For the full list of Logging access controls, see Access controls.

You don't directly write logs to a log bucket. Rather, you write logs to a Cloud project, folder, or organization. The sinks in the parent resource then route the logs to destinations, including log buckets. A sink routes logs to a log bucket destination when the logs match the sink's filter and the sink has permission to route the logs to the log bucket.

If a log sink routes logs to a log bucket in the same Cloud project, then the log sink doesn't require permissions.

If a log sink routes logs to a log bucket in a different Cloud project, then you must grant the log sink the logging.buckets.write permission. To grant this permission in the Cloud project that contains the log bucket, use the Logs Bucket Writer role. This role should be granted for a log sink's service account using an IAM condition that matches a specific log bucket.

For instructions on granting permissions for a service account to write to a log bucket in a different Cloud project, see Destination permissions.

Reading from a log bucket

The logging.views.listLogs permission is required to read logs from a log bucket. This permission is available as part of the Logs View Accessor role. For the full list of Logging access controls, see Access controls.

Each log bucket has a set of log views. To read logs from a log bucket, you need access to a log view on the log bucket. For more information on log views, see Managing log views.

Setting these permissions using an IAM condition is recommended. For more information on adding users to a log view using an IAM condition, see Adding users to a log view.

To read logs from a log bucket, do the following:

gcloud

To read logs from a log bucket, run the gcloud logging read command:

gcloud logging read --bucket=BUCKET_ID --location=LOCATION --view=VIEW_ID

Console

For instructions on reading logs from a log bucket, see Refine scope.

Configuring custom retention

Note: Effective July 15, 2021, storage costs will apply to logs data retained longer than 30 days. For pricing, see the Cloud Logging pricing summary . Currently, charges for retaining logs past 30 days are not enforced.

When you create a log bucket, you have the option to customize the period for how long Cloud Logging stores the bucket's logs. You can configure the retention period for any user-defined log bucket and also for the _Default log bucket.

To update the retention period for a log bucket, do the following:

gcloud

For example, to update the retention period for the _Default log bucket, run this gcloud command-line tool command, after setting a value for RETENTION_DAYS:

gcloud logging buckets update _Default --location=global --retention-days=RETENTION_DAYS

For example, to retain the logs in the _Default bucket for a year, run the following command:

gcloud logging buckets update _Default --location=global --retention-days=365

CONSOLE

To update a log bucket's retention period, complete the following steps:

1. From the Logging menu, select Logs Storage.

   Go to Logs Storage

2. For the bucket you want to update, click More .

3. Select Edit bucket.

4. In the Retention field, enter the number of days, between 1 day and 3650 days, that you want Cloud Logging to retain your logs.

5. Click Update bucket. Your new retention duration appears in the Logs bucket list.

Access control

Identity and Access Management roles and permissions govern access to Logging data. Following is a summary of the common roles and permissions a Cloud project member needs to access log buckets.

Cloud Logging recommends limiting privileges when configuring Identity and Access Management roles and permissions. See Using IAM securely for details.

Logs buckets activity	User access	IAM permissions	IAM roles and recommended access control settings
Managing log bucket configurations	Who can create, list, update, delete, undelete, and view details of log buckets.	logging.buckets.{create,list,get,update,delete,undelete}	These permissions are available as part of the Logging Configuration Writer or Logging Admin roles. You can also create a custom role with more limited permissions. Note: These permissions should be granted on the Cloud project, folder, organization, or billing account that contains the bucket.
Writing log entries to a log bucket	Who can write log entries to a specific log bucket.	logging.buckets.write	If a log sink routes log entries to a log bucket in the same Cloud project, the log sink doesn't require permissions. If a log sink routes logs to a log bucket in a different Cloud project, you must grant the log sink the logging.buckets.write permission. Use the Logs Bucket Writer role to grant this permission in the Cloud project that contains the log bucket. This role should be granted for a log sink's service account using an IAM condition that matches a specific log bucket. For an example on setting an IAM condition on a logs sink using the gcloud tool, see Routing logs from one project to a bucket in a different Cloud project.
Reading log entries from a log bucket	Who can view logs entries from a specific log bucket using a log view.	logging.views.{access,listLogs,listResourceKeys,listResourceValues}	Use the Logs View Accessor role to grant this permission. This role should be granted using an IAM condition that matches a specific log view. For information on setting this IAM condition on a log view, see Adding users to a log view.

For the full list of Logging access controls, see Access controls.

Troubleshooting and common questions

If you encounter problems when using log buckets, refer to the following troubleshooting steps and answers to common questions.

Why can't I delete this bucket?

First, make sure you have the correct permissions to delete the bucket.

Next, determine whether the bucket is locked by listing the bucket's attributes. If the bucket is locked, check the bucket's retention period. You can't delete a locked bucket until all of the logs in the bucket have fulfilled the bucket's retention period.

Why do I see logs for a Cloud project even though I excluded them from my _Default sink?

You might be viewing logs in a log bucket in a centralized Cloud project, which aggregates logs from across your organization.

If you're accessing logs in a centralized Cloud project and see logs that you excluded from the _Default sink, you might be viewing the logs under one of the following conditions:

• Viewing the logs using the Legacy Logs Viewer, which doesn't support viewing centralized logs.

  To troubleshoot this issue, switch to using the Logs Explorer.

• Viewing the logs using the Logs Explorer with Scope by project selected in the Refine scope panel, which shows you logs generated by the Cloud project regardless of where you store them.

  To troubleshoot this issue, instead select Scope by storage in the Refine scope panel for the Logs Explorer and then elect the _Default bucket in your Cloud project. You shouldn't see the excluded logs anymore.

Why can't I create logs-based metrics for the log bucket?

Logs-based metrics apply only to a single Google Cloud project. You can't create them for logs buckets or for other Google Cloud resources such as Cloud Billing accounts or organizations.

What's next

For information on the log bucket API methods, refer to the LogBucket reference documentation.

For information on addressing common use cases with log buckets, see the following topics:

• Aggregating your organization's logs into a central logs bucket.

• Configuring multi-tenant logging for Google Kubernetes Engine (GKE) clusters.

• Regionalizing your project's logs using logs buckets.