Aggregate and store your organization's logs

This document describes how to manage the logs generated by the resources contained in your Google Cloud organization by using a non-intercepting aggregated sink.

You can configure an aggregated sink to be intercepting or non-intercepting, depending on whether you want control over which logs can be queried in, or routed through the sinks in child resources. In this tutorial, you create an aggregated sink that routes your organization's audit logs to a log bucket. You can configure the filter of the sink to route other types of log entries. For more information about aggregated sinks, see Collate and route organization- and folder-level logs to supported destinations.

In this tutorial, you perform the following steps:

1. Creating the Cloud Logging bucket for storing the aggregated logs.

2. Creating a non-intercepting aggregated sink at the organization level to route the logs to the new log bucket.

3. Configuring read access to the new log bucket.

4. Querying and viewing your logs from the Logs Explorer page.

Note: If you're getting started with Google Cloud, you can aggregate and store your organization's logs as part of the Google Cloud setup process.

Before you begin

Ensure the following:

• To get the permissions that you need to create log buckets and aggregated log sinks, ask your administrator to grant you the following IAM roles:

  • Logs Configuration Writer (roles/logging.configWriter) on the project
  • Logs Configuration Writer (roles/logging.configWriter) on your organization

  For more information about granting roles, see Manage access.

  You might also be able to get the required permissions through custom roles or other predefined roles.

• To get the permissions that you need to grant principals roles, ask your administrator to grant you the Owner (roles/owner) IAM role on a project.

• If you use VPC Service Controls, then you must add an ingress rule to the service perimeter. For more information about VPC Service Controls limitations, see Aggregated sinks and VPC Service Controls limitations.

Create a log bucket

Note: You can skip this step if you use a log bucket that already exists. You can use the _Default bucket in your Google Cloud project, but we recommend that you create a separate bucket for this use case.

Log buckets store the logs that are routed from other Google Cloud projects, folders, or organizations. For more information, see Configure log buckets.

To create the log bucket in the Google Cloud project that you want to aggregate logs into, complete the following steps:

1. Navigate to the Google Cloud console, or click the following button:

   Go to Google Cloud console

2. In a Cloud Shell terminal, run the gcloud logging buckets create command.

   Before running the following command, make the following replacements:

   • BUCKET_NAME: The name of the log bucket.
   • LOCATION: The location of the log bucket.
   • PROJECT_ID: The identifier of the project in which to create the log bucket.

   Execute the gcloud logging buckets create command:

    gcloud logging buckets create BUCKET_NAME \
      --location=LOCATION --project=PROJECT_ID
   

   Note: After you create your log bucket, you can't change your bucket's region.

3. Verify that the log bucket was created:

   gcloud logging buckets list --project=PROJECT_ID
   

4. Optional: Set the retention period of the logs in the bucket. This example extends the retention of logs stored in the bucket to 365 days:

   gcloud logging buckets update BUCKET_NAME \
     --location=LOCATION --project=PROJECT_ID \
     --retention-days=365
   

Create the aggregated sink

You can route logs to a log bucket by creating a sink. A sink includes an inclusion filter, an optional exclusion filter, and a destination. In this tutorial, the destination is your new log bucket. For more information about sinks, see Route logs to supported destinations.

Set up the sink at the organization level

To create an aggregated sink, complete the following steps:

1. Run the gcloud logging sinks create command.

   Before running the following command, make the following replacements:

   • SINK_NAME: The name of the log sink.
   • PROJECT_ID: The identifier of the project which stores the log bucket.
   • LOCATION: The location of the log bucket.
   • BUCKET_NAME: The name of the log bucket.
   • ORGANIZATION_ID: The identifier of the organization.

   Execute the gcloud logging sinks create command:

   gcloud logging sinks create SINK_NAME \
   logging.googleapis.com/projects/PROJECT_ID/locations/LOCATION/buckets/BUCKET_NAME  \
     --log-filter='logName:cloudaudit.googleapis.com' \
     --description="Audit logs from my organization" \
     --organization=ORGANIZATION_ID \
     --include-children
   

   The --include-children flag is important so that logs from all the Google Cloud projects within your organization are also included. For more information, see Collate and route organization-level logs to supported destinations.

   Note: The non-intercepting sink you created doesn't redirect log entries to the log bucket. Instead, your log entries are stored in the log bucket, but also sent to the originating project for routing by project-level sinks. As a result, a log entry might be stored in multiple log buckets. To avoid storing duplicate copies of logs, either create an intercepting sink or add an exclusion filter to your project-level sink.

2. Verify that the sink was created:

   gcloud logging sinks list --organization=ORGANIZATION_ID
   

3. Get the name of the service account:

   gcloud logging sinks describe SINK_NAME --organization=ORGANIZATION_ID
   

   The output looks similar to the following:

   writerIdentity: serviceAccount:[e-mail]
   

4. Copy the value of the serviceAccount field into your clipboard.

   Note: To route logs to a resource protected by a service perimeter, you must add the service account for that sink to an access level and then assign it to the destination service perimeter. This isn't necessary for non-aggregated sinks. For details, see VPC Service Controls: Cloud Logging.

Grant access to the sink

After creating the sink, you must grant your sink permission to write to your log bucket. You can grant permission by using the Google Cloud console or by editing the Identity and Access Management (IAM) policy, as described in Set destination permissions.

Note: If you're manually editing the IAM policy, you must include the serviceAccount: prefix, but in the console you must exclude the prefix.

To grant your sink permission to write logs, do the following:

1. In the navigation panel of the Google Cloud console, select IAM:

   Go to IAM

2. Select the Google Cloud project that contains your log bucket.

3. Click Grant access.

4. In the New principals field, add the service account without the serviceAccount: prefix.

5. In the Select a role menu, select Logs Bucket Writer.

6. Click Save.

Generate logs to assist in sink verification

To verify that your aggregated sink is properly configured, try the following:

1. Generate audit logs that should be routed to your log bucket.

   • If you have many Google Cloud projects in your organization, then you might have enough audit-log traffic that you don't need to create any for validation purposes. Go to the next step.

   • Otherwise, go to a different project, create a Compute Engine VM instance, and then delete the instance you created. Audit logs are written when a VM is created, started, and deleted.

2. Follow the procedure in the section titled View logs in the Logs Explorer page to view your audit logs. Be sure that you select the _AllLogs view.

Configure read access to a log bucket

Note: To grant a principal a role on a Google Cloud project, you must have a role of Owner (roles/owner) on the Google Cloud project.

When you create a log bucket, Cloud Logging automatically creates a log view named _AllLogs. This view includes every log entry stored in the log bucket.

In the following steps, you grant a principal the role of roles/logging.viewAccessor along with an IAM condition that restricts the grant to the view named _AllLogs:

1. In the navigation panel of the Google Cloud console, select IAM:

   Go to IAM

   Make sure you've selected the Google Cloud project you're using to aggregate the logs.

2. Click Add.

3. In the New principal field, add a principal.

4. In the Select a role menu, select Logs Views Accessor.

   Note: If you don't add a condition to this role, then the principal has access to all log views on all user-defined log buckets in the Google Cloud project.

5. Add an IAM condition to the binding:

   1. Click Add condition, enter a title and description.
   2. In the Condition type menu, scroll to Resource, and then select Name.
   3. In the Operator menu, select Ends with.

   4. In the Value field, enter the full name of the log view:

      locations/LOCATION/buckets/BUCKET_NAME/views/_AllLogs
      

   5. Click Save to save the condition.

6. Click Save to save the binding.

View logs in the Logs Explorer page

To view the logs in your log bucket, do the following:

1. In the navigation panel of the Google Cloud console, select Logging, and then select Logs Explorer:

   Go to Logs Explorer

2. Select Refine Scope.

3. On the Refine scope panel, select Scope by storage.

4. Select the log view, or log views, whose log entries you want to see. For example, to view all logs, select the view named _AllLogs.

5. Click Apply.

   The Logs Explorer refreshes to show logs from your log bucket.

   For information about using the Logs Explorer, see Using the Logs Explorer.