This document describes how to manage the logs generated by the resources contained in your Google Cloud organization by using a non-intercepting aggregated sink.
You can configure an aggregated sink to be intercepting or non-intercepting, depending on whether you want control over which logs can be queried in, or routed through the sinks in child resources. In this tutorial, you create an aggregated sink that routes your organization's audit logs to a Google Cloud project. Then, in the Google Cloud project, you create a log sink that routes the aggregated audit logs to a log bucket.
For more information about aggregated sinks, see Collate and route organization- and folder-level logs to supported destinations.
In this tutorial, you perform the following steps:
You start by creating a log bucket and a log sink in the Google Cloud project where you want to store your aggregated logs.
Next, you create a non-intercepting aggregated sink at the organization level to route logs to the Google Cloud project.
Next, you configure read access to log views on the new log bucket.
Lastly, you query and view your logs from the Logs Explorer page.
Before you begin
Ensure the following:
-
To get the permissions that you need to create log buckets and aggregated log sinks, ask your administrator to grant you the following IAM roles:
-
Logs Configuration Writer (
roles/logging.configWriter
) on the project -
Logs Configuration Writer (
roles/logging.configWriter
) on your organization
For more information about granting roles, see Manage access.
You might also be able to get the required permissions through custom roles or other predefined roles.
-
Logs Configuration Writer (
-
To get the permissions that you need to grant principals roles, ask your administrator to grant you the Owner (
roles/owner
) IAM role on a project. If you use VPC Service Controls, then you must add an ingress rule to the service perimeter. For more information about VPC Service Controls limitations, see Aggregated sinks and VPC Service Controls limitations.
Create a log bucket
Log buckets store the logs that are routed from other Google Cloud projects, folders, or organizations. For more information, see Configure log buckets.
To create the log bucket in the Google Cloud project that you want to aggregate logs into, complete the following steps:
-
Go to the Google Cloud console:
In a Cloud Shell terminal, run the
gcloud logging buckets create
command.Before running the following command, make the following replacements:
- BUCKET_NAME: The name of the log bucket.
- LOCATION: The location of the log bucket.
- PROJECT_ID: The identifier of the project in which to create the log bucket.
Execute the
gcloud logging buckets create
command:gcloud logging buckets create BUCKET_NAME \ --location=LOCATION --project=PROJECT_ID
Verify that the log bucket was created:
gcloud logging buckets list --project=PROJECT_ID
Optional: Set the retention period of the logs in the bucket. This example extends the retention of logs stored in the bucket to 365 days:
gcloud logging buckets update BUCKET_NAME \ --location=LOCATION --project=PROJECT_ID \ --retention-days=365
Create the project-level log sink
You route log entries to a log bucket by creating a sink. A sink includes an inclusion filter, an optional exclusion filter, and a destination. In this tutorial, the destination is your new log bucket. For more information about sinks, see Route logs to supported destinations.
To create a sink that routes log entries to the log bucket you just created,
make the following replacements and run the
gcloud logging sinks create
command:
- PROJECT_LEVEL_SINK_NAME: The name of the project-level log sink.
SINK_DESTINATION: The log bucket where your logs are routed. The destination path format for a log bucket is the following:
logging.googleapis.com/projects/PROJECT_ID/locations/LOCATION/buckets/BUCKET_NAME
PROJECT_ID: The identifier of the project in which to create the log sink. Set this flag to the same project where you created the log bucket.
Include the following flags:
--log-filter
: Use this flag to set a filter that matches the log entries you want to include in your sink. In this tutorial, the filter is set to select all audit log entries. If you don't set a filter, then all logs from your Google Cloud project are routed to the destination.--description
: Use this flag to describe the purpose or use case for the sink.
Execute the following command:
gcloud logging sinks create PROJECT_LEVEL_SINK_NAME SINK_DESTINATION --project=PROJECT_ID --log-filter='logName:cloudaudit.googleapis.com' \ --description="Audit logs from my organization" \
Create the aggregated sink
Aggregated sinks combine and route log entries from the resources contained by an organization or folder to a destination.
In this tutorial, you create an aggregated sink that is non-intercepting. This means every log entry that is routed by the aggregated sink is also routed by the sinks in the resource in which the log entry originates. For example, an audit log that originates in a project is routed by the aggregated sink and the sinks in that project. Therefore, it is possible for you to store multiple copies of a log entry.
You can create intercepting sinks. For more information, see Collate and route organization- and folder- level logs to support destinations.
Set up the sink at the organization level
To create an aggregated sink that is non-intercepting and that routes log entries to a project, complete the following steps:
Run the
gcloud logging sinks create
command.Before running the following command, make the following replacements:
- SINK_NAME: The name of the log sink.
- PROJECT_ID: The identifier of the project which stores the log bucket.
- ORGANIZATION_ID: The identifier of the organization.
Execute the
gcloud logging sinks create
command:gcloud logging sinks create SINK_NAME \ logging.googleapis.com/projects/PROJECT_ID \ --log-filter='logName:cloudaudit.googleapis.com' \ --description="Audit logs from my organization" \ --organization=ORGANIZATION_ID \ --include-children
The
--include-children
flag is important so that logs from all the Google Cloud projects within your organization are also included. For more information, see Collate and route organization-level logs to supported destinations.Verify that the sink was created:
gcloud logging sinks list --organization=ORGANIZATION_ID
Get the name of the service account:
gcloud logging sinks describe SINK_NAME --organization=ORGANIZATION_ID
The output looks similar to the following:
writerIdentity: serviceAccount:o1234567890-ORGANIZATION_ID@gcp-sa-logging.iam.gserviceaccount.com
Copy the value of the
serviceAccount
field into your clipboard.
Grant access to the sink
After creating the aggregated sink, you must grant permission for the sink to write logs to the project that you set as the destination. You can grant permission by using the Google Cloud console or by editing the Identity and Access Management (IAM) policy, as described in Set destination permissions.
To grant your sink permission to write logs, do the following:
-
In the Google Cloud console, go to the IAM page:
If you use the search bar to find this page, then select the result whose subheading is IAM & Admin.
Select the Google Cloud project that contains your log bucket.
Click
Grant access.In the New principals field, add the service account without the
serviceAccount:
prefix.In the Select a role menu, select Logs Writer.
Click Save.
Generate logs to assist in sink verification
To verify that your aggregated sink is properly configured, try the following:
Generate audit logs that should be routed to your log bucket.
If you have many Google Cloud projects in your organization, then you might have enough audit-log traffic that you don't need to create any for validation purposes. Go to the next step.
Otherwise, go to a different project, create a Compute Engine VM instance, and then delete the instance you created. Audit logs are written when a VM is created, started, and deleted.
Follow the procedure in the section titled View logs in the Logs Explorer page to view your audit logs. Be sure that you select the
_AllLogs
view.
Configure read access to a log view on a log bucket
When you create a log bucket, Cloud Logging automatically creates a
log view named _AllLogs
.
This view includes every log entry stored in the log bucket.
To restrict a principal to having access only to specific log entries, create a log view, and then do one of the following:
Grant them the role of
roles/logging.viewAccessor
along with an IAM condition that restricts the grant to the log view.On the IAM policy associated with the log view, grant a principal access. We recommend this approach when you create a large number of log views.
For more information about these two approaches, see Control access to a log view.
In the following steps, you grant a principal the role of
roles/logging.viewAccessor
along with an IAM condition
that restricts the grant to the view named _AllLogs
:
-
In the Google Cloud console, go to the IAM page:
If you use the search bar to find this page, then select the result whose subheading is IAM & Admin.
Make sure you've selected the Google Cloud project you're using to aggregate the logs.
Click
Add.In the New principal field, add a principal.
In the Select a role menu, select Logs Views Accessor.
Add an IAM condition to the binding:
- Click Add condition, enter a title and description.
- In the Condition type menu, scroll to Resource, and then select Name.
- In the Operator menu, select Ends with.
In the Value field, enter the full name of the log view:
locations/LOCATION/buckets/BUCKET_NAME/views/_AllLogs
Click Save to save the condition.
Click Save to save the binding.
View logs in the Logs Explorer page
To view the logs in your log bucket, do the following:
-
In the Google Cloud console, go to the Logs Explorer page:
If you use the search bar to find this page, then select the result whose subheading is Logging.
Select Refine Scope.
On the Refine scope panel, select Scope by storage.
Select the log view, or log views, whose log entries you want to see. For example, to view all logs, select the view named
_AllLogs
.Click Apply.
The Logs Explorer refreshes to show logs from your log bucket.
For information about using the Logs Explorer, see Using the Logs Explorer.