This document describes how to create aggregated sinks. Aggregated sinks let you combine and route logs that are generated by the Google Cloud resources in your organization or folder to a centralized location.
Overview
Aggregated sinks combine and route log entries from the resources contained by an organization or folder to a destination.
If you want control over which log entries can be queried in these resources, or routed through the sinks in these resources, then you can configure an aggregated sink to be non-intercepting or intercepting:
A non-intercepting aggregated sink routes log entries through sinks in child resources. With this sink, you maintain visibility of log entries in the resources in which they were generated. Non-intercepting sinks aren't visible to child resources.
For example, you might create a non-intercepting aggregated sink that routes all log entries generated from the folders contained by an organization to a central log bucket. The log entries are stored in the central log bucket, and also in the resources in which the log entries were generated.
An intercepting aggregated sink prevents log entries from being routed through sinks in child resources, except for
_Required
sinks. This sink can be useful in preventing duplicate copies of log entries from being stored in multiple places.For example, consider Data Access audit logs, which can be large in volume and expensive to store multiple copies of. If you've enabled Data Access audit logs, you might create a folder-level intercepting sink that routes all Data Access audit logs to a central project for analysis. This intercepting sink also prevents sinks in child resources from routing copies of the logs elsewhere.
Intercepting sinks prevent logs being passed through the Log Router of child resources, unless the logs also match the
_Required
sink. Because the logs are intercepted, the logs don't count towards log-based metrics or log-based alerting policies in the child resources. You can view intercepting sinks in the Log Router page of child resources.
For information about managing sinks, see Route logs to supported destinations: Manage sinks.
You can create up to 200 sinks per folder or organization.
Supported destinations
You can use non-intercepting aggregated sinks to route log entries within or between the same organizations and folders to the following destinations:
Cloud Logging bucket: Provides storage in Cloud Logging. A log bucket can store log entries that are received by multiple Google Cloud projects. The log bucket can be in the same project in which log entries originate, or in a different project. For information about viewing log entries stored in log buckets, see Query and view logs overview and View logs routed to Cloud Logging buckets.
You can combine your Cloud Logging data with other data by upgrading a log bucket to use Log Analytics, and then creating a linked dataset, which is a read-only dataset that can be queried by the BigQuery Studio and Looker Studio pages.
BigQuery dataset: Provides storage of log entries in a writeable BigQuery dataset. The BigQuery dataset can be in the same project in which log entries originate, or in a different project. You can use big data analysis capabilities on the stored log entries. For information about viewing log entries routed to BigQuery, see View logs routed to BigQuery.
- Cloud Storage bucket: Provides storage of log entries in Cloud Storage. The Cloud Storage bucket can be in the same project in which log entries originate, or in a different project. Log entries are stored as JSON files. For information about viewing log entries routed to Cloud Storage, see View logs routed to Cloud Storage.
Pub/Sub topic: Provides support for third-party integrations. Log entries are formatted into JSON and then routed to a Pub/Sub topic. The topic can be in the same project in which log entries originate, or in a different project. For information about viewing log entries routed to Pub/Sub, see View logs routed to Pub/Sub.
Google Cloud project: Route log entries to another Google Cloud project. In this configuration, the sinks in the destination project processes the log entries.
Best practices for intercepting sinks
When you create an intercepting sink, we recommend that you do the following:
Consider whether child resources need independent control of routing their log entries. If a child resource needs independent control of certain log entries, ensure that your intercepting sink doesn't route those log entries.
Add contact information to the description of an intercepting sink. This might be helpful if those who manage the intercepting sink are different from those who manage the projects whose log entries are being intercepted.
Test your sink configuration by first creating a non-intercepting aggregated sink to ensure the correct log entries are being routed.
Aggregated sinks and VPC Service Controls
The following limitations apply when you use aggregated sinks and VPC Service Controls:
Aggregated sinks can access data from projects inside a service perimeter. To restrict aggregated sinks from accessing data inside a perimeter, we recommend using IAM to manage Logging permissions.
VPC Service Controls doesn't support adding folder or organization resources to service perimeters. Therefore, you can't use VPC Service Controls to protect folder- and organization-level logs, including aggregate logs. To manage Logging permissions at the folder or organizational level, we recommend using IAM.
If you route logs by using a folder- or organization-level sink to a resource that a service perimeter protects, then you must add an ingress rule to the service perimeter. The ingress rule must allow access to the resource from the service account that the aggregated sink uses. For more information, refer to the following pages:
When you specify an ingress or egress policy for a service perimeter, you can't use
ANY_SERVICE_ACCOUNT
andANY_USER_ACCOUNT
as an identity type when you use a log sink to route logs to Cloud Storage resources. However, you can useANY_IDENTITY
as the identity type.
Before you begin
Before you create a sink, ensure the following:
You have a Google Cloud folder or organization with log entries that you can see in the Logs Explorer.
You have one of the following IAM roles for the Google Cloud organization or folder from which you're routing log entries.
- Owner (
roles/owner
) - Logging Admin (
roles/logging.admin
) - Logs Configuration Writer (
roles/logging.configWriter
)
The permissions contained in these roles let you create, delete, or modify sinks. For information about setting IAM roles, see the Logging Access control guide.
- Owner (
You have a resource in a supported destination or have the ability to create one.
The destination must be created before the sink, through either Google Cloud CLI, Google Cloud console, or the Google Cloud APIs. You can create the destination in any Google Cloud project in any organization, but you must ensure that the service account from the sink has permissions to write to the destination.
Select the tab for how you plan to use the samples on this page:
Console
When you use the Google Cloud console to access Google Cloud services and APIs, you don't need to set up authentication.
gcloud
In the Google Cloud console, activate Cloud Shell.
At the bottom of the Google Cloud console, a Cloud Shell session starts and displays a command-line prompt. Cloud Shell is a shell environment with the Google Cloud CLI already installed and with values already set for your current project. It can take a few seconds for the session to initialize.
REST
To use the REST API samples on this page in a local development environment, you use the credentials you provide to the gcloud CLI.
Install the Google Cloud CLI, then initialize it by running the following command:
gcloud init
For more information, see Authenticate for using REST in the Google Cloud authentication documentation.
Create an aggregated sink
Console
To create an aggregated sink for your folder or organization, do the following:
-
In the Google Cloud console, go to the Log Router page:
If you use the search bar to find this page, then select the result whose subheading is Logging.
Select an existing folder or organization.
Select Create sink.
In the Sink details panel, enter the following details:
Sink name: Provide an identifier for the sink; note that after you create the sink, you can't rename the sink but you can delete it and create a new sink.
Sink description (optional): Describe the purpose or use case for the sink.
Do one of the following:
To create an intercepting sink, in the Select sink service menu, select Google Cloud project, and then enter the fully-qualified name for the destination. For information about the syntax, see Destination path formats.
To create a non-intercepting sink, go to the Select sink service menu and do one of the following:
To route log entries to a different Google Cloud project, select Google Cloud project, and then enter the fully-qualified name for the destination. For information about the syntax, see Destination path formats.
To route log entries to a service that is in the same Google Cloud project, select one of the following options:
- Cloud Logging bucket: Select or create a Logging bucket.
- BigQuery dataset: Select or create the particular dataset to receive the routed log entries. You also have the option to use partitioned tables.
- Cloud Storage bucket: Select or create the particular Cloud Storage bucket to receive the routed log entries.
- Pub/Sub topic: Select or create the particular topic to receive the routed log entries.
- Splunk: Select the Pub/Sub topic for your Splunk service.
To route log entries to a service that is in a different Google Cloud project, do the following:
- Select Other resource.
Enter the fully-qualified name for the destination. For information about the syntax, see the Destination path formats.
For example, if your sink destination is a Pub/Sub topic, then the
destination
looks like the following:pubsub.googleapis.com/projects/PROJECT_ID/topics/TOPIC_ID
In the Choose logs to include in sink panel, do one of the following:
To create an intercepting sink, select Intercept logs ingested by this organization and all child resources.
To create a non-intercepting aggregated sink, select Include logs ingested by this resource and all child resources.
Complete the dialog by entering a filter expression in the Build inclusion filter field that matches the log entries you want to include. If you don't set a filter, all log entries from your selected resource are routed to the destination.
For example, you might want to build a filter to route all Data Access audit logs to a single Logging bucket. This filter looks like the following:
LOG_ID("cloudaudit.googleapis.com/data_access") OR LOG_ID("externalaudit.googleapis.com/data_access")
For filter examples, see Create filters for aggregated sinks.
Note that the length of a filter can't exceed 20,000 characters.
Optional: To verify you entered the correct filter, select Preview logs. This opens the Logs Explorer in a new tab with the filter pre-populated.
Optional: In the Choose logs to exclude from the sink panel, do the following:
In the Exclusion filter name field, enter a name.
In the Build an exclusion filter field, enter a filter expression that matches the log entries you want to exclude. You can also use the
sample
function to select a portion of the log entries to exclude.For example, to exclude the log entries from a specific project from being routed to the destination, add the following exclusion filter:
logName:projects/PROJECT_ID
To exclude log entries from multiple projects, use the logical-OR operator to join
logName
clauses.
You can create up to 50 exclusion filters per sink. Note that the length of a filter can't exceed 20,000 characters.
Select Create sink.
Grant the service account for the sink the permission to write log entries to your sink's destination. For more information, see Set destination permissions.
gcloud
To create an aggregated sink, use the
logging sinks create
command:
To create a sink, call the
gcloud logging sinks create
command, and ensure that you include the--include-children
option.Before using the following command, make the following replacements:
- SINK_NAME: The name of the log sink. You can't change the name of a sink after you create it.
- SINK_DESTINATION: The service or project to where you want your log entries routed.
- INCLUSION_FILTER: The inclusion filter for a sink. For filter examples, see Create filters for aggregated sinks.
- FOLDER_ID: The ID of the folder. If you want to create a sink
at the organization level, then replace
--folder=FOLDER_ID
with-- organization=ORGANIZATION_ID
.
Execute the
gcloud logging sinks create
command:gcloud logging sinks create SINK_NAME \ SINK_DESTINATION --include-children \ --folder=FOLDER_ID --log-filter="INCLUSION_FILTER"
You can also provide the following options:
- To create an intercepting sink, include the
--intercept-children
option.
For example, if you're creating an aggregated sink at the folder level and whose destination is a Pub/Sub topic, your command might look like the following:
gcloud logging sinks create SINK_NAME \ pubsub.googleapis.com/projects/PROJECT_ID/topics/TOPIC_ID --include-children \ --folder=FOLDER_ID --log-filter="logName:activity"
Grant the service account for the sink permission to write to your sink destination. For more information, see Set destination permissions.
REST
To create an aggregated sink, use the
organizations.sinks.create
or
folders.sinks.create
Logging API method.
Prepare the arguments to the method as follows:
Set the
parent
field to be the Google Cloud organization or folder in which to create the sink. The parent must be one of the following:organizations/ORGANIZATION_ID
folders/FOLDER_ID
In the
LogSink
object in the method request body, do one of the following:Set
includeChildren
toTrue
.To create an intercepting sink, also set the
interceptChildren
field toTrue
.
Set the
filter
field to match the log entries you want to include.For filter examples, see Create filters for aggregated sinks.
The length of a filter can't exceed 20,000 characters.
Set the remaining
LogSink
fields as you would for any sink. For more information, see Route logs to supported destinations.Call
organizations.sinks.create
orfolders.sinks.create
to create the sink.Grant the service account for the sink permission to write to your sink destination. For more information, see Set destination permissions.
Any changes made to a sink might take a few minutes to apply.
Create filters for aggregated sinks
Like any sink, your aggregated sink contains a filter that selects individual log entries. For examples of filters that you might use to create your aggregated sink, see Sample queries using the Logs Explorer.
Following are some examples of filter comparisons that are useful when using the aggregated sinks feature. Some examples use the following notation:
:
is the substring operator. Don't substitute the=
operator....
represents any additional filter comparisons.- Variables are indicated by colored text. Replace them with valid values.
Note that the length of a filter can't exceed 20,000 characters.
For more details about the filtering syntax, see Logging query language.
Select the log source
For an aggregated sink, for each child resource of the organization or folder, the sink's inclusion and exclusion filters are applied to each log entry that is sent to the child resource. A log entry that matches the inclusion filter and that isn't excluded, is routed.
If you want your sink to route log entries from all child resources, then don't specify a project, folder, or organization in your sink's inclusion and exclusion filters. For example, suppose you configure an aggregated sink for an organization with the following filter:
resource.type="gce_instance"
With the previous filter, log entries with a resource type of Compute Engine instances that are written to any child of that organization are routed by the aggregated sink to the destination.
However, there might be situations where you want to use an aggregated sink
to route log entries only from specific child resources. For example, for
compliance
reasons you might want to store audit logs from specific folders or projects
in their own Cloud Storage bucket. In these situations, configure your
inclusion filter to specify each child resource whose log entries you want
routed. If you want to route log entries from a folder and all projects within
that folder,
then the filter must list the folder and each of the projects contained by
that folder, and also join the statements with an OR
clause.
The following filters restrict log entries to specific Google Cloud projects, folders, or organizations:
logName:"projects/PROJECT_ID/logs/" AND ...
logName:("projects/PROJECT_A_ID/logs/" OR "projects/PROJECT_B_ID/logs/") AND ...
logName:"folders/FOLDER_ID/logs/" AND ...
logName:"organizations/ORGANIZATION_ID/logs/" AND ...
For example, to route only log entries written to Compute Engine instances
that were written to the folder my-folder
, use the following filter:
logName:"folders/my-folder/logs/" AND resource.type="gce_instance"
With the previous filter, log entries written to any resource other than
my-folder
, including log entries written to Google Cloud projects that are
children of my-folder
, aren't routed to the destination.
Select the monitored resource
To route log entries from only a specific monitored resource in a Google Cloud project, use multiple comparisons to specify the resource exactly:
logName:"projects/PROJECT_ID/logs" AND resource.type=RESOURCE_TYPE AND resource.labels.instance_id=INSTANCE_ID
For a list of resource types, see Monitored resource types.
Select a sample of log entries
To route a random sample of log entries, add the sample
built-in
function. For example, to route only ten percent of the log entries matching
your current filter, use this addition:
sample(insertId, 0.10) AND ...
For more information, see the
sample
function.
For more information about Cloud Logging filters, see Logging query language.
Set destination permissions
This section describes how to grant Logging the Identity and Access Management permissions to write log entries to your sink's destination. For the full list of Logging roles and permissions, see Access control.
When you create or update a sink that routes log entries to any destination other than a log bucket in the current project, a service account for that sink is required. Logging automatically creates and manages the service account for you:
- As of May 22, 2023, when you create a sink and no service account for the underlying resource exists, Logging creates the service account. Logging uses the same service account for all sinks in the underlying resource. Resources can be a Google Cloud project, an organization, a folder, or a billing account.
- Before May 22, 2023, Logging created a service account for each sink. As of May 22, 2023, Logging uses a shared service account for all sinks in the underlying resource.
The writer identity of a sink is the identifier of the service account associated with that sink. All sinks have a writer identity unless they write to a log bucket in the current Google Cloud project.
To route log entries to a resource protected by a service perimeter, you must add the service account for that sink to an access level and then assign it to the destination service perimeter. This isn't necessary for non-aggregated sinks. For details, see VPC Service Controls: Cloud Logging.
To set permissions for your sink to route to its destination, do the following:
Console
To get information about the service account for your sink, do the following:
-
In the Google Cloud console, go to the Log Router page:
If you use the search bar to find this page, then select the result whose subheading is Logging.
Select more_vert Menu and then select View sink details. The writer identity appears in the Sink details panel.
If the value of the
writerIdentity
field contains an email address, then proceed to the next step. When the value isNone
, you don't need to configure destination permissions.Copy the sink's writer identity into your clipboard. The
serviceAccount:
string is part of the service account identity. For example:serviceAccount:service-123456789012@gcp-sa-logging.iam.gserviceaccount.com
Add the service account as an IAM principal in the destination project:
-
In the Google Cloud console, go to the IAM page:
If you use the search bar to find this page, then select the result whose subheading is IAM & Admin.
Select the destination project.
Click
Grant access.Grant the service account the required IAM role:
- For Cloud Storage destinations, add the sink's writer identity
as a principal by using IAM, and then grant it the
Storage Object Creator role
(
roles/storage.objectCreator
). - For BigQuery destinations, add the sink's writer identity
as a principal by using IAM, and then grant it the
BigQuery Data Editor role
(
roles/bigquery.dataEditor
). - For Pub/Sub destinations, including Splunk, add the sink's writer identity
as a principal by using IAM, and then grant it the
Pub/Sub Publisher role
(
roles/pubsub.publisher
). - For Logging bucket destinations in different
Google Cloud projects, add the sink's writer identity as a principal by
using IAM, and then grant it the
Logs Bucket Writer role
(
roles/logging.bucketWriter
). - For Google Cloud projects destinations, add the sink's
writer identity as a principal by using IAM, and then grant it the
Logs Writer role
(
roles/logging.logWriter
). Specifically, a principal needs thelogging.logEntries.route
permission.
- For Cloud Storage destinations, add the sink's writer identity
as a principal by using IAM, and then grant it the
Storage Object Creator role
(
-
-
gcloud
Ensure that you have Owner access on the Google Cloud project that contains the destination. If you don't have Owner access to the destination of the sink, then ask a project owner to add the writer identity as a principal.
To get information about the service account for your sink, call the
gcloud logging sinks describe
method.Before using the following command, make the following replacements:
- SINK_NAME: The name of the log sink. You can't change the name of a sink after you create it.
Execute the
gcloud logging sinks describe
command:gcloud logging sinks describe SINK_NAME
If the sink details contain a field labeled
writerIdentity
, then proceed to the next step. When the details don't include awriterIdentity
field, you don't need to configure destination permissions for the sink.Copy the sink's writer identity into your clipboard. The
serviceAccount:
string is part of the service account identity.The writer identity for the service account looks similar to the following:
serviceAccount:service-123456789012@gcp-sa-logging.iam.gserviceaccount.com
To add the service account as an IAM principal in the destination project, call the
gcloud projects add-iam-policy-binding
command.Before using the following command, make the following replacements:
- PROJECT_ID: The identifier of the project.
- PRINCIPAL: An identifier for the principal that you want to
grant the role to. Principal identifiers usually have the following form:
PRINCIPAL-TYPE:ID
. For example,user:my-user@example.com
. For a full list of the formats thatPRINCIPAL
can have, see Principal identifiers. ROLE: An IAM role.
- For Cloud Storage destinations, add the sink's writer identity
as a principal by using IAM, and then grant it the
Storage Object Creator role
(
roles/storage.objectCreator
). - For BigQuery destinations, add the sink's writer identity
as a principal by using IAM, and then grant it the
BigQuery Data Editor role
(
roles/bigquery.dataEditor
). - For Pub/Sub destinations, including Splunk, add the sink's writer identity
as a principal by using IAM, and then grant it the
Pub/Sub Publisher role
(
roles/pubsub.publisher
). - For Logging bucket destinations in different
Google Cloud projects, add the sink's writer identity as a principal by
using IAM, and then grant it the
Logs Bucket Writer role
(
roles/logging.bucketWriter
). - For Google Cloud projects destinations, add the sink's
writer identity as a principal by using IAM, and then grant it the
Logs Writer role
(
roles/logging.logWriter
). Specifically, a principal needs thelogging.logEntries.route
permission.
- For Cloud Storage destinations, add the sink's writer identity
as a principal by using IAM, and then grant it the
Storage Object Creator role
(
Execute the
gcloud projects add-iam-policy-binding
command:gcloud projects add-iam-policy-binding PROJECT_ID --member=PRINCIPAL --role=ROLE
REST
We recommend that you use the Google Cloud console or the Google Cloud CLI to grant a role to the service account.
What's next
Learn how to create log views on a log bucket. Log views let you grant principals read-access to a subset of the log entries stored in a log bucket.
For information about managing existing sinks, see Route logs to supported destinations: Manage sinks.
If you encounter issues as you use sinks to route logs, see Troubleshoot routing and sinks.
To learn how to view your logs in their destinations, as well as how the logs are formatted and organized, see View logs in sink destinations