Aggregated Exports

You can create an aggregated export sink that can export log entries from all the projects, folders, and billing accounts of an organization. As an example, you might use this feature to export audit log entries from an organization's projects to a central location.

Caution: If you enable this feature, your aggregated export sink might export a very large number of log entries. To avoid exporting too many log entries, design your aggregated export sink filter carefully, as described on this page. For details on potential costs, see Stackdriver Pricing.

Concept

Without the aggregated export feature, an export sink is limited to exporting log entries from the exact resource in which the sink was created: a project, organization, folder, or billing account.

With the aggregated export feature, if you create a sink in an organization or folder, and if you set the sink's includeChildren parameter to True, then that sink can export log entries from the organization or folder, plus (recursively) from any contained folders, billing accounts, or projects. You can use the sink's filter to choose log entries from specific projects, specific resource types, or specific named logs.

For information about export sinks and how to create them, see Exporting Logs in the API and using the command-line tool to create sinks.

Creating the export destination

The export destination for log sinks has to be created before the sink, through either the Cloud SDK, the Cloud Console, or the Cloud APIs.

The supported export destinations for export sinks are:

• A Cloud Storage bucket
• A Cloud Pub/Subtopic
• A BigQuery table

The export destination can be created in any GCP project, in any organization, provided the service account from the log sink has permissions to write to the export destination.

Creating an aggregated export sink

To create an aggregated export sink in folders, billing accounts, or organizations, you can use either Google Cloud Endpoints or the gcloud command-line tool.

Note: Export sinks for folders, billing accounts, and organizations cannot be created from Cloud Console. You can use the Cloud Console to create export sinks at the project level only.

Logging API

To create a logging sink, use the organizations.sinks.create, folders.sinks.create or billingAccounts.sinks.create in the Stackdriver Logging API. Prepare the arguments to the method as follows:

1. Set the parent parameter to be the organization, folder, or billing account in which to create the sink. The parent must be one of the following:

   organizations/[ORGANIZATION_ID]
   folders/[FOLDER_ID]
   billingAccounts/[BILLING_ACCOUNT_ID]
   

   Notes:

   • You must have the Logs Configuration Writer IAM role for the parent to create the sink. For more information about Stackdriver Logging IAM roles, see the Access control guide.

2. In the LogSink object in the method request body:

   • Set includeChildren to True.

   • Set the filter property, keeping in mind that log entries from all your projects will be matched against the filter.

     For some examples of useful filters, see Using filters with aggregated exports on this page.

   • Set the remaining LogSink fields as you would for any sink. For more information, see Creating sinks.

3. Call organizations.sinks.create or folders.sinks.create to create the sink.

4. Retrieve the service account name used to create the sink from the API response.

5. Give that service account permission to write to your export destination.

   If you do not have permission to make that change to the export destination, then send the service account name to someone who can make that change for you.

   For more information about granting service accounts permissions for resources, see Granting roles to service accounts.

Cloud SDK

To create a logging sink, use the logging sinks create command.

1. Supply the sink name, export destination, logs filter, and the ID of the folder, billing account, or organization.

   For example, set up an aggregated export on the folder level like this:

   gcloud logging sinks create [SINK_NAME]  \
   storage.googleapis.com/my-folder-bucket --include-children \
   --folder=[FOLDER_ID] --log-filter="logName:activity"
   

   Notes:

   • In order to create a sink on the organization level, replace --folder=[FOLDER_ID] by --organization=[ORG_ID]. For a billing account, replace with --billing-account=[BILLING_ACCOUNT_ID].

   • For the sink to include all projects within the organization, the --include-children flag must be set, even when the --organization flag is passed to create. When set to false (the default), a sink will only export logs from the host resource.

   • You must have the Logs Configuration Writer IAM role for the parent to create the sink. For more information about Stackdriver Logging IAM roles, see the Access control guide.

   • For some examples of useful filters, see Using filters with aggregated exports on this page.

2. Retrieve the service account name used to create the sink from the command output.

3. Give that service account permission to write to your export destination.

   If you do not have permission to make that change to the export destination, then send the service account name to someone who can make that change for you.

   For more information about granting service accounts permissions for resources, see Granting roles to service accounts.

Using filters with aggregated exports

Like any sink, your aggregated export sink contains a filter that selects individual log entries. Following are some examples of filter comparisons that are useful when using the aggregated export feature. For more details about filters, see Advanced Logs Filters. Some examples use the following notation:

• : is the substring operator. Do not substitute the = operator.
• ... represents any additional filter comparisons.

Choosing audit logs

To export audit logs, choose one of the following sample comparisons. The first alternative chooses both Admin Activity and Data Access audit logs:

logName:"/logs/cloudaudit.googleapis.com" AND ...
logName:"/logs/cloudaudit.googleapis.com%2Factivity" AND ...
logName:"/logs/cloudaudit.googleapis.com%2Fdata_access" AND ...

For information about audit logs, see Cloud Audit Logging.

Choosing projects

To export logs from specific projects, folders, or organizations, use one of the following sample comparisons:

logName:"projects/[PROJECT_ID]/logs/" AND ...
logName:("projects/[PROJ_A]/logs/" OR "projects/[PROJ_B]/logs/") AND ...
logName:"folders/[FOLDER_ID]/logs/" AND ...
logName:"organizations/[ORGANIZATION_ID]/logs/" AND ...

Choosing resources

To export logs from only a specific resource in a project, use multiple comparisons to specify the resource exactly:

logName:"projects/[PROJECT_ID]/logs" AND
  resource.type=[RESOURCE_TYPE] AND
  resource.labels.instance_id=[INSTANCE_ID]

For a list of resource types, see Monitored Resource List.

Sampling log entries

To export a random sample of log entries, add the sample built-in function. For example, to export only 10% of the log entries matching your current filter, use this addition:

sample(insertId, 0.10) AND ...

For more information, see Sample function.

For more information about Stackdriver Logging filters, see Advanced log filters.