Aggregated sinks

This page describes how to create an aggregated sink that can route log entries from all the Google Cloud projects, folders, and billing accounts of a Google Cloud organization. For instance, you might aggregate and route audit log entries from an organization's Cloud projects to a central destination.

Concept

Without the aggregated sink feature, sinks are limited to routing log entries from the exact resource in which the sink was created: a Google Cloud project, organization, folder, or billing account.

To use aggregated sinks, create a sink in a Google Cloud organization or folder and set the sink's includeChildren parameter to True. That sink can then route log entries from the organization or folder, plus (recursively) from any contained folders, billing accounts, or projects. You can use the sink's filter to specify log entries from projects, resource types, or named logs.

For information about sinks and how to create them, go to Exporting logs in the API and using the command-line tool to create sinks.

Create the destination

The destination for log sinks has to be created before the sink, through either gcloud command-line tool, Cloud Console, or the Google Cloud APIs.

The supported destinations for sinks are the following:

The destination can be created in any Cloud project in any organization, if the service account from the log sink has permissions to write to the destination.

Create an aggregated sink

To create an aggregated sink in Google Cloud folders, billing accounts, or organizations, you can use either the Cloud Logging API or gcloud command-line tool.

API

To create a logging sink, use organizations.sinks.create, folders.sinks.create or billingAccounts.sinks.create in the Logging API. Prepare the arguments to the method as follows:

  1. Set the parent parameter to be the Google Cloud organization, folder, or billing account in which to create the sink. The parent must be one of the following:

    • organizations/ORGANIZATION_ID
    • folders/FOLDER_ID
    • billingAccounts/BILLING_ACCOUNT_ID
  2. In the LogSink object in the method request body, do the following:

    • Set includeChildren to True.

    • Set the filter property. Logging matches all log entries from all your projects against the filter.

      For some examples of useful filters, go to Create filters for aggregated sinks.

    • Set the remaining LogSink fields as you would for any sink. For more information, see Creating sinks.

  3. Call organizations.sinks.create or folders.sinks.create to create the sink.

  4. Retrieve the service account name from the writer_identity field returned from the API response.

  5. Give that service account permission to write to your sink destination.

    If you don't have permission to make that change to the sink destination, then send the service account name to someone who can make that change for you.

    For more information about granting service accounts permissions for resources, review the following links for your sink destination:

    Cloud Storage buckets Pub/Sub topics BigQuery tables Cloud Logging buckets

command-line

To create a logging sink, use the logging sinks create command.

  1. Supply the sink name, sink destination, filter, and the ID of the folder, billing account, or organization.

    For example, set up an aggregated sink on the folder level like this:

    gcloud logging sinks create SINK_NAME \
    storage.googleapis.com/BUCKET_NAME --include-children \
    --folder=FOLDER_ID --log-filter="logName:activity"

    Notes:

    • To create a sink on the organization level, replace --folder=[FOLDER_ID] with --organization=[ORGANIZATION_ID]. For a billing account, replace with --billing-account=[BILLING_ACCOUNT_ID].

    • For the sink to include all projects within the organization, the --include-children flag must be set, even when the --organization flag is passed to create. When set to false (the default), a sink will only route logs from the host resource.

    • You need the Logs Configuration Writer IAM role for the parent to create the sink. For more information about Logging IAM roles, review the Access control guide.

    • For some examples of useful filters, go to the gcloud command-line tool examples for Creating sinks.

  2. Retrieve the service account name used to create the sink from the command output.

  3. Give that service account permission to write to your sink destination.

    If you don't have permission to make that change to the sink destination, then send the service account name to someone who can make that change for you.

    For more information about granting service accounts permissions for resources, review the following links for your sink destination:

Create filters for aggregated sinks

Like any sink, your aggregated sink contains a filter that selects individual log entries. For more details about filters, go to Logging query language.

Following are some examples of filter comparisons that are useful when using the aggregated sinks feature. Some examples use the following notation:

  • : is the substring operator. Don't substitute the = operator.
  • ... represents any additional filter comparisons.
  • Variables are indicated by colored text. Replace them with valid values.

Select the log source

To route logs from specific Cloud projects, folders, or organizations, use one of the following sample comparisons:

logName:"projects/PROJECT_ID/logs/" AND ... 
logName:("projects/PROJECT_A_ID/logs/" OR "projects/PROJECT_B_ID/logs/") AND ... 
logName:"folders/FOLDER_ID/logs/" AND ... 
logName:"organizations/ORGANIZATION_ID/logs/" AND ... 

Select the monitored resource

To route logs from only a specific monitored resource in a Cloud project, use multiple comparisons to specify the resource exactly:

logName:"projects/PROJECT_ID/logs" AND
resource.type=RESOURCE_TYPE AND
resource.labels.instance_id=INSTANCE_ID

For a list of resource types, go to Monitored resource types.

Select a sample of log entries

To route a random sample of log entries, add the sample built-in function. For example, to route only ten percent of the log entries matching your current filter, use this addition:

sample(insertId, 0.10) AND ...

For more information, review the sample function.

For more information about Cloud Logging filters, go to Logging query language.

Pricing

There aren't any charges for routing logs, but destination charges might apply. For details, review the appropriate product's pricing page: