You can create an aggregated export sink that can export log entries from all the projects, folders, and billing accounts of a Google Cloud organization. For instance, you might aggregate and export audit log entries from an organization's projects to a central location.
Concept
Without the aggregated export feature, an export sink is limited to exporting log entries from the exact resource in which the sink was created: a Google Cloud project, organization, folder, or billing account.
To use the aggregated export feature, create a sink in a Google Cloud
organization or folder and set the sink's includeChildren
parameter to True
.
That sink can then export log entries from the organization or folder, plus
(recursively) from any contained folders, billing accounts, or projects. You
can use the sink's query to specify log entries from projects, resource types,
or named logs.
For information about export sinks and how to create them, go to Exporting logs in the API and using the command-line tool to create sinks.
Creating the export destination
The export destination for log sinks has to be created before the sink, through
either gcloud
command-line tool, Cloud Console, or the Google Cloud APIs.
The supported export destinations for export sinks are:
The export destination can be created in any Google Cloud project, in any organization, if the service account from the log sink has permissions to write to the export destination.
Creating an aggregated export sink
To create an aggregated export sink in Google Cloud folders, billing
accounts, or organizations, you can use
either the Stackdriver Logging API or
gcloud
command-line tool.
Logging API
To create a logging sink, use organizations.sinks.create, folders.sinks.create or billingAccounts.sinks.create in the Logging API. Prepare the arguments to the method as follows:
Set the
parent
parameter to be the Google Cloud organization, folder, or billing account in which to create the sink. The parent must be one of the following:organizations/[ORGANIZATION_ID]
folders/[FOLDER_ID]
billingAccounts/[BILLING_ACCOUNT_ID]
In the LogSink object in the method request body:
Set
includeChildren
toTrue
.Set the
query
property, keeping in mind that log entries from all your projects will be matched against the query.For some examples of useful queries, go to Using queries with aggregated exports.
Set the remaining LogSink fields as you would for any sink. For more information, see Creating sinks.
Call organizations.sinks.create or folders.sinks.create to create the sink.
Retrieve the service account name used to create the sink from the API response.
Give that service account permission to write to your export destination.
If you do not have permission to make that change to the export destination, then send the service account name to someone who can make that change for you.
For more information about granting service accounts permissions for resources, review Granting roles to service accounts.
gcloud
To create a logging sink, use the
logging sinks create
command.
Supply the sink name, export destination, logs query, and the ID of the folder, billing account, or organization.
For example, set up an aggregated export on the folder level like this:
gcloud logging sinks create [SINK_NAME] \ storage.googleapis.com/[BUCKET_NAME] --include-children \ --folder=[FOLDER_ID] --log-filter="logName:activity"
Notes:
To create a sink on the organization level, replace
--folder=[FOLDER_ID]
with--organization=[ORGANIZATION_ID]
. For a billing account, replace with--billing-account=[BILLING_ACCOUNT_ID]
.For the sink to include all projects within the organization, the
--include-children
flag must be set, even when the--organization
flag is passed tocreate
. When set tofalse
(the default), a sink will only export logs from the host resource.You need the Logs Configuration Writer Cloud IAM role for the parent to create the sink. For more information about Logging Cloud IAM roles, review the Access control guide.
For some examples of useful queries, go to Using queries with aggregated exports.
Retrieve the service account name used to create the sink from the command output.
Give that service account permission to write to your export destination.
If you do not have permission to make that change to the export destination, then send the service account name to someone who can make that change for you.
For more information about granting service accounts permissions for resources, review Granting roles to service accounts.
Using queries with aggregated exports
Like any export sink, your aggregated export sink contains a query that selects individual log entries. For more details about queries, go to Advanced logs queries.
Following are some examples of query comparisons that are useful when using the aggregated export feature. Some examples use the following notation:
:
is the substring operator. Do not substitute the=
operator....
represents any additional query comparisons.- Variables are indicated by brackets
[]
). Replace them with valid values.
Choosing audit logs
To export audit logs, choose one of the following sample comparisons:
log_name:"/logs/cloudaudit.googleapis.com"
log_name:"/logs/cloudaudit.googleapis.com%2Factivity" AND...
log_name:"/logs/cloudaudit.googleapis.com%2Fdata_access" AND...
log_name:"/logs/cloudaudit.googleapis.com%2Fsystem_event" AND...
For information about audit logs, go to Cloud Audit Logs.
Choosing export source
To export logs from specific projects, folders, or organizations, use one of the following sample comparisons:
logName:"projects/[PROJECT_ID]/logs/" AND ...
logName:("projects/[PROJECT_A_ID]/logs/" OR "projects/[PROJECT_B_ID]/logs/") AND ...
logName:"folders/[FOLDER_ID]/logs/" AND ...
logName:"organizations/[ORGANIZATION_ID]/logs/" AND ...
Choosing resources
To export logs from only a specific resource in a project, use multiple comparisons to specify the resource exactly:
logName:"projects/[PROJECT_ID]/logs" AND resource.type=[RESOURCE_TYPE] AND resource.labels.instance_id=[INSTANCE_ID]
For a list of resource types, go to Monitored resource types.
Sampling log entries
To export a random sample of log entries, add the sample
built-in
function. For example, to export only 10% of the log entries matching your
current query, use this addition:
sample(insertId, 0.10) AND ...
For more information, review the
sample
function.
For more information about Stackdriver Logging queries, go to Advanced logs queries.