Shape the future of software operations and make your voice heard by taking the 2021 State of DevOps survey.

Overview of logs exports

This page provides a conceptual overview of export sinks, which control how Cloud Logging routes logs.

Using sinks, you can route some or all of your logs to supported destinations or exclude log entries from being stored in Cloud Logging. You might want to control how your logs are routed for the following reasons:

  • To store logs that are unlikely to be read but that must be retained for compliance purposes.
  • To use big-data analysis tools on your logs.
  • To stream your logs to other applications, other repositories, or third parties.

How sinks work

All log entries written to the Cloud Logging API pass through the Log Router. Every time a log entry arrives in a Cloud project, folder, billing account, or organization resource, Logging compares the log entry to the filters of the sinks associated with the resource.

Depending on the log sink's configuration, which includes filters and a destination, every log entry received by Cloud Logging falls into one or more of these categories:

  • Stored in Cloud Logging and not routed elsewhere
  • Stored in Cloud Logging and routed to a supported destination
  • Not stored in Cloud Logging but routed to a supported destination
  • Neither stored in Cloud Logging nor routed elsewhere
    • These logs are excluded entirely

Cloud Logging provides two predefined log sinks for each Google Cloud project: _Required and _Default. All logs that are generated in a Google Cloud project are automatically processed through these two log sinks and then are stored in the correspondingly named _Required and _Default log buckets.

Log sinks act independently of each other. Regardless of how the predefined log sinks process your log entries, you can create sinks to route some or all of your logs to various supported destinations or to exclude them entirely from being stored by Cloud Logging.

Sinks are usually created in Google Cloud projects. To set up sinks at the organization, folder, or billing account levels, use aggregated sinks.

For more details on how sinks route logs, see Log Router overview.

Supported destinations

Routing logs involves creating a sink with a filter that selects the log entries you want to route, and choosing a destination from the following options:

  • Cloud Storage: JSON files stored in Cloud Storage buckets; provides inexpensive, long-term storage.
  • BigQuery: Tables created in BigQuery datasets; provides big data analysis capabilities.
  • Pub/Sub: JSON-formatted messages delivered to Pub/Sub topics; supports third-party integrations, such as Splunk, with Logging.
  • Cloud Logging: Log entries held in log buckets; provides storage in Cloud Logging with customizable retention periods.

Sink properties and terminology

Sinks have a number of properties, including the aforementioned filters and a destination:

  • Sink identifier: A name for the sink. For example, my-vm-error-sink.

  • Parent resource: The Google Cloud resource in which you create the sink. The parent can be any of the following:

    • projects/PROJECT_ID
    • folders/FOLDER_ID
    • billingAccounts/BILLING_ACCOUNT_ID
    • organizations/ORGANIZATION_ID

    Sinks route logs that belong to its parent resource. You can also use aggregated sinks to combine and route logs from all the Cloud projects, folders, and billing accounts of a Google Cloud organization.

    The full resource name of a sink includes its parent resource and sink identifier. For example:

    projects/PROJECT_ID/sinks/SINK_ID

  • Inclusion filter: Selects which log entries to route through this sink. For inclusion filter examples, see Sample queries.

  • Exclusion filter: Selects which log entries to explicitly exclude from routing, even if the log entries match the sink's inclusion filter.

    Note that a sink can contain multiple exclusion filters. If any log entry matches any of the filters, the log entry is excluded from routing.

  • Destination: A place to send the log entries matching your filter. The following are the names of the supported destinations:

    • Cloud Storage buckets : storage.googleapis.com/BUCKET_ID

    • BigQuery datasets: bigquery.googleapis.com/projects/PROJECT_ID/datasets/DATASET_ID

    • Pub/Sub topics: pubsub.googleapis.com/projects/PROJECT_ID/topics/TOPIC_ID

    • Logging log buckets: logging.googleapis.com/projects/PROJECT_ID/locations/REGION/buckets/BUCKET_ID

    In the Logs Explorer, you can also specify a Custom destination option when creating a sink to send your logs from one Cloud project to a destination in another Cloud project.

    You can route logs to destinations in any Cloud project, if the destination authorizes the sink's service account as a writer.

  • Writer identity: A service account name. The destination's owner must give this service account permission to write to the destination. When routing logs, Logging adopts this identity for authorization. For increased security, new sinks get a unique service account:

    GENERATED_ID_1@GENERATED_ID_2.iam.gserviceaccount.com

    For more information, see destination permissions.

  • includeChildren: This property is described in Aggregated sinks. It is only relevant to sinks created for organizations, folders, or billing accounts.

For more details about sinks, see the LogSink type.

Access control

To create or modify a sink, you must have the Identity and Access Management roles Owner or Logging/Logs Configuration Writer in the sink's parent resource. To view existing sinks, you must have the IAM roles Viewer or Logging/Logs Viewer in the sink's parent resource. For more information, see Access control.

To route logs to a destination, the sink's writer service account must be permitted to write to the destination. For more information about writer identities, read Sink properties on this page.

Pricing

Cloud Logging doesn't charge to route logs, but destination charges might apply. For details, review the appropriate service's pricing details:

Note also that if you send and then exclude your Virtual Private Cloud flow logs from Cloud Logging, VPC flow log generation charges apply in addition to the destination charges.

Next steps

Route your logs

To learn how to create and manage sinks to route your logs, see the following pages:

Find and use your logs

To learn about the format of routed log entries and how the logs are organized in destinations, see Using exported logs.

Explore Logging routing scenarios

The following tutorials describe scenarios for which you might want to route logs. Each tutorial details the requirements, setup, and usage, and shows how to share the routed logs.