Routing and storage overview

This page explains how Cloud Logging processes log entries, and describes the key components of Logging routing and storage.

At a high level, this is how Cloud Logging routes and stores log entries:

Figure illustrating how Cloud Logging routes logs entries.

Log Router

Cloud Logging receives log entries through the Cloud Logging API where they pass through the Log Router. The sinks in the Log Router check each log entry against existing inclusion and exclusion filters that determine if the log entry should be sent to storage destinations, including in Cloud Logging buckets, or excluded entirely from ingestion by Cloud Logging. You can use sinks to route logs to multiple destinations.

To reliably route logs, the Log Router also stores the logs temporarily, which buffers against temporary disruptions on any sink. Note that the Log Router's temporary storage is distinct from the longer term storage provided by Logging buckets.

Log sinks

Sinks route log entries to storage destinations and are also used to exclude log entries from being written to Cloud Logging. Log sinks combine a filter expression and a destination.

Cloud Logging provides two predefined log sinks for each Google Cloud project: _Required and _Default. All logs that are generated in a Google Cloud project are automatically processed through these two log sinks and then are stored in the correspondingly named _Required and _Default log buckets.

Log sinks act independently of each other. Regardless of how the predefined log sinks process your log entries, you can create your own log sinks to route some or all of your logs to various supported destinations or to exclude them entirely from being stored by Cloud Logging.

Depending on the log sink's configuration, every log entry received by Cloud Logging falls into one or more of these categories:

  • Stored in Cloud Logging and not routed elsewhere
  • Stored in Cloud Logging and routed to a supported destination
  • Not stored in Cloud Logging but routed to a supported destination
  • Neither stored in Cloud Logging nor routed elsewhere
    • These logs are excluded entirely

You can create sinks at the Google Cloud project level. To set up sinks at the organization or folder levels, use aggregated sinks.

For more information, see Overview of log exports.

Exclusion filters

Each sink lets you enter one or more exclusion filters, which lets you exclude matching log entries from being routed to the sink's destination.

For more information on using exclusion filters, see Logs exclusions.

Supported destinations

You can use the Log Router to route certain logs to supported destinations in any Cloud project. Logging supports the following sink destinations:

  • Cloud Storage: JSON-formatted files stored in Cloud Storage buckets.
  • BigQuery: Tables created in BigQuery datasets.
  • Pub/Sub: JSON-formatted messages delivered to Pub/Sub topics. Supports third-party integrations, such as Splunk, with Logging.
  • Cloud Logging: Log entries held in log buckets.

For more information on routing logs to supported destinations, see Configuring sinks.

Log buckets

Cloud Logging uses log buckets as containers in your Google Cloud projects to store and organize your logs data. The logs that you store in Cloud Logging are indexed, optimized, and delivered to let you analyze your logs in real time. These are different storage entities than the similarly named Cloud Storage buckets.

You can create sinks to route all, or just a subset, of your logs to any log bucket. This flexibility allows you to choose the Cloud project in which your logs are stored and what other logs are stored with them.

Cloud Logging provides every Cloud project with the _Required and _Default log buckets. In the default configuration, all logs generated in a Cloud project are stored in the _Required and _Default log buckets, which live in the Cloud project that the logs are generated in. You can also create custom log buckets.

For more information, see Managing log buckets.

Log views

Log views let you control who has access to the logs within your log buckets.

Cloud Logging automatically creates the _AllLogs view for every bucket, which shows all logs. Cloud Logging also creates a view for the _Default bucket called _Default, which shows all logs except Data Access audit logs.

Because log buckets can contain logs from multiple Cloud projects, you might want to control which Cloud projects different users can view logs from. You can create custom log views, which give you more granular access control for those buckets.

For more information, see Managing log views.

Log-based metrics

Logs-based metrics are Cloud Monitoring metrics that are based on the content of log entries. If Cloud Logging receives a log entry for a Cloud project that matches the filter of one of the Cloud project's metrics, then that log entry is reflected in the metric data.

Sink exclusion filters aren't applied to log-based metrics. Even if you exclude logs from being ingested by Cloud Logging API and the logs aren't stored in any log buckets, you could see those logs counted in the logs-based metrics.

For more information, see Overview of logs-based metrics.

What's next