This page explains how Cloud Logging processes log entries, and describes the key components of Logging routing and storage.
At a high level, this is how Cloud Logging routes and stores log entries:
Cloud Logging receives log entries through the Cloud Logging API where they pass through the Log Router. The sinks in the Log Router check each log entry against existing inclusion filters and exclusion filters that determine if the log entry should be sent to storage destinations, including in Cloud Logging buckets, or excluded entirely from ingestion by Cloud Logging. You can use sinks to route logs to multiple destinations.
To reliably route logs, the Log Router also stores the logs temporarily, which buffers against temporary disruptions on any sink. Note that the Log Router's temporary storage is distinct from the longer term storage provided by Logging buckets.
Sinks control how Cloud Logging routes logs. Using sinks, you can route some or all of your logs to supported destinations or exclude log entries from being stored in Cloud Logging. Some of the reasons that you might want to control how your logs are routed include the following:
- To store logs that are unlikely to be read but that must be retained for compliance purposes.
- To use big-data analysis tools on your logs.
- To stream your logs to other applications, other repositories, or third parties.
Cloud Logging provides two predefined log sinks for each Google Cloud project:
_Default. All logs that are generated in a Google Cloud project
are automatically processed through these two log sinks and then are stored in
the correspondingly named
_Default log buckets.
Log sinks act independently of each other. Regardless of how the predefined log sinks process your log entries, you can create your own log sinks to route some or all of your logs to various supported destinations or to exclude them entirely from being stored by Cloud Logging.
All log entries written to the Cloud Logging API pass through the Log Router. When a log entry arrives in a Cloud project, folder, billing account, or organization resource, Logging compares the log entry to the filters of the sinks associated with the resource.
Depending on the log sink's configuration, every log entry received by Cloud Logging falls into one or more of these categories:
- Stored in Cloud Logging and not routed elsewhere
- Stored in Cloud Logging and routed to a supported destination
- Not stored in Cloud Logging but routed to a supported destination
- Neither stored in Cloud Logging nor routed elsewhere
- These logs are excluded entirely
Sinks route logs that belong to parent resources; you usually create sinks at the Google Cloud project level. You can also create aggregated sinks to combine and route logs from the Cloud projects, folders, and billing accounts contained by a Google Cloud organization or folder.
You can't route log entries that Logging received before your sink was created because routing happens for new log entries only. If you need to route log entries retroactively, see Copy logs.
If you don't set a filter for a sink, every log entry for the Cloud project is routed to the destination unless it is explicitly excluded through the sink's exclusion filters.
When you create a sink, you can set multiple exclusion filters, letting you exclude matching log entries from being routed to the sink's destination or from being ingested by Cloud Logging. You create exclusion filters by using the Logging query language.
Logs are excluded after they are received by the Logging API.
Therefore, excluding logs doesn't reduce the number of
entries.write API calls.
Excluded log entries aren't visible in the Logs Explorer, and they aren't available to Error Reporting or Cloud Debugger.
User-defined logs-based metrics are computed from log entries in both included and excluded logs. For more information, see Monitor your logs.
You can use the Log Router to route certain logs to supported destinations in any Cloud project. Logging supports the following sink destinations:
- Cloud Storage: JSON files stored in Cloud Storage buckets; provides inexpensive, long-term storage.
- BigQuery: Tables created in BigQuery datasets; provides big data analysis capabilities.
- Pub/Sub: JSON-formatted messages delivered to Pub/Sub topics; supports third-party integrations, such as Splunk, with Logging.
- Cloud Logging: Log entries held in log buckets; provides storage in Cloud Logging with customizable retention periods.
For more information on routing logs to supported destinations, see Configure sinks.
Cloud Logging uses log buckets as containers in your Google Cloud projects to store and organize your logs data. The logs that you store in Cloud Logging are indexed, optimized, and delivered to let you analyze your logs in real time. These are different storage entities than the similarly named Cloud Storage buckets.
For each Cloud project, Logging automatically
creates two log buckets:
automatically creates log sinks named
_Default that, in the
default configuration, automatically route
logs to the correspondingly named buckets. Note that you can disable or limit
the logs that are routed to the
_Default log bucket.
Additionally, you can create user-defined buckets for any Google Cloud project.
You create sinks to route all, or just a subset, of your logs to any log bucket. This flexibility allows you to choose the Cloud project in which your logs are stored and what other logs are stored with them.
For more information, see Configure log buckets.
_Required log bucket
Cloud Logging automatically routes the following types of logs to the
Cloud Logging retains the logs in this bucket for 400 days; you can't change this retention period.
You can't modify or delete the
_Required bucket. You can't disable the
_Required sink, which routes logs to the
Neither ingestion pricing nor storage pricing applies to the logs data stored in
_Required log bucket.
_Default log bucket
Any log entry that isn't ingested by the
_Required bucket is routed by the
_Default sink to the
_Default bucket, unless you disable or otherwise edit
_Default sink. For instructions on modifying sinks, see
You can't delete the
Logs held in the
_Default bucket are retained for
30 days, unless you
configure custom retention for the
Cloud Logging pricing applies
to the logs data held in the
User-defined log buckets
You can also create user-defined log buckets in any Cloud project. By applying log sinks to your user-defined log buckets, you can route any subset of your logs to any log bucket, letting you choose which Cloud project your logs are stored in and which other logs are stored with them.
For example, for any log generated in Project-A, you can configure a sink to route that log to user-defined buckets in Project-A or Project-B.
Cloud Logging pricing applies to the logs data held in this bucket, regardless of the log type.
You can configure custom retention for the bucket.
For information on managing your user-defined log buckets, including deleting or updating them, see Configure and manage log buckets.
When you create your log bucket, you can choose to store your logs in any of the following regions:
In addition to these regions, you also have the option to set the location to
global, which means that you don't need to specify where your logs are
You can create an organization policy to ensure that your organization meets your compliance and regulatory needs. Using an organization policy, you can specify in which regions your organization can create new log buckets. You can also restrict your organization from creating new log buckets in specified regions.
Cloud Logging doesn't enforce your newly created organization policy on your existing log bucket; it only enforces the policy on new log buckets.
For information on creating a location-based organization policy, refer to Restrict resource locations.
Cloud Logging retains logs according to retention rules applying to the log bucket type where the logs are held.
You can configure Cloud Logging to retain logs between 1 day and 3650 days. Custom retention rules apply to all the logs in a bucket, regardless of the log type or whether that log has been copied from another location.
For information on setting retention rules for a log bucket, see Configure custom retention.
Stopping logs ingestion
To learn how to stop ingesting logs into your Google Cloud project, see Stop logs ingestion.
Log views let you control who has access to the logs within your log buckets.
Cloud Logging automatically creates the
_AllLogs view for every bucket,
which shows all logs. Cloud Logging also creates a view for the
_Default, which shows all logs except Data Access audit logs.
Because log buckets can contain logs from multiple Cloud projects, you might want to control which Cloud projects different users can view logs from. You can create custom log views, which give you more granular access control for those buckets.
For more information, see Manage log views.
Logs-based metrics are Cloud Monitoring metrics that are based on the content of log entries. If Cloud Logging receives a log entry for a Cloud project that matches the filter of one of the Cloud project's metrics, then that log entry is reflected in the metric data.
Sink exclusion filters aren't applied to log-based metrics. Even if you exclude logs from being ingested by Cloud Logging API and the logs aren't stored in any log buckets, you could see those logs counted in the logs-based metrics.
For more information, see Overview of logs-based metrics.
Be aware of the following limitations:
Custom views on a log bucket are currently in Preview.
You can't create logs-based metrics at the log bucket-level. These metrics are calculated by the Log Router and apply to both ingested and excluded logs only in the Cloud project in which they're received.
Error Reporting is a global service built on Cloud Logging and doesn't analyze logs stored in a regional log buckets or logs routed to other Cloud projects.
For more information, see Using Error Reporting with regionalized logs.
Cloud Logging doesn't charge to route logs, but destination charges might apply. For details, review the appropriate service's pricing details:
Note also that if you send and then exclude your Virtual Private Cloud flow logs from Cloud Logging, VPC flow log generation charges apply in addition to the destination charges.
To help you route and store Cloud Logging data, see the following documents:
To create sinks to route logs to supported destinations, see Configure sinks.
To learn about the format of routed log entries and how the logs are organized in destinations, see Find routed logs.
For routing and sinks troubleshooting information, see Troubleshoot routing and sinks.
To learn how to help meet your organization's compliance needs, go to Enable customer-managed encryption keys for Log Router.
For information on addressing common use cases with log buckets, see the following documents and tutorials:
For best practices on using routing for data governance, read the following papers: