Shape the future of software operations and make your voice heard by taking the 2021 State of DevOps survey.

Basic concepts

Cloud Logging is part of the Google Cloud's operations suite of products. It includes storage for logs, a user interface called the Logs Explorer, and an API to manage logs programmatically. Logging lets you read and write log entries, query your logs, and control how you route and use your logs.

Google Cloud projects and other resources

Logs are associated primarily with Google Cloud projects, although other Google Cloud resources, such as organizations, folders, and billing accounts, can also have logs.

Log entries

A log entry records status or describes specific events or transactions that take place in computer systems. Log entries are written by your own code, Google Cloud services the code is running on, third-party applications, and the infrastructure that Google Cloud depends on.

Some log entries describe specific events that take place within the system. You can use these log entries to output messages that assure users that things are working well or to provide information when things fail.

Other log entries might describe the details of transactions processed by a system or component. For example, a load balancer logs every request that it receives. A load balancer also records information like the requested URL and the HTTP response code, and it might record which backend served the request.

Because logs in modern systems descend from text files written to disk, a log entry is analogous to a line in a log file and can be considered the basic unit of logging.

Your Google Cloud project receives log entries when you begin to use the services that produce log entries, like Compute Engine or BigQuery. You also get log entries when you connect Cloud Monitoring to AWS, when you install the Logging agent on your VM instances, and when you call the entries.write method in the Logging API.

A log entry minimally consists of the following:

  • A timestamp that indicates either when the event took place or when it was received by Cloud Logging.
  • The monitored resource that produced the log entry.
  • A payload, also known as a message, either provided as unstructured textual data or as structured textual data in JSON format.
  • The name of the log to which it belongs.

Log entries can also carry associated metadata. Such metadata might include the severity for each associated log entry.

For more information on log entry data format, see the LogEntry type.

Logs

A log is a named collection of log entries within a Google Cloud resource, such as a project. Logs exist only if they contain log entries.

A log's name is identified by the full path of the resource to which the log entries belong, followed by a simple log ID, like syslog, or a structured ID that includes the log's writer, like compute.googleapis.com/activity.

Each log entry includes the name of its log.

Log types

To learn about the types of logs available in Cloud Logging, see Available logs.

Retention period

Log entries are held in Cloud Logging for a limited time known as the retention period. After that, the entries are deleted. The retention periods for different types of logs are listed in Logging Quotas and limits.

You can configure the retention periods of some of your logs. For details, see Storing logs: Custom retention.

If you also want to back up your logs, see the section Exporting logs using sinks on this page.

Query and filter

A query is an expression in the Logging query language that returns the log entries that match the expression.

A filter is an expression in the Logging query language used in sinks to route logs that match the expression to a storage destination. You also use filters when creating logs-based metrics to route logs that match that expression to Cloud Monitoring.

Monitored resources

Each log entry indicates where it came from by including the name of a monitored resource. Examples include individual Compute Engine VM instances, Google Kubernetes Engine containers, database instances, and so on.

For a complete listing of monitored resource types, see Monitored resources and services.

Queries

A query is an expression in the Logging query language. It is used in the Logs Explorer and the Logging API to select log entries, such as those from a particular VM instance or those arriving in a particular time period with a particular severity level.

Log Router

All logs, including audit logs, platform logs, and user-written logs, are sent to the Cloud Logging API where they pass through the Log Router. The Log Router checks each log entry against existing rules to determine which log entries to ingest (store), which log entries to include in exports, and which log entries to discard.

For more details, see Log Router overview.

Exporting logs using sinks

Log entries received by Logging can be routed to Cloud Storage buckets, BigQuery datasets, and Pub/Sub topics.

You export logs by configuring log sinks, which then continue to export log entries as they arrive in Logging. A sink includes a destination and a filter that selects the log entries to export.

For details, see Overview of logs exports.

Logs-based metrics

Metrics are a feature of Cloud Monitoring. A logs-based metric is a metric whose value is the number of log entries that match a query that you specify.

For details, see Overview of logs-based metrics.

Access control

The ability to access logs is controlled by granting Identity and Access Management roles or permissions to members.

Most logs can be read by any member with the IAM Viewer role. To read Data Access audit logs or Access Transparency logs, the member requires either the IAM Owner role or a custom role with special permissions.

For more information on required permissions, see Access control.