This series of articles explores best practices for common logging export scenarios.
Stackdriver Logging provides an operational datastore for logs and provides rich export capabilities. You might export your logs for several reasons, such as retaining logs for long-term storage (months or years) to meet compliance requirements or for running data analytics against the metrics extracted from the logs. Stackdriver Logging can export to Cloud Storage, BigQuery, and Cloud Pub/Sub.
Stackdriver Logging can export all logging for an organization, using aggregated exports, or for a specific project, using logs export. Using logging filters, you can include or exclude specific projects or cloud resources. For example, you could export all Compute Engine logs, but exclude high-volume logs from Cloud Load Balancer. This approach gives you the flexibility to export all logs or specific logs.
Using aggregated exports, your organization can export logs from all projects or from a single folder. With this functionality, you can enforce logging export policy across all your organization's projects. You can use organization-level IAM controls in order to limit user access to just modifying the logging export configuration.
As an alternative to aggregated exports, logs export is enabled per project rather than for the entire organization. Logs export is otherwise identical to aggregated exports.
Ways to export
There are three ways to export logs from Stackdriver:
- To files: JSON files stored in Cloud Storage.
- To BigQuery: logging tables created in a BigQuery dataset.
- To Cloud Pub/Sub: JSON messages delivered to a Cloud Pub/Sub topic.
What gets exported
Stackdriver Logging export includes two main types of logs:
- Stackdriver monitored-services logs
These logs include the logs written from services in your cloud infrastructure and logs for managed services (Compute Engine, Cloud SQL, Cloud Datastore, and so on) and other services covered in the monitored services list. Other included logs are those delivered by the Stackdriver Logging Agent such as
mongodb, and all others covered in the agent logs list. You can configure Logging Agent to report additional logs by setting the underlying Fluentd configuration.
- Cloud Audit Logging
Cloud Audit Logging maintains two types of audit logs for each project and organization: Admin Activity and Data Access. Google Cloud Platform (GCP) services write audit log entries to these logs to help you answer the questions "who did what, where, and when?" within your GCP projects. Admin Activity logs contain log entries for API calls or other administrative actions that modify the configuration or metadata of resources. Data Access audit logs record API calls that create, modify, or read user-provided data. See the list of services that produce audit logs.
Depending on the type of log, there are three distinct logging payload formats.
The contents are represented as a single string. The logs reported by the Stackdriver Logging agent (including
syslog) and the Cloud SQL logs are both examples of logs that use this format.
The contents are represented as a protocol buffer and vary depending on the specific content being logged. The Admin Activity and Data Access audit logs are both exported in this format. These logs have different JSON and table structures in BigQuery based on the exported entry type.
The contents are represented as a JSON object and vary depending on the specific content being logged. The activity logs from Compute Engine and the Compute Engine autoscaler are examples that use this format.
The schemas and fields documentation provides detailed information about mapping the log formats to BigQuery table and JSON export file structures. Consider the specific logging payload format when you write queries against BigQuery export or when you parse the file or Cloud Pub/Sub export JSON files. The detailed format of the log is listed in the API definition for LogEntry.
Logging export scenarios
Articles in this series describe scenarios for which you might want to export logs. Each scenario details the requirements, setup, and usage, and shows how to share the exports.
- Scenario 1 – Export for Compliance Requirements
- Scenario 2 – Export for Security and Access Analytics
- Scenario 3 – Export to Splunk