Overview of Logs Export

You can export copies of some or all of your logs outside of Stackdriver Logging.

Exporting involves writing a filter that selects the log entries you want to export, and choosing a destination in Cloud Storage, BigQuery, or Cloud Pub/Sub. The filter and destination are held in an object called a sink. Sinks can be created in projects, organizations, folders, and billing accounts.

There are no costs or limitations in Stackdriver Logging for exporting logs, but the export destinations charge for storing or transmitting the log data.

Why export logs?

You might want to export logs for the following reasons:

  • To store logs for extended periods. Stackdriver Logging typically holds logs for weeks, not years. For more information, see the Quota Policy.
  • To use big data analysis tools on your logs.
  • To stream your logs to other applications, other repositories, or third parties.

Sink properties and terminology

Sinks have the following properties:

  • Sink identifier: A name for the sink. For example, "my-vm-error-sink".

  • Parent resource: The resource in which you create the sink. The parent is most often a project, but it can be any of the following:

    "projects/[PROJECT_ID]"
    "folders/[FOLDER_ID]"
    "billingAccounts/[BILLINGACCOUNT_ID]"
    "organizations/[ORGANIZATION_ID]"
    

    The sink can only export logs that belong to its parent resource. For the one exception to this rule, see the following Aggregated Exports property.

    The full resource name of a sink includes its parent resource and sink identifier. For example:

    "projects/[PROJECT_ID]/sinks/[SINK_ID]"
    
  • Logs filter: Selects which log entries to export through this sink. For more details, see Advanced Logs Filters. For example, the following filter selects all log entries with severity ERROR or higher from a specified Compute Engine VM instance:

    resource.type = gce_instance AND
       resource.labels.instance_id = "[INSTANCE_ID]" AND
       severity >= ERROR
    
  • Destination: A single place to send the log entries matching your filter. There are three supported destinations:

    • Cloud Storage buckets provide inexpensive, long-term storage:

      storage.googleapis.com/[BUCKET_ID]
      
    • BigQuery datasets provide big data analysis capabilities:

      bigquery.googleapis.com/projects/[PROJECT_ID]/datasets/[DATASET_ID]
      
    • Cloud Pub/Sub topics stream your log entries to other applications or repositories.

      pubsub.googleapis.com/projects/[PROJECT_ID]/topics/[TOPIC_ID]
      

    You can export logs to destinations in any project, so long as the destination authorizes the sink's service account as a writer.

  • Writer identity: A service account name. The destination's owner must give this service account permission to write to the destination. When exporting logs, Stackdriver Logging adopts this identity for authorization. For increased security, new sinks get a unique service account:

    [GENERATED_ID_1]@[GENERATED_ID_2].iam.gserviceaccount.com
    

    For more information, see Destination permissions.

  • Aggregated Exports. The includeChildren property is described in Aggregated Exports. It is only relevant to sinks created for organizations.

For more details about sinks, see the LogSink type, the projects.sinks.create API method, and Exporting logs in the API.

How sinks work

Every time a log entry arrives in a project, folder, billing account, or organization resource, Stackdriver Logging compares the log entry to the sinks in that resource. Each sink whose filter matches the log entry writes a copy of the log entry to the sink's destination.

Since exporting happens for new log entries only, you cannot export log entries that Stackdriver Logging received before your sink was created.

Access control

To create or modify a sink, you must have the IAM roles Owner or Logging/Logs Configuration Writer in the sink's parent resource. To view existing sinks, you must have the IAM roles Viewer or Logging/Logs Viewer in the sink's parent resource. For more information, see Access Control.

To export logs to a destination, the sink's writer service account must be permitted to write to the destination. For more information about writer identities, see the preceding section, Sink properties.

To secure exported logs from unauthorized access, you must use the access control features of your destination. Sinks can export any log entries, including private Data Access audit logs.

Troubleshooting

Here are some common problems with exports and what to do about them:

  • Errors from the destination: Check the specification of the destination in the sink. Use projects.sinks.get to find the writer identity for your sink, and be sure that identity is permitted to write to your destination. Your sink begins exporting logs when the errors are corrected.

  • No logs are being exported: Here are some possible reasons:

    • No matching log entries have been received since you created your sink. Only new log entries are exported.
    • Your filter is incorrect. A misspelled word in the filter might prevent any log entries from matching. Try the same filter in the Logs Viewer's advanced filter interface to see if it matches any log entries.
    • There is a delay before you can see your exported logs in the destination. This is especially true of Cloud Storage destinations. See Exported logs availability.

Next steps

To learn how to export logs, see the following:

To learn about the format of exported log entries and how the exported logs are organized in destinations, see Using Exported Logs.

Send feedback about...

Stackdriver Logging