Logs exports

This page provides a conceptual overview of logs exports using Stackdriver Logging. For instructions on how to export your logs, go to Next steps on this page.

You can export copies of some or all of your logs outside of Stackdriver Logging. You might want to export logs for the following reasons:

  • To store logs for extended periods. Logging typically holds logs for weeks, not years. For more information, go to Logs retention periods.
  • To use big-data analysis tools on your logs.
  • To stream your logs to other applications, other repositories, or third parties.

How exports work

The following diagram illustrates how exported log entries are treated in Stackdriver Logging:

The life of a log entry.

Exporting involves writing a query that selects the log entries you want to export, and choosing a destination in Cloud Storage, BigQuery, or Cloud Pub/Sub. The query and destination are held in an object called a sink. Sinks can be created in GCP projects, organizations, folders, and billing accounts.

There are no costs or limitations in Logging for exporting logs, but the export destinations charge for storing or transmitting the log data.

Sink properties and terminology

Sinks have the following properties:

  • Sink identifier: A name for the sink. For example, my-vm-error-sink.

  • Parent resource: The resource in which you create the sink. The parent is most often a project, but it can be any of the following:

    "projects/[PROJECT_ID]"
    "folders/[FOLDER_ID]"
    "billingAccounts/[BILLINGACCOUNT_ID]"
    "organizations/[ORGANIZATION_ID]"
    

    The sink can only export logs that belong to its parent resource. For the one exception to this rule, see the following Aggregated exports property.

    The full resource name of a sink includes its parent resource and sink identifier. For example:

    "projects/[PROJECT_ID]/sinks/[SINK_ID]"
    
  • Logs filter: Selects which log entries to export through this sink. For query examples, go to the Query library.

  • Destination: A single place to send the log entries matching your query. There are 3 supported destinations:

    • Cloud Storage buckets provide inexpensive, long-term storage:

      storage.googleapis.com/[BUCKET_ID]
      
    • BigQuery datasets provide big data analysis capabilities:

      bigquery.googleapis.com/projects/[PROJECT_ID]/datasets/[DATASET_ID]
      
    • Cloud Pub/Sub topics stream your log entries to other applications or repositories.

      pubsub.googleapis.com/projects/[PROJECT_ID]/topics/[TOPIC_ID]
      

    In the Logs Viewer, you can also use the Custom destination option when creating an export to send your logs from one project to a destination in another project. For more information, go to creating sinks.

    You can export logs to destinations in any project, so long as the export destination authorizes the sink's service account as a writer.

  • Writer identity: A service account name. The export destination's owner must give this service account permission to write to the export destination. When exporting logs, Logging adopts this identity for authorization. For increased security, new sinks get a unique service account:

    [GENERATED_ID_1]@[GENERATED_ID_2].iam.gserviceaccount.com
    

    For more information, go to destination permissions.

  • Aggregated exports. The includeChildren property is described in Aggregated exports. It is only relevant to sinks created for organizations or folders.

For more details about sinks, review the LogSink type, the projects.sinks.create API method, and Exporting logs in the API.

How sinks work

Every time a log entry arrives in a project, folder, billing account, or organization resource, Logging compares the log entry to the sinks in that resource. Each sink whose query matches the log entry writes a copy of the log entry to the sink's export destination.

Since exporting happens for new log entries only, you cannot export log entries that Logging received before your sink was created.

Access control

To create or modify a sink, you must have the Cloud Identity and Access Management roles Owner or Logging/Logs Configuration Writer in the sink's parent resource. To view existing sinks, you must have the Cloud IAM roles Viewer or Logging/Logs Viewer in the sink's parent resource. For more information, go to Access control.

To export logs to a destination, the sink's writer service account must be permitted to write to the destination. For more information about writer identities, read Sink properties on this page.

To secure exported logs from unauthorized access, you must use the access control features of your export destination. Sinks can export any log entries, including private Data Access audit logs.

Pricing

Exported logs don't incur Stackdriver Logging charges, but destination charges might apply. For details, review the appropriate product's pricing page:

Note also that if you send and then exclude your Virtual Private Cloud flow logs from Stackdriver Logging, VPC flow log generation charges apply in addition to the destination charges.

Next steps

Export your logs

To learn how to export your logs, review the following pages:

Find and use your exported logs

To learn about the format of exported log entries and how the exported logs are organized in destinations, go to Using exported logs.

Bu sayfayı yararlı buldunuz mu? Lütfen görüşünüzü bildirin:

Şunun hakkında geri bildirim gönderin...

Stackdriver Logging
Yardım mı gerekiyor? Destek sayfamızı ziyaret edin.