Logs exclusions

This page describes how to exclude logs from ingestion using the Cloud Console and the Logging API.

Cloud Logging lets you exclude logs from your Logs Buckets by adding an exclusion filter on the logs sinks that route logs to your Logs Buckets. Using the Logging query language, you write a filter that defines which logs to exclude from the Logs Bucket.

The Logs Storage page tracks the volume of logs in your project. The Logs Router gives you tools to disable all logs ingestion or exclude (discard) log entries you're not interested in, so that you can minimize any charges for logs over your monthly allotment. For more information about how excluded log entries are treated, go to How exclusions work on this page.

For details on Cloud Logging costs, see Pricing. Note that if you send and then exclude your Virtual Private Cloud flow logs from Cloud Logging, VPC flow log generation charges apply.

Tracking logs usage

To track your project's logs volume, go to the Logs Storage page in the Cloud Logging console:

Go to Logs Storage

The top of the page displays a summary of statistics for the logs that your project is receiving:

Logs storage usage summary statistics

The following statistics are reported:

  • This month's ingested log volume: The amount of logs your project has received since the first date of the current month.

  • Last month's ingested log volume: The amount of logs your project received in the last calendar month.

  • Projected ingestion log volume: The estimated amount of logs your project will receive by the end of the current month, based on current usage.

The log volumes don't include certain audit logs: all Admin Activity audit logs and all System Event audit logs. Those logs are free and cannot be excluded or disabled.

How exclusions work

The following diagram illustrates how excluded log entries are treated in Cloud Logging:

Figure illustrating how Cloud Logging routes logs entries.

The following conditions apply to excluded log entries in Logging:

  • Excluded log entries don't count against the Logging allotment provided to projects. See Logging pricing for details.

  • Excluded log entries aren't visible in the Logs Explorer, and they aren't available to Error Reporting or Cloud Debugger.

  • You can export log entries outside of Cloud Logging using log sinks that include sink destinations. These same logs can also be excluded from ingestion. For more information, read Logs exports.

  • Audit logs that cannot be disabled cannot be excluded either; however those types of audit logs are free.

By creating exclusion filters you can control which log entries you exclude (discard). For example, you could exclude log entries from a single VM instance rather than from all VM instances.

Exclusion limits

You can have up to 50 exclusion filters in a project.

Exclusion timing

Logs are excluded after they are received by the Logging API. Therefore, excluding logs does not reduce the number of entries.write API calls.

Creating exclusion filters

You can create an exclusion filter for a new logs sink or for an existing sink.

Creating exclusion filters for a new sink

To create an exclusion filter for a new logs sink using the Logs Router, do the following:

  1. Follow the Create sink procedures.

  2. In the Choose logs to filter out of sink (optional) step, you can create the exclusion filter to exclude the logs.

  3. Click Add exclusion.

  4. Enter a name in the Exclusion filter name field.

  5. Enter the Exclusion filter rate.

    Provide an integer between 0 and 100.The incoming logs matching the exclusion filter are sampled according to that value.

    A value of 0 samples 0 percent of the logs matching the filter; therefore, 0 is equivalent to disabling the exclusion filter. A value of 100 samples 100 percent of all logs; therefore, all logs matching the exclusion filter are excluded from the destination. A value of 50 samples 50 percent of the logs matching the exclusion filter; therefore, 50 percent of the logs that match the exclusion are excluded, while the other 50 percent are routed to the destination.

    You can create up to 50 exclusion filters per sink.

  6. In the Build an exclusion filter section, enter a filter expression that matches the log entries you want to exclude. For an overview of the Logging query language, see Logging query language.

  7. Click Add exclusion to add additional filters as needed.

  8. When finished, click Create sink.

Creating exclusion filters for an existing sink

To create an exclusion filter for an existing logs sink using the Logs Router, do the following:

  1. From the Logging menu, select Logs Router.

    Go to Logs Router

  2. For the sink you want to add the exclusion filter, click More .

  3. Click Edit sink.

  4. In the Choose logs to filter out of sink (optional) section, create the exclusion filter to exclude the logs.

  5. Click Add exclusion.

  6. Enter a name in the Exclusion filter name field.

  7. Enter the Exclusion filter rate.

    Provide an integer between 0 and 100.The incoming logs matching the exclusion filter are sampled according to that value.

    A value of 0 samples 0 percent of the logs matching the filter; therefore, 0 is equivalent to disabling the exclusion filter. A value of 100 samples 100 percent of all logs; therefore, all logs matching the exclusion filter are excluded from the destination. A value of 50 samples 50 percent of the logs matching the exclusion filter; therefore, 50 percent of the logs that match the exclusion are excluded, while the other 50 percent are routed to the destination.

    You can create up to 50 exclusion filters per sink.

  8. In the Build an exclusion filter section, enter a filter expression that matches the log entries you want to exclude. For an overview of the Logging query language, see Logging query language.

  9. Click Add exclusion to add additional filters as needed.

  10. When finished, click Update sink.

When creating your filter, you may encounter one of the following situations:

  • If you edited the filter for the _Default sink, you might want to restore the default filter. To do so, enter the following in the Build inclusion filter field:

    NOT LOG_ID("cloudaudit.googleapis.com/activity") AND NOT \
    LOG_ID("externalaudit.googleapis.com/activity") AND NOT \
    LOG_ID("cloudaudit.googleapis.com/system_event") AND NOT \
    LOG_ID("externalaudit.googleapis.com/system_event") AND NOT \
    LOG_ID("cloudaudit.googleapis.com/access_transparency") AND NOT \
    LOG_ID("externalaudit.googleapis.com/access_transparency")
    
  • You might want to route all logs to a destination and not exclude any logs. To do so, leave the Build inclusion filter field empty.

  • You might want to exclude all logs from reaching a destination. To do so, disable the sink for that destination.

Viewing exclusion filters

To view your current exclusion filters, do the following:

  1. From the Logging menu, select Logs Router.

    Go to Logs Router

  2. For the sink you want to view the exclusion filters, click More .

  3. Select View sink details.

  4. A panel displays the sink's details, including the exclusion filters.

    Image of the Sink details panel

    In this example, the details of the _Default bucket are shown, and logging has been disabled, indicated by the google-ui-logs-ingestion-off filter.

Editing exclusions

You can edit your existing exclusion filters to exclude more or fewer log entries.

  1. From the Logging menu, select Logs Router.

    Go to Logs Router

  2. For the sink you want to view the exclusion filters, click More .

  3. Click Edit sink.

  4. In the Build an exclusion filter section, edit the filter expression to match the log entries you want to exclude.

  5. Click Add exclusion to add additional filters as needed.

  6. When finished, click Update sink.

Removing exclusions

To edit, disable, or delete an exclusion filter, follow the Editing exclusions guide to delete the exclusion for a particular sink.

Exclusions in the API

To create exclusion filters in the Logging API, use the projects.exclusions.create method.

There are also methods to view, delete, and update exclusion filters.

There are also exclusion methods in the API for logs received by organizations, billing accounts, and folders. Those exclusions can only be created in the Logging API; they aren't supported in the Cloud Logging console.

For examples of logs queries that might be useful in exclusions, go to Sample queries.

Sampled exclusions in the API

To exclude less than 100 percent of the matched log entries, use the sample function in your logs query.

Exporting excluded logs

You can export log entries to Cloud Storage, BigQuery, or Pub/Sub before you exclude them, so that you don't permanently lose the log entries you exclude.

To start your exclusion and export, do the following:

  1. Create a query that matches the log entries you want to exclude and export.

    Tip: Write the query so that it doesn't match any audit logs that are enabled by default. Matching these audit log entries doesn't affect exclusions, but it does result in exporting more log entries.

  2. Create an export sink using your logs query, and start exporting the matching log entries.

  3. Create an exclusion filter using your logs query and start excluding the matching log entries.

To stop your exclusions and export, disable the exclusion filter before you stop the export sink.

For more details about how to export logs, see Exporting Logs.

Exports pricing

Exported logs don't incur Cloud Logging charges, but destination charges might apply. For details, review the appropriate product's pricing page:

Note also that if you send and then exclude your Virtual Private Cloud flow logs from Cloud Logging, VPC flow log generation charges apply in addition to the destination charges.