Logs exclusions

This page describes how to exclude logs from ingestion using the Cloud Console and the Logging API.

The Resource usage page tracks the volume of logs in your project. The Logs Router gives you tools to disable all logs ingestion or exclude (discard) log entries you're not interested in, so that you can minimize any charges for logs over your monthly allotment. For more information about how excluded log entries are treated, go to How exclusions work on this page.

For details on Cloud Logging costs, see Pricing. Note that if you send and then exclude your Virtual Private Cloud flow logs from Cloud Logging, VPC flow log generation charges apply.

Tracking logs usage

To track your project's logs volume, go to the Resource usage page in the Cloud Logging console:

Go to Resource usage

The top of the page displays a summary of statistics for the logs that your project is receiving:

Resource usage summary statistics

Four statistics are reported:

  • Last month's ingested log volume: The amount of logs your project received in the last calendar month.

  • This month's ingested log volume: The amount of logs your project has received since the first date of the current month.

  • Excluded log volume: The amount of logs that you have excluded from your project since the first date of the current month. This number is not included in This month's ingested log volume. Excluding logs is described later on this page.

  • Projected ingestion log volume: The estimated amount of logs your project will receive by the end of the current month, based on current usage.

The log volumes don't include certain audit logs: all Admin Activity audit logs and all System Event audit logs. Those logs are free and cannot be excluded or disabled.

The breakdown in logs usage by resource type is also displayed. For details, go to Viewing resource-type exclusions on this page.

How exclusions work

The following diagram illustrates how excluded log entries are treated in Cloud Logging:

Figure illustrating how Cloud Logging routes logs entries.

The following conditions apply to excluded log entries in Logging:

  • Excluded log entries don't count against the Logging allotment provided to projects. See Logging pricing for details.

  • Excluded log entries aren't visible in the Logs Explorer, and they aren't available to Error Reporting or Cloud Debugger.

  • You can export log entries outside of Cloud Logging using log sinks that include sink destinations. These same logs can also be excluded from ingestion. For more information, read Logs exports.

  • Audit logs that cannot be disabled cannot be excluded either; however those types of audit logs are free.

There are two kinds of exclusions:

  • Exclusion filters give you the flexibility to select log entries for exclusion based on filter expressions. You can use exclusion filters to choose a random sample of log entries to exclude.

  • Resource-type exclusions let you block all logs from specific resource types.

Logging considers both kinds of exclusions when deciding whether or not to exclude a log entry. If any resource-type exclusion or any exclusion filter matches the log entry, then that log entry is excluded.

By creating exclusion filters you can control which log entries you exclude (discard). For example, you could exclude log entries from a single VM instance rather than from all VM instances.

If you use both exclusion filters and resource-type exclusions, they might overlap. A log entry is excluded if it is from a disabled resource type, or if it matches one of the exclusion filters discussed in this section. Note that this is a technical distinction, since as mentioned previously, Logging implements resource-type exclusions with exclusion filters.

The per-resource-type table on the Ingestions tab of the Resource usage page reflects both resource-type exclusions and exclusion filters. Even if you don't use resource-type exclusions, that table lets you track the effect of your exclusion filters.

Exclusion limits

You can have up to 50 exclusion filters in a project. This limit includes exclusion filters and resource-type exclusions created in the Cloud Logging console or in the API.

Exclusion timing

Logs are excluded after they are received by the Logging API. Therefore, excluding logs does not reduce the number of entries.write API calls.

Creating exclusion filters

You can create an exclusion filter for a new logs sink or for an existing sink.

Creating exclusion filters for a new sink

To create an exclusion filter for a new logs sink using the Logs Router, do the following:

  1. Follow the Create sink procedures.

  2. In the Choose logs to filter out of sink (optional) step, you can create the exclusion filter to exclude the logs.

  3. Click Add exclusion.

  4. Enter a name in the Exclusion filter name field.

  5. Enter the Exclusion filter rate.

  6. In the Build an exclusion filter section, enter a filter expression that matches the log entries you want to exclude.

  7. Click Add exclusion to add additional filters as needed.

  8. When finished, click Create sink.

Creating exclusion filters for an existing sink

To create an exclusion filter for an existing logs sink using the Logs Router, do the following:

  1. From the Logging menu, select Logs Router.

    Go to Logs Router

  2. For the sink you want to add the exclusion filter, click More .

  3. Click Edit sink.

  4. In the Choose logs to filter out of sink (optional) section, create the exclusion filter to exclude the logs.

  5. Click Add exclusion.

  6. Enter a name in the Exclusion filter name field.

  7. Enter the Exclusion filter rate.

  8. In the Build an exclusion filter section, enter a filter expression that matches the log entries you want to exclude.

  9. Click Add exclusion to add additional filters as needed.

  10. When finished, click Update sink.

When creating your filter, you may encounter one of the following situations:

  • If you edited the filter for the _Default sink, you might want to restore the default filter. To do so, enter the following in the Build inclusion filter field:

    NOT LOG_ID("cloudaudit.googleapis.com/activity") AND NOT \
    LOG_ID("externalaudit.googleapis.com/activity") AND NOT \
    LOG_ID("cloudaudit.googleapis.com/system_event") AND NOT \
    LOG_ID("externalaudit.googleapis.com/system_event") AND NOT \
    LOG_ID("cloudaudit.googleapis.com/access_transparency") AND NOT \
    LOG_ID("externalaudit.googleapis.com/access_transparency")
    
  • You might want to route all logs to a destination and not exclude any logs. To do so, leave the Build inclusion filter field empty.

  • You might want to exclude all logs from reaching a destination. To do so, disable the sink for that destination.

Viewing exclusion filters

To view your current exclusion filters, do the following:

  1. From the Logging menu, select Logs Router.

    Go to Logs Router

  2. For the sink you want to view the exclusion filters, click More .

  3. Select View sink details.

  4. A panel displays the sink's details, including the exclusion filters.

    Image of the Sink details panel

    In this example, the details of the _Default bucket are shown, and logging has been disabled, indicated by the google-ui-logs-ingestion-off filter.

Editing exclusions

You can edit your existing exclusion filters to exclude more or fewer log entries.

  1. From the Logging menu, select Logs Router.

    Go to Logs Router

  2. For the sink you want to view the exclusion filters, click More .

  3. Click Edit sink.

  4. In the Build an exclusion filter section, edit the filter expression to match the log entries you want to exclude.

  5. Click Add exclusion to add additional filters as needed.

  6. When finished, click Update sink.

Best practice: Don't edit or delete exclusion filters created by Logging as part of the resource-type exclusions. Manage those filters with the Disable log source and Enable log source options on the Ingestions tab.

Removing exclusions

To edit, disable, or delete an exclusion filter, follow the Editing exclusions guide to delete the exclusion for a particular sink.

Using resource-type exclusions

By default, your project receives all logs from all resource types. To discard all the logs from specific resource types, use resource type exclusions.

Resource-type exclusions are a feature of the Cloud Logging console. When you create a resource-type exclusion, Logging creates an exclusion filter that implements the exclusion. For more information, go to How exclusions work.

Creating resource-type exclusions

To exclude (discard) all the logs from a specific resource type, complete the Creating exclusion filters steps to create an exclusion filter for a new or existing logs sink.

Viewing resource-type exclusions

To view your logs usage by resource type and your resource-type exclusions, do the following:

  1. Go to the Resource usage page in the Cloud Logging console:

    Go to Resource usage

  2. Select the Ingestions tab (the default) under the summary statistics. The Logs Ingestion table displays your logs usage by resource type:

    Resource Usage Table

The table shows logs usage information for each resource type that has sent logs to your project this current and past month. There could be resource types that only sent logs last month but not this month, which are also listed in this table.

The Ingestion Status column is an approximate indication of whether there are exclusions related to each resource type. The status can be any of the following:

  • Not ingested: There are one or more exclusions that exactly target this resource type at a 100% sample rate. That means the exclusion's filter consists of exactly resource.type=[THIS_RESOURCE_TYPE].

  • All ingested: There have been no log entries from this resource type excluded so far this month, and there are no exclusions that exactly target this resource type.

  • Partially ingested: There are one or more exclusions that target this resource type with a sampling rate between 0 percent and 100 percent. If this resource type has had any log entries excluded this month, then this status remains until the end of the month, even if all exclusions are currently removed. For more information, go to Editing exclusions on this page.

Alternatively, you can inspect resource-type exclusions on the Exclusions tab. Logging implements resource-type exclusions by creating exclusion filters. For details, go to Viewing exclusion filters.

Exclusions in the API

To create exclusion filters in the Logging API, use the projects.exclusions.create method.

There are also methods to view, delete, and update exclusion filters.

There are also exclusion methods in the API for logs received by organizations, billing accounts, and folders. Those exclusions can only be created in the Logging API; they aren't supported in the Cloud Logging console.

For examples of logs queries that might be useful in exclusions, go to Sample queries.

Resource-type exclusions in the API

Resource-type exclusions aren't a separate kind of exclusion in the API. To create an exclusion that discards all log entries from a particular resource type, create an exclusion filter with a logs query that specifies the resource type:

resource.type = [THE_RESOURCE_TYPE]

Sampled exclusions in the API

To exclude less than 100 percent of the matched log entries, use the sample function in your logs query.

Exporting excluded logs

You can export log entries to Cloud Storage, BigQuery, or Pub/Sub before you exclude them, so that you don't permanently lose the log entries you exclude.

To start your exclusion and export, do the following:

  1. Create an advanced logs query that matches the log entries you want to exclude and export.

    Tip: Write the filter so that it doesn't match any audit logs that are enabled by default. Matching these audit log entries doesn't affect exclusions, but it does result in exporting more log entries.

  2. Create an export sink using your logs query, and start exporting the matching log entries.

  3. Create an exclusion filter using your logs query and start excluding the matching log entries.

To stop your exclusions and export, disable the exclusion filter before you stop the export sink.

For more details about how to export logs, see Exporting Logs.

Exports pricing

Exported logs don't incur Cloud Logging charges, but destination charges might apply. For details, review the appropriate product's pricing page:

Note also that if you send and then exclude your Virtual Private Cloud flow logs from Cloud Logging, VPC flow log generation charges apply in addition to the destination charges.