Logs Exclusions

The Logs ingestion page in Stackdriver Logging tracks the volume of logs in your project. The page also gives you tools to disable all logs ingestion or exclude (discard) log entries you're not interested in, so that you can minimize any charges for logs over your monthly allotment. For more information about how excluded log entries are treated, see Overview of exclusions on this page.

For details on Stackdriver Logging costs, see Stackdriver Pricing. Note that if you send and then exclude your Virtual Private Cloud flow logs from Stackdriver Logging, VPC flow log generation charges apply.

Tracking logs usage

To track your project's logs volume, go the Logs ingestion page in the Stackdriver Logging console:

Go to the Logs ingestion page

At the top of the page, you see a summary of statistics for the logs that your project is receiving:

Logs Ingestion Summary

Four numbers are reported:

  • Last month's ingested log volume: The amount of logs your project received in the last calendar month.

  • This month's ingested log volume: The amount of logs your project received since the first date of the current month.

  • Excluded log volume: The amount of logs that you have excluded from your project since the first date of the current month. This number is not included in This month's ingested log volume. Excluding logs is described later on this page.

  • Projected ingestion log volume: The estimated amount of logs your project will receive by the end of the current month, based on current usage.

The log volumes do not include the enabled-by-default audit logs—all Admin Activity logs plus BigQuery Data Access logs. Those logs are free and cannot be excluded.

You can also see the breakdown in logs usage by resource type. See Viewing resource-type exclusions on this page.

Stopping all logs ingestion

To immediately disable all logs ingestion, from the Logs ingestion page, do the following:

  1. Go to the Logs ingestion page of the Stackdriver Logging console for the project whose logs you want to manage.

    Go to the Logs ingestion page

  2. Above the summary statistics, click the Logs Enabled button.

  3. You see a dialog Disable all logs ingestion?. Click Disable Logs.

  4. If you have successfully disabled all logs ingestion, you see the following changes in the Stackdriver Logging console:

    Logs Ingestion Disabled

    • The button that you clicked in step 2 now reads as Logs Disabled.
    • A banner is displayed at the top each page of the Stackdriver Logging console: Logs ingestion is turned off. No logs are being sent to Stackdriver Logging.
    • In the Exclusions tab of the Logs ingestion page, there is a filter named google-ui-logs-ingestion-off. Its Percent to Exclude value is set at 100% and its Exclusion Status is Active.

To immediately re-enable logs ingestion, click the Logs Disabled button above the summary statistics. The button now reads as Logs Enabled and the banner is no longer displayed on your Stackdriver Logging console.

Overview of exclusions

The following diagram illustrates how excluded log entries are treated in Stackdriver Logging:

Life of a log

The following conditions apply to excluded log entries in Logging:

  • Excluded log entries do not count against the Logging allotment provided to projects. See Logging details for details.

  • Excluded log entries are not visible in the Logs Viewer; they are not counted in logs-based metrics; and they are not available to Stackdriver Error Reporting or Stackdriver Debugger.

  • You can export log entries before they are excluded. For more information, see Exporting Logs.

  • You cannot exclude any audit logs that are enabled by default; however, audit logs that are enabled by default are free.

  • You cannot exclude AWS logs or any logs from sources other than GCP.

There are two kinds of exclusions:

  • Exclusion filters give you the flexibility to select log entries for exclusion based on filter expressions. You can use exclusion filters to choose a random sample of log entries to exclude. For details, see Using exclusion filters.

  • Resource-type exclusions let you block all logs from specific resource types. For details, see Using resource-type exclusions.

When deciding whether or not to exclude a log entry, Logging considers both kinds of exclusions. If any resource-type exclusion or any exclusion filter matches the log entry, then that log entry is excluded.

Using exclusion filters

By creating exclusion filters you can control exactly which log entries you exclude (discard). For example, you could exclude log entries from a single VM instance rather than from all VM instances.

If you use both exclusion filters and resource-type exclusions, they might overlap. A log entry is excluded if it is from a disabled resource type, or if it matches one of the exclusion filters discussed in this section. Note that this is a technical distinction, since as mentioned previously, Logging implements resource-type exclusions with exclusion filters.

The per-resource-type table on the Ingestions tab of the Logs ingestion page reflects both resource-type exclusions and exclusion filters. Even if you do not use resource-type exclusions, that table lets you track the effect of your exclusion filters.

Viewing exclusion filters

To view your current exclusion filters—including the filters created by Logging to implement your resource-type exclusions—do the following:

  1. Visit the Logs ingestion page of the Stackdriver Logging console. Select the project whose logs you want to manage.

    Go to the Logs ingestion page

  2. Under the summary statistics, click the Exclusions tab. In the Exclusion Filters table, you see a list of your exclusion filters (if any):

    Exclusions Filter table

If you have stopped all logs ingestion, your list includes an exclusion filter named google-ui-logs-ingestion-off. You can edit, delete or disable this filter using the menu to the right side of the filter.

Creating exclusion filters

To create an exclusion filter using the Stackdriver Logging console, do the following:

  1. Go to the Logs ingestion page of the Stackdriver Logging console and select the Exclusions tab.

    Go to the Logs ingestion page

  2. Click Create Exclusion. You see the Exclusion Editor beside the Logs Viewer panel:

    Create Exclusion

  3. In the Logs Viewer panel, enter a filter expression that matches the log entries you want to exclude. For more information about the panel, see The user interfaces.

  4. In the Exclusion Editor, complete your exclusion filter by filling in the text boxes:

    • Name: A name that identifies the exclusion, such as low-severity.
    • Description: A longer description of the filter, such as Exclude logs whose severity is less than WARNING.
    • Percent to Exclude: Enter an integer or floating-point value from 0 to 100. For example, enter 100 or 100.0 to exclude all matching log entries. Enter 99.50 to exclude 99.5% of the matching entries but keep 0.5% of them in Logging.
  5. Click Create Exclusion to create and start the exclusion.

Tip: If you want to prevent your exclusion filter from being used, then select Disable Exclusion in the menu to the right of your exclusion filter. You can also edit or delete your exclusion filter in the same menu.

Stopping exclusions

You can stop excluding some or all logs in several ways:

  • To stop excluding all logs: On the Logs ingestion page, click the Logs Disabled button at the top of the page. If you have previously disabled all logs ingestion, then the button will turn from **Logs Disabled to Logs Enabled. See Stopping logs ingestion for details.

  • To stop excluding by resource type: On the Ingestions tab of the Logs ingestion page, click Enable log source in the menu to the right side of a resource type you want to stop excluding.

  • To edit, disable, or delete an exclusion filter: On the Exclusions tab of the Logs ingestion page, use the menu to the right of any exclusion filters that target log entries you want to receive. See Editing exclusions for details.

Tip: Check all your exclusion filters, because the same log entries could be targeted by more than one filter.

Editing exclusions

You can edit your existing exclusion filters to exclude more or fewer log entries.

  1. Go to the Logs ingestion page in the Stackdriver Logging console and click the Exclusions tab.

    Go to the Logs ingestion page

  2. Choose an exclusion filter and select Edit Exclusion from the menu at the right side of the filter listing.

  3. Change the advanced logs filter or change the Percent to Exclude value. If you change the filter, check the preview of matching log entries. You cannot change the name of an existing exclusion filter.

  4. Click Update Exclusion.

Best practice: Do not edit or delete exclusion filters created by Logging as part of the resource-type exclusions. Manage those filters with the Disable log source and Enable log source options on the Ingestions tab.

Using resource-type exclusions

By default, your project receives all logs from all resource types. To discard all the logs from specific resource types, use resource type exclusions.

Resource-type exclusions are a feature of the Stackdriver Logging console. When you create a resource-type exclusion, Logging creates an exclusion filter that implements the exclusion. For more information, see Using exclusion filters.

Viewing resource-type exclusions

To view your logs usage by resource type and to see your resource-type exclusions, do the following:

  1. Go to the Logs ingestion page in the Stackdriver Logging console:

    Go to the Logs ingestion page

  2. Select the Ingestions tab (the default) under the summary statistics. In the Logs Ingestion table, you see your logs usage by resource type:

    Resource Usage Table

The table shows logs usage information for each resource type that has sent logs to your project this current and past month. There could be resource types that only sent logs last month but not this month, which are also listed in this table.

The Ingestion Status column is an approximate indication of whether there are exclusions related to each resource type. The status can be any of the following:

  • Not ingested: There are one or more exclusions that exactly target this resource type at a 100% sample rate. That means the exclusion's filter consists of exactly resource.type=[THIS_RESOURCE_TYPE].

  • All ingested: There have been no log entries from this resource type excluded so far this month, and there are no exclusions that exactly target this resource type.

  • Partially ingested: There are one or more exclusions that target this resource type with a sampling rate between 0 percent and 100 percent. If this resource type has had any log entries excluded this month, then this status remains until the end of the month, even if all exclusions are currently removed. See Editing exclusions for more information.

Alternatively, you can inspect resource-type exclusions on the Exclusions tab. Logging implements resource-type exclusions by creating exclusion filters. See Viewing exclusion filters.

Creating resource-type exclusions

To exclude (discard) all the logs from a specific resource type, create a resource-type exclusion. Follow these steps:

  1. Go to the Logs ingestion page in the Stackdriver Logging console:

    Go to the Logs ingestion page

  2. Select the Ingestions tab (the default) under the statistics summary. In the Logs Ingestion table, you see your logs usage by resource type, as shown in the screenshot in the previous section.

  3. Locate the table row for the resource type you want to exclude.

  4. Select Create exclusion filter based on this resource in the menu on the right side of the table row.

  5. In the Exclusion Editor, complete your exclusion filter by filling in the text boxes:

    • Name: A name that identifies the exclusion, such as low-severity.
    • Description: A longer description of the filter, such as Exclude logs whose severity is less than WARNING.
    • Percent to Exclude: Enter an integer or floating-point value from 0 to 100. For example, enter 100 or 100.0 to exclude all matching log entries. Enter 99.50 to exclude 99.5% of the matching entries but keep 0.5% of them in Logging.
  6. Click Create Exclusion to create and start the exclusion.

To stop excluding logs from the resource type, click Enable log source in the menu.

Exclusions in the API

To create exclusion filters in the Stackdriver Logging API, use the projects.exclusions.create method. There are also methods to view, delete, and update exclusion filters.

There are also exclusion methods in the API for logs received by organizations, billing accounts, and folders. Those exclusions can only be created in the Stackdriver Logging API; they are not supported in the Stackdriver Logging console.

For examples of logs filters that might be useful in exclusions, see Advanced Logs Filters.

Resource-type exclusions in the API

Resource-type exclusions are not a separate kind of exclusion in the API. To create an exclusion that discards all log entries from a particular resource type, create an exclusion filter with a logs filter that specifies the resource type:

resource.type = [THE_RESOURCE_TYPE]

Sampled exclusions in the API

To exclude less than 100 percent of the matched log entries, use the sample function in your logs filter.

Exclusion limits

You can have up to 50 exclusion filters in a project. This includes exclusion filters and resource-type exclusions created in the Stackdriver Logging console or in the API.

Exporting excluded logs

You can export log entries to Cloud Storage, BigQuery, or Cloud Pub/Sub before you exclude them, so that you don't permanently lose the log entries you exclude.

Exported logs incur destination charges. Note also that if you send and then exclude your Virtual Private Cloud flow logs from Stackdriver Logging, VPC flow log generation charges apply in addition to the destination charges.

To start your exclusion and export, do the following:

  1. Create an advanced logs filter that matches the log entries you want to exclude and export.

    Tip: Write the filter so that it does not match any audit logs that are enabled by default. Matching these audit log entries doesn't affect exclusions, but it does result in exporting more log entries.

  2. Create an export sink using your logs filter, and start exporting the matching log entries.

  3. Create an exclusion filter using your logs filter and start excluding the matching log entries.

To stop your exclusions and export, disable the exclusion filter before you stop the export sink.

For more details about how to export logs, see Exporting Logs.

¿Te sirvió esta página? Envíanos tu opinión:

Enviar comentarios sobre…

¿Necesitas ayuda? Visita nuestra página de asistencia.