Logs exclusions

This page describes how to exclude logs from ingestion using the Logs Viewer (Cloud Console) and the Logging API.

The Logs ingestion page in the Logs Viewer tracks the volume of logs in your project. The Logs Viewer also gives you tools to disable all logs ingestion or exclude (discard) log entries you're not interested in, so that you can minimize any charges for logs over your monthly allotment. For more information about how excluded log entries are treated, go to How exclusions work on this page.

For details on Stackdriver Logging costs, see Stackdriver pricing. Note that if you send and then exclude your Virtual Private Cloud flow logs from Stackdriver Logging, VPC flow log generation charges apply.

Tracking logs usage

To track your project's logs volume, go to the Logs ingestion page in the Stackdriver Logging console:

Go to the Logs ingestion page

The top of the page displays a summary of statistics for the logs that your project is receiving:

Logs Ingestion Summary

Four statistics are reported:

  • Last month's ingested log volume: The amount of logs your project received in the last calendar month.

  • This month's ingested log volume: The amount of logs your project has received since the first date of the current month.

  • Excluded log volume: The amount of logs that you have excluded from your project since the first date of the current month. This number is not included in This month's ingested log volume. Excluding logs is described later on this page.

  • Projected ingestion log volume: The estimated amount of logs your project will receive by the end of the current month, based on current usage.

The log volumes do not include certain audit logs: all Admin Activity audit logs and all System Event audit logs. Those logs are free and cannot be excluded or disabled.

The breakdown in logs usage by resource type is also displayed. For details, go to Viewing resource-type exclusions on this page.

Stopping all logs ingestion

You can disable logs ingestion, so that you won't be charged for any logs ingested for your project. Note that certain non-configurable audit logs cannot be disabled; you might continue to see these in your project but you won't incur any costs for them.

To immediately stop ingestion for all configurable logs, do the following:

  1. Go to the Logs ingestion page of the Stackdriver Logging console for the project whose logs you want to manage:

    Go to Logs ingestion

  2. Above the summary statistics, click Logs Enabled.

  3. In Disable all logs ingestion?, click Reduce Logs Ingestion.

    The following changes in the Stackdriver Logging console appear:

    • The button that you clicked in step 2 now reads as Logs Disabled.
    • A banner is displayed on each page of the Stackdriver Logging console: Configurable logs ingestion is OFF...
    • In the Exclusions tab on the Logs ingestion page, there is a query named google-ui-logs-ingestion-off. Its Percent to Exclude value is set at 100% and its Exclusion Status is Active.

To immediately re-enable logs ingestion, click Logs Disabled. The button now reads as Logs Enabled.

How exclusions work

The following diagram illustrates how excluded log entries are treated in Stackdriver Logging:

Life of a log

The following conditions apply to excluded log entries in Logging:

  • Excluded log entries do not count against the Logging allotment provided to projects. See Logging details for details.

  • Excluded log entries are not visible in the Logs Viewer; they are not counted in logs-based metrics; and they are not available to Stackdriver Error Reporting or Stackdriver Debugger.

  • You can export log entries before they are excluded. For more information, read Logs exports.

  • You cannot exclude any audit logs that are enabled by default; however, audit logs that are enabled by default are free.

There are two kinds of exclusions:

  • Exclusion queries give you the flexibility to select log entries for exclusion based on query expressions. You can use exclusion queries to choose a random sample of log entries to exclude.

  • Resource-type exclusions let you block all logs from specific resource types.

Logging considers both kinds of exclusions when deciding whether or not to exclude a log entry. If any resource-type exclusion or any exclusion query matches the log entry, then that log entry is excluded.

Using exclusion queries

By creating exclusion queries you can control which log entries you exclude (discard). For example, you could exclude log entries from a single VM instance rather than from all VM instances.

If you use both exclusion queries and resource-type exclusions, they might overlap. A log entry is excluded if it is from a disabled resource type, or if it matches one of the exclusion queries discussed in this section. Note that this is a technical distinction, since as mentioned previously, Logging implements resource-type exclusions with exclusion queries.

The per-resource-type table on the Ingestions tab of the Logs ingestion page reflects both resource-type exclusions and exclusion queries. Even if you do not use resource-type exclusions, that table lets you track the effect of your exclusion queries.

Exclusion limits

You can have up to 50 exclusion queries in a project. This includes exclusion queries and resource-type exclusions created in the Stackdriver Logging console or in the API.

Viewing exclusion queries

To view your current exclusion queries—including the queries created by Logging to implement your resource-type exclusions—do the following:

  1. Visit the Logs ingestion page of the Stackdriver Logging console. Select the project whose logs you want to manage:

    Go to the Logs ingestion page

  2. Under the summary statistics, click the Exclusions tab. The Exclusion queries table displays a list of your exclusion queries (if any):

    Exclusions query table

If you have stopped all logs ingestion, your list includes an exclusion query named google-ui-logs-ingestion-off. You can edit, delete, or disable this query using the menu on the right side of the query.

Creating exclusion queries

To create an exclusion query using the Stackdriver Logging console, do the following:

  1. Go to the Logs ingestion page of the Stackdriver Logging console and select the Exclusions tab:

    Go to the Logs ingestion page

  2. Click Create Exclusion. The Exclusion Editor pane appears:

    Create Exclusion

  3. In the Logs Viewer panel, enter a query expression that matches the log entries you want to exclude. For more information about the panel, see The user interfaces.

  4. In Exclusion Editor, complete your exclusion query by filling in the text boxes:

    • Name: A name that identifies the exclusion, such as low-severity.
    • Description: A longer description of the query, such as Exclude logs whose severity is less than WARNING.
    • Percent to Exclude: Enter an integer or floating-point value from 0 to 100. For example, enter 100 or 100.0 to exclude all matching log entries. Enter 99.50 to exclude 99.5% of the matching entries but keep 0.5% of them in Logging.
  5. Click Create Exclusion to create and start the exclusion.

Stopping exclusions

You can stop excluding some or all logs in several ways:

  • To stop excluding all logs: See Stopping logs ingestion for details.

  • To stop excluding by resource type: On the Ingestions tab of the Logs ingestion page, click Enable log source in the menu to the right side of a resource type you want to stop excluding.

  • To edit, disable, or delete an exclusion query: On the Exclusions tab of the Logs ingestion page, use the menu to the right of any exclusion queries that target log entries you want to receive. See Editing exclusions for details.

Editing exclusions

You can edit your existing exclusion queries to exclude more or fewer log entries.

  1. Go to the Logs ingestion page in the Stackdriver Logging console and click the Exclusions tab:

    Go to the Logs ingestion page

  2. Choose an exclusion query and select Edit Exclusion from the menu at the right side of the query listing.

  3. Change the advanced logs query or change the Percent to Exclude value. If you change the query, check the preview for matching log entries. You cannot change the name of an existing exclusion query.

  4. Click Update Exclusion.

Best practice: Do not edit or delete exclusion queries created by Logging as part of the resource-type exclusions. Manage those queries with the Disable log source and Enable log source options on the Ingestions tab.

Using resource-type exclusions

By default, your project receives all logs from all resource types. To discard all the logs from specific resource types, use resource type exclusions.

Resource-type exclusions are a feature of the Stackdriver Logging console. When you create a resource-type exclusion, Logging creates an exclusion query that implements the exclusion. For more information, go to Using exclusion queries.

Viewing resource-type exclusions

To view your logs usage by resource type and your resource-type exclusions, do the following:

  1. Go to the Logs ingestion page in the Stackdriver Logging console:

    Go to the Logs ingestion page

  2. Select the Ingestions tab (the default) under the summary statistics. The Logs Ingestion table displays your logs usage by resource type:

    Resource Usage Table

The table shows logs usage information for each resource type that has sent logs to your project this current and past month. There could be resource types that only sent logs last month but not this month, which are also listed in this table.

The Ingestion Status column is an approximate indication of whether there are exclusions related to each resource type. The status can be any of the following:

  • Not ingested: There are one or more exclusions that exactly target this resource type at a 100% sample rate. That means the exclusion's query consists of exactly resource.type=[THIS_RESOURCE_TYPE].

  • All ingested: There have been no log entries from this resource type excluded so far this month, and there are no exclusions that exactly target this resource type.

  • Partially ingested: There are one or more exclusions that target this resource type with a sampling rate between 0 percent and 100 percent. If this resource type has had any log entries excluded this month, then this status remains until the end of the month, even if all exclusions are currently removed. For more information, go to Editing exclusions on this page.

Alternatively, you can inspect resource-type exclusions on the Exclusions tab. Logging implements resource-type exclusions by creating exclusion queries. For details, go to Viewing exclusion queries.

Creating resource-type exclusions

To exclude (discard) all the logs from a specific resource type, create a resource-type exclusion. Follow these steps:

  1. Go to the Logs ingestion page in the Stackdriver Logging console:

    Go to the Logs ingestion page

  2. Select the Ingestions tab (the default) under the statistics summary. The Logs Ingestion table displays your logs usage by resource type.

  3. Locate the table row for the resource type you want to exclude.

  4. Select Create exclusion query based on this resource in the menu on the right side of the table row.

  5. In the Exclusion Editor, complete your exclusion query by filling in the text boxes:

    • Name: A name that identifies the exclusion, such as low-severity.
    • Description: A longer description of the query, such as Exclude logs whose severity is less than WARNING.
    • Percent to Exclude: Enter an integer or floating-point value from 0 to 100. For example, enter 100 or 100.0 to exclude all matching log entries. Enter 99.50 to exclude 99.5% of the matching entries but keep 0.5% of them in Logging.
  6. Click Create Exclusion to create and start the exclusion.

To stop excluding logs from the resource type, click Enable log source in the menu.

Exclusions in the API

To create exclusion queries in the Logging API, use the projects.exclusions.create method. There are also methods to view, delete, and update exclusion queries.

There are also exclusion methods in the API for logs received by organizations, billing accounts, and folders. Those exclusions can only be created in the Logging API; they are not supported in the Stackdriver Logging console.

For examples of logs queries that might be useful in exclusions, go to Sample queries.

Resource-type exclusions in the API

Resource-type exclusions are not a separate kind of exclusion in the API. To create an exclusion that discards all log entries from a particular resource type, create an exclusion query with a logs query that specifies the resource type:

resource.type = [THE_RESOURCE_TYPE]

Sampled exclusions in the API

To exclude less than 100 percent of the matched log entries, use the sample function in your logs query.

Exporting excluded logs

You can export log entries to Cloud Storage, BigQuery, or Pub/Sub before you exclude them, so that you don't permanently lose the log entries you exclude.

To start your exclusion and export, do the following:

  1. Create an advanced logs query that matches the log entries you want to exclude and export.

    Tip: Write the query so that it does not match any audit logs that are enabled by default. Matching these audit log entries doesn't affect exclusions, but it does result in exporting more log entries.

  2. Create an export sink using your logs query, and start exporting the matching log entries.

  3. Create an exclusion query using your logs query and start excluding the matching log entries.

To stop your exclusions and export, disable the exclusion query before you stop the export sink.

For more details about how to export logs, see Exporting Logs.

Exports pricing

Exported logs don't incur Stackdriver Logging charges, but destination charges might apply. For details, review the appropriate product's pricing page:

Note also that if you send and then exclude your Virtual Private Cloud flow logs from Stackdriver Logging, VPC flow log generation charges apply in addition to the destination charges.

Hai trovato utile questa pagina? Facci sapere cosa ne pensi:

Invia feedback per...

Stackdriver Logging Documentation