The Logs ingestion page in Stackdriver Logging tracks the volume of logs in your project. The page also gives you tools to exclude (discard) log entries you're not interested in, so that you can minimize any charges for logs over your monthly allotment. For more information about how excluded log entries are treated, see Overview of exclusions on this page.
Tracking logs usage
To track your project's logs volume, go the Logs ingestion page in the Logs Viewer:
At the top of the page, you see a summary of the logs that your projects is receiving:
Four numbers are reported:
Last month's ingested log volume: The amount of logs your project received in the last calendar month.
This month's ingested log volume: The amount of logs your project received since the first date of the current month.
Excluded log volume: The amount of logs that you have excluded from your project since the first date of the current month. This number is not included in This month's ingested log volume. Excluding logs is described later on this page.
Projected ingestion log volume: The estimated amount of logs your project will receive by the end of the current month, based on current usage.
The log volumes do not include the enabled-by-default audit logs—all Admin Activity logs plus BigQuery Data Access logs. Those logs are free and cannot be excluded.
You can also see the breakdown in logs usage by resource type. See Viewing resource-type exclusions on this page.
Overview of exclusions
The following diagram illustrates how excluded log entries are treated in Stackdriver Logging:
The following conditions apply to excluded log entries in Logging:
Excluded log entries do not count against the Logging allotment provided to projects. See Logging details for details.
Excluded log entries are not visible in the Logs Viewer; they are not counted in logs-based metrics; and they are not available to Stackdriver Error Reporting or Stackdriver Debugger.
You can export log entries before they are excluded. For more information, see Exporting Logs.
You cannot exclude any audit logs that are enabled by default; however, audit logs that are enabled by default are free.
You cannot exclude AWS logs or any logs from sources other than GCP.
There are two kinds of exclusions:
Resource-type exclusions are easy to use. They block all logs from specific resource types. For details, see Using resource-type exclusions on this page.
Exclusion filters are more flexible. They select log entries for exclusion based on filter expressions. Exclusion filters can also choose a random sample of log entries to exclude. For details, see Using exclusion filters.
When deciding whether or not to exclude a log entry, Logging considers both kinds of exclusions. If any resource-type exclusion or any exclusion filter matches the log entry, then that log entry is excluded.
Using resource-type exclusions
By default, your project receives all logs from all resource types. To discard all the logs from specific resource types, use resource type exclusions.
Resource-type exclusions are a convenience feature in the Logs Viewer. When you create a resource-type exclusion, Logging creates an exclusion filter that implements the exclusion. For more information, see Using exclusion filters.
Viewing resource-type exclusions
To view your logs usage by resource type and to see your resource-type exclusions, do the following:
Go to the Logs ingestion page of the Logs Viewer.
Select the Ingestions tab (the default) under the summary information. You see your logs usage by resource type:
The table shows logs usage information for each resource type that has sent logs to your project this current and past month. There could be resource types that only sent logs last month but not this month, which are also listed in this table.
The Ingestion Status column is an approximate indication of whether there are exclusions related to each resource type. The status can be any of the following:
Not ingested: There are one or more exclusions that exactly target this resource type at a 100% sample rate. That means the exclusion's filter consists of exactly
All ingested: There have been no log entries from this resource type excluded so far this month, and there are no exclusions that exactly target this resource type.
Partially ingested: There are one or more exclusions that target this resource type with a sampling rate between 0 percent and 100 percent. If this resource type has had any log entries excluded this month, then this status remains until the end of the month, even if all exclusions are currently removed. See Editing exclusions for more information.
Alternatively, you can inspect resource-type exclusions on the Exclusions tab. Logging implements resource-type exclusions by creating exclusion filters. See Viewing exclusion filters.
Creating resource-type exclusions
To exclude all logs from a resource type, do the following:
Go to the Logs ingestion page of the Logs Viewer.
Select the Ingestions tab (the default) under the statistics. You see your logs usage by resource type, as shown in the screenshot in the previous section.
Locate the table row for the resource type you want to disable.
Click Disable log source in the menu on the right side of the table row.
To stop excluding logs from the resource type, click Enable log source in the menu.
Using exclusion filters
By creating exclusion filters you can control exactly which log entries you exclude. For example, you could exclude log entries from a single VM instance rather than from all VM instances.
If you use both exclusion filters and resource-type exclusions, they can overlap. A log entry is excluded if it is from a disabled resource type, or if it matches one of the exclusion filters discussed in this section. Note that this is a technical distinction, since as mentioned previously, Logging implements resource-type exclusions with exclusion filters.
The per-resource-type table on the Ingestions tab of the Logs ingestion page reflects both resource-type exclusions and exclusion filters. Even if you do not use resource-type exclusions, that table lets you track the effect of your exclusion filters.
Viewing exclusion filters
To view your current exclusion filters—including the filters created by Logging to implement your resource-type exclusions—do the following:
Visit the Logs ingestion page of the Logs Viewer. Select the project whose logs you want to manage.
Under the summary statistics, click the Exclusions tab. You see a list of your exclusion filters (if any):
Creating exclusion filters
To create an exclusion filter using the Logs Viewer, do the following:
Visit the Logs ingestion page of the Logs Viewer and select the Exclusions tab.
Click Create Exclusion. You see the Exclusion Editor beside a Logs Viewer panel:
In the Logs Viewer panel, enter a filter expression that matches the log entries you want to exclude. For more information about the panel, see The user interfaces.
In the Exclusion Editor, complete your exclusion filter by filling in the text boxes:
- Name: A name that identifies the exclusion, such as
- Description: A longer description of the filter, such as
Exclude logs whose severity is less than WARNING.
- Percent to Exclude: Enter an integer or floating-point value from 0
to 100. For example, enter
100.0to exclude all matching log entries. Enter
99.50to exclude 99.5% of the matching entries but keep 0.5% of them in Logging.
- Name: A name that identifies the exclusion, such as
Click Create Exclusion to create and start the exclusion.
Tip: If you want to prevent your exclusion filter from being used, then select Disable in the menu next to your exclusion filter.
To stop excluding some or all logs, do one or both of the following:
On the Ingestions tab of the Logs ingestion page, click Enable log source in the menu to the right side of a resource type you want to stop excluding.
On the Exclusions tab of the Logs ingestion page, delete or disable any exclusion filters that target log entries you want to receive.
Tip: Check all your exclusion filters, because the same log entries could be targeted by more than one filter.
You can edit your existing exclusion filters to exclude more or fewer log entries.
Go to the Logs ingestion page of the Logs Viewer and click the Exclusions tab.
Choose an exclusion filter and select Edit Exclusion from the menu at the right side of the filter listing.
Change the advanced logs filter or change the Percent to Exclude value. If you change the filter, check the preview of matching log entries. You cannot change the name of the exclusion filter.
Click Update Exclusion.
Best practice: Do not edit exclusion filters created by Logging as part of the resource-type exclusions. Manage those filters with the Disable log source and Enable log source options on the Ingestions tab.
Exclusions in the API
To create exclusion filters in the Stackdriver Logging API, use the projects.exclusions.create method. There are also methods to view, delete, and update exclusion filters.
There are also exclusion methods in the API for logs received by organizations, billing accounts, and folders. Those exclusions can only be created in the Stackdriver Logging API; they are not supported in the Logs Viewer.
For examples of logs filters that might be useful in exclusions, see Advanced Logs Filters.
Resource-type exclusions in the API
Resource-type exclusions are not a separate kind of exclusion in the API. To create an exclusion that discards all log entries from a particular resource type, create an exclusion filter with a logs filter that specifies the resource type:
resource.type = [THE_RESOURCE_TYPE]
Sampled exclusions in the API
To exclude less than 100 percent of the matched log entries, use
sample function in your
You can have up to 50 exclusion filters in a project. This includes exclusion filters and resource-type exclusions created in the Logs Viewer or in the API.
Exporting excluded logs
You can export log entries before you exclude them, so that you don't permanently lose the log entries you exclude.
To start your exclusion and export, do the following:
Create an advanced logs filter that matches the log entries you want to exclude and export.
Tip: Write the filter so that it does not match any default Cloud Auditing log entries. Matching the audit log entries doesn't matter for exclusions, but it does result in exporting more log entries.
Create an export sink using your logs filter. Start exporting the matching log entries.
Create an exclusion filter using your logs filter. Start excluding the matching log entries.
To stop your exclusions and export, stop the exclusion filter before you stop the export sink.