Advanced Logs Filters

Learn how to write advanced logs filters, which are expressions that can specify a set of log entries from any number of logs. Advanced log filters can be used in the Logs Viewer, the Stackdriver Logging API, or the command-line interface.

Introduction

An advanced logs filter is a Boolean expression that specifies a subset of all the log entries in your project. It can be used to do any of the following:

  • Choose log entries from specific logs or log services.
  • Choose log entries within a time range.
  • Choose log entries that satisfy conditions on metadata or user-defined fields.
  • Choose a sampling percentage of all log entries.

To use advanced filters in the Logs Viewer, select the menu (▾) at the end of the search bar and choose Convert to advanced filter. You see the advanced filter interface pictured below:

An advanced logs filter

Following is a simple example of an advanced filter:

v1 log entries

metadata.serviceName = "compute.googleapis.com" AND
metadata.severity >= ERROR AND
NOT textPayload:robot

v2 log entries

resource.type = "gce_instance" AND
severity >= ERROR AND
NOT textPayload:robot

This filter matches log entries from Google Compute Engine that have severity values of at least ERROR and whose textPayload field does not contain the string robot anywhere inside it. String comparisons are not case sensitive. The names metadata, serviceName, resource, severity, and textPayload are defined in the LogEntry and LogEntry v2 types.

Suggestions and highlighting as you type When you type an advanced filter into the Logs Viewer, you might see suggestions for specific log entry fields. The suggestions come from both the filter language and the set of actual log entries that the Logs Viewer has loaded. To choose a suggestion, press TAB. If you don't see suggestions, try pressing CTRL+SPACE. The different parts of a filter expression are highlighted in different colors. If your expression is a bright red then the filter you've typed so far might have an error or might be incomplete.

Unquoted text: You can omit quotation marks around text strings that don't contain whitespace or certain special characters. This is called unquoted text, and examples are the words ERROR and robot in the preceding example. The string "appengine.googleapis.com" is quoted because it contains periods. If you wish to include a quotation mark within a string, precede the quotation mark with a backslash (\).

Best practice: It is best to quote strings that you are comparing against field values, because quoting guards against mistakes that change the meaning of your comparisons and are difficult to debug. It is safe to omit the quotes on words that consist of a letter followed by a sequence of letters, digits, and underscore (_) characters.

Advanced filter syntax

This section discusses how advanced filters are structured and how matching is performed.

Syntax notation

The advanced filter syntax is described using the following notation:

  • a = e means that a is a name for the expression e.
  • a b means "a followed by b."
  • a | b means "a or b."
  • ( e ) is used for grouping.
  • [ e ] means that e is optional.
  • { e } means that e can be repeated zero or more times.
  • "abc" means that abc must be written just as it appears.

Filter summary

This section provides a quick overview of the advanced filter syntax. Some details have been omitted, but they are explained in the following sections as you need them:

  • An advanced filter is a string containing an expression, a Boolean combination of comparisons:

    expression = ["NOT"] comparison { ("AND" | "OR") ["NOT"] comparison }
    
  • A comparison matches a named log entry field with a value. All the common comparison operators can be used. The has operator, a colon, allows a string value to match any substring of the log entry field.

    comparison = name OP value
               | value
    OP = "<=" | "<" | ">=" | ">"  | "!=" | "=" | ":"
    

    A comparison consisting of only a value is called a global restriction. The value is compared with every field in the log entry using the has operator.

  • Values appearing in advanced filters can be numbers, strings, functions, or parenthesized expressions. Strings are used to represent arbitrary text, plus Boolean, enumeration, and byte-string values.

    value = number
          | string
          | function
          | "(" expression ")"
    

The following sections provide more details about filters and matching.

Boolean expressions

expression =
    ["NOT" | "-"] comparison {["AND" | "OR"] ["NOT" | "-"] comparison}

The Boolean operators AND and OR are short-circuit operators. The NOT operator has the highest precedence, followed by OR and AND in that order. For example, the following two expressions are equivalent:

a OR NOT b AND NOT c OR d
(a OR (NOT b)) AND ((NOT c) OR d)

You can omit the AND operator between comparisons, and you can use - (hyphen-minus) in place of NOT. For example, the following two advanced filters are the same:

c=d AND NOT e=f
c=d -e=f

This documentation does not omit AND or use - instead of NOT.

Comparisons

This section discusses "name OP value" comparisons. Comparisons that are global restrictions are discussed in a following section.

comparison = name OP value
OP = "<=" | "<" | ">=" | ">" | "!=" | "=" | ":"
name = identifier { "." identifier }
identifier = unquoted_text | string
value = number | string | unquoted_text | function | "*"
      | "(" expression ")"

The left-hand side of a comparison is typically the path name of a scalar log entry field, as discussed in the following Names section. The right-hand side is typically a scalar value that is converted to the field's type and compared against it. See the Values and conversions section for more details, including the handling of arrays.

The log entry field can be an array of values. See the following Names section for details.

If the right-hand side value is a parenthesized Boolean combination of comparison values, then the field name and the comparison operator are applied to each of the values. For example, the following two advanced filters are the same:

v1 log entries

structPayload.cat = ("siamese" OR "shorthair")

structPayload.cat = "siamese" OR
structPayload.cat = "shorthair"

v2 log entries

jsonPayload.cat = ("siamese" OR "shorthair")

jsonPayload.cat = "siamese" OR
jsonPayload.cat = "shorthair"

Names

All log entries are instances of type LogEntry or LogEntry (v2), which means the first identifier in a field path must be a field defined in the LogEntry type. If your first path identifier is metadata, then the next identifier must be a field in the LogEntryMetadata type. If your first path identifier is httpRequest, then the next identifier must be a field in the HttpRequest type. Following is a list of all the names you can use in your advanced filters:

v1 log entries

For details, see the v1 LogEntry type.

log
insertId
httpRequest.FIELD          # The value of a field in an HttpRequest object.
metadata.FIELD             # The value of a field within a LogEntryMetadata object.
metadata.labels.KEY        # The value associated with a metadata label key.
protoPayload.FIELD         # The value of a field within a protocol buffer object.
structPayload.FIELD        # The value of a field within a JSON object.
textPayload

v2 log entries

For details, see the v2 LogEntry type.

logName
insertId
severity
timestamp
resource.type              # The name of a resource type. Example: "gce_instance".
resource.labels.KEY        # The value associated with a resource label key.
httpRequest.FIELD          # the value of a field in an HttpRequest object.
labels.KEY                 # The value associated with a label key.
operation.FIELD            # The value of a field in a LogEntryOperation object.
protoPayload.FIELD         # The value of a field within a protocol buffer object.
jsonPayload.FIELD          # The value of a field within a JSON object.
textPayload

For any given log entry, only one of the payload fields can be present: textPayload, protoPayload, structPayload (v1), or jsonPayload (v2). The fields httpRequest and operation are optional in log entries. For more information on using names that reference objects or arrays, see Object and array types.

Service names and resource types

For faster searches, specify a log service name (v1) or a monitored resource type (v2). For a list of service names and resource types, see Monitored Resource Types.

For example, Google Container Engine uses service name container.googleapis.com in the v1 API, and has two resource types, container and gke_cluster, in the v2 API. The following example shows how to limit your searches to Container Engine resources:

v1 log entries

metadata.serviceName = "container.googleapis.com"

v2 log entries

resource.type = ("container" OR "gke_cluster")

Service names and resource types are indexed. Using substring matches for them results in slower queries.

Missing fields

If you use a field name in a filter, and that field does not appear in a log entry, then the field can be missing, undefined, or defaulted:

  • If the field is inside the log entry's payload (jsonPayload, structPayload, or protoPayload), or if it is in the labels section of the log entry, then the field is missing. Stackdriver Logging cannot tell if the field is permitted or not. Unless otherwise noted, all comparisons involving missing fields fail silently.

    Examples: jsonPayload.nearest_store, protoPayload.name.nickname

  • If the field is defined in the LogEntry type or its component types LogEntryMetadata (v1), HttpRequest, LogEntryOperation, or LogEntrySourceLocation, then the field is defaulted. The field's default value is used in all comparisons.

    Examples: httpRequest.remoteIp, trace, operation.producer

  • Otherwise, the field is undefined, because Stackdriver Logging knows what fields are defined in the fixed parts of LogEntry. Using an undefined field is an error.

    Examples: thud, operation.thud, textPayload.thud

To test if a missing or defaulted field exists without testing for a particular value in the field, use the :* comparison. For example, the following comparison succeeds if the field operation.id is explicitly present in a log entry:

operation.id:*

Object and array types

This section presents some terminology used to describe how Stackdriver Logging processes names like a.b.c.

This documentation uses the terms scalar, object, and array when describing log entry fields. Each log entry field has a type, which can be a scalar type, like double or string, or an object type that has other fields inside it. A protocol buffer message and a JSON object are examples of object types.

Likewise, each log entry field can hold either a single instance of its type or an array of instances. A protocol buffer repeated field and a JSON field containing an array of values—all of the same type—are examples of array fields.

There are some types used in Stackdriver Logging that are really object types, but since they can be converted to and from strings, they are treated as scalars. These types include Duration and Timestamp.

Path names in comparisons

If a comparison uses a field named a.b.c, the field is treated as follows:

  • In the typical case, a is an object containing field b, b is an object containing field c, and c has a scalar type. The value of the c field is tested against the right-side value of the comparison. The handling of missing fields is described in Missing fields.

  • Field c can be an array, in which case a comparison involving a.b.c compares all the instances a.b.c[0], a.b.c[1], .... If any of the instances matches then the comparison succeeds. (The notation [0], [1], ..., cannot be used in advanced filters; it is only for this explanation.)

  • Field b can likewise be an array, in which case a comparison involving a.b.c compares all the instances a.b[0].c, a.b[1].c, .... If any of the instances matches then the comparison succeeds.

Values and conversions

The first step in evaluating a comparison is to convert the right-hand side value to the type of the log entry field. Scalar field types are permitted in comparisons, along with two additional types whose values are represented as strings: Duration and Timestamp. For a list of scalar types, see the scalar protocol buffer types list. The following table explains what values can be converted to the log field types:

Field type Permitted filter value
bool "true" or "false" in any letter case. Example: "True", true.
bytes A string containing any sequence of bytes. Example: "\377\377".
Duration A string containing a signed decimal number followed by one of the units ns, us, ms, s, m, or h. Durations are accurate to nanoseconds. Example: "3.2s".
enum The name of an enumeration type literal, case-insensitive. Example: WARNING, which is a value of type LogSeverity.
double Any number, with or without a sign and an exponent part, or the special value strings "NaN", "-Infinity", and "Infinity" (either capitalized or not). Example: -3.2e-8, "nan".
intNN Any signed integer that does not exceed the size of the type. Example: -3.
string Any string that contains UTF-8 encoded or 7-bit ASCII text. Embedded quotation marks must be escaped with a backslash.
Timestamp A string in RFC3339 format. Example: "2014-10-02T15:01:23.045Z". In filter expressions, timestamps can specify a timezone with Z or ±hh:mm. Timestamps are represented to nanosecond accuracy.
uintNN Any unsigned integer that does not exceed the size of the type. Example: 1234.

If an attempted conversion fails, then the comparison fails.

When a conversion requires a string, you can also use a number or unquoted text if they do not contain special characters such as spaces and operators. Similarly, when a conversion requires a number, you can use a string whose content is a number.

The types intNN and uintNN represent integer types of various sizes, such as int32 and uint64. When writing a value to be converted to a 64-bit integer type, you should write the value as a string, such as "9223372036854775807".

Types of log fields

Here is how the type of a log entry field is determined:

  • Log fields defined in the types LogEntry (v1), LogEntry (v2), and in their component types are protocol buffer fields. Protocol buffer fields have explicit types.

  • Log fields that are part of protoPayload objects are also protocol buffer fields and have explicit types. The name of the protocol buffer type is stored in the field "@type" of protoPayload. See JSON mapping for more information.

  • Log fields inside of structPayload (v1) or jsonPayload (v2) have types that are inferred from the field's value when the log entry is received:

    • Fields whose values are unquoted numbers have type double.
    • Fields whose values are true or false have type bool.
    • Fields whose values are strings have type string.

    Long (64-bit) integers are stored in string fields, because they cannot be represented exactly as double values.

  • The Duration and Timestamp types are recognized only in protocol buffer fields. Elsewhere, those values are stored in string fields.

Comparison operators

The meaning of the equality (=, !=) and inequality (<, <=, >, >=) operators depends on the underlying type of the left-hand field name.

  • All numeric types: Equality and inequality have their normal meaning for numbers.
  • bool: Equality means the same Boolean value. Inequality is defined by true>false.
  • enum: Equality means the same enumeration value. Inequality uses the underlying numeric values of the enumeration literals.
  • Duration: Equality means the same duration length. Inequality is based on the length of the duration. Example: as durations, "1s">"999ms".
  • Timestamp: Equality means the same instant in time. If a and b are Timestamp values, a < b means a is earlier in time than b.
  • bytes: Operands are compared byte by byte, left-to-right.
  • string: Comparisons ignore letter case. Specifically, both operands are first normalized using NFKC_CF Unicode normalization and then use lexicographic comparisons.

The substring operator (:) is applicable to strings and bytes and is handled like equality except that the right-hand operand need only equal some part of the left-hand field. Substring matches on indexed fields do not take advantage of the index.

Global restrictions

    comparison = name OP value
               | value

The second form of the comparison syntax, consisting of a single value, is called a global restriction. In the following example, the first two advanced filters each contain a single global restriction. The third filter is a Boolean combination of global restrictions:

3.14159
"The Cat in The Hat"
(cat AND (hat OR bat))

A global restriction compares its value with all fields in a log entry using the has (:) operator. The value is converted to each field's type before the comparison, and if the conversion fails then the match fails for that field. If the value matches any field, then the global restriction succeeds.

In the third filter in the previous example, the global restriction is applied to each value separately and then combined, just as it would if the expression were written without parentheses. That is, cat and either hat or bat must appear in some field of the log entry, but not necessarily in the same field.

A global restriction is an easy way to search your logs for a particular value. For example, if you are looking in your activity log for entries containing any mention of GCE_OPERATION_DONE, you can use the following filters:

v1 log entries

log = "compute.googleapis.com/activity_log" AND
"GCE_OPERATION_DONE"

v2 log entries

logName = "projects/my-project-id/logs/compute.googleapis.com%2Factivity_log" AND
"GCE_OPERATION_DONE"

Although global restrictions are easy, they may also be slow. See Finding log entries quickly.

Functions

You can use built-in functions as global restrictions in advanced filters:

function = identifier "(" [ argument { "," argument } ] ")"
argument = value | name | "(" expression ")"

The functions are described in the following sections.

sample

The sample function selects a fraction of the total number of log entries:

sample([FIELD], [FRACTION])

[FIELD] is the name of a field in the log entry, such as logName or jsonPayload.a_field. The value of the field determines whether the log entry is in the sample. The field must contain a string or numeric value. The field insertId is a good choice, because every log entry has a different value for that field.

[FRACTION] is the fraction of log entries to include. It is a number greater than 0.0 and no greater than 1.0. For example, if you specify 0.01, then the sample contains roughly one percent of all log entries. If [FRACTION] is 1, then all the log entries that have values for [FIELD] are chosen.

Example: The following filters return roughly 25 percent of the log entries from log syslog:

v1 log entries

log = syslog AND sample(insertId, 0.25)

v2 log entries

logName = "projects/my-project/logs/syslog" AND sample(insertId, 0.25)

Details: There is no randomess in selecting log entries for the sample. The value in [FIELD] is hashed, and the specified fraction of hash values is chosen for the sample.

If the hashed values are not uniformly distributed, then the resulting sample can be skewed. In the worst case, when [FIELD] always contains the same value, the resulting sample contains either all of the log entries or no log entries.

If [FIELD] is not given a value in a log entry, it is treated as follows:

  • If [FIELD] is part of the log entry's payload or labels sections, then the log entry is not selected for the sample, even if [FRACTION] is 1.
  • If [FIELD] is not part of the payload, then it has a definition in the LogEntry type. The field is treated as if its default value were supplied. Whether the log entry is chosen for the sample depends on how the field's default value is hashed and on the value of [FRACTION].

For more information on missing and defaulted fields, see Missing fields.

To exclude log entries with defaulted fields from the sample, use the field-exists operator, :*. The following filter produces a 1 percent sample of log entries that have explicitly supplied a value for field:

field:* AND sample(field, 0.01)

ip_in_net

The ip_in_net function determines if an IP address in a log entry is contained in a subnet. You might use this to tell if a request comes from an internal or external source, for example:

ip_in_net([FIELD], [SUBNET])

[FIELD] is a string-valued field in the log entry that contains an IP address or range. The field can be repeating, in which case only one of the repeated fields has to have an address or range contained in the subnet.

[SUBNET] is a string constant for an IP address or range. It is an error if [SUBNET] is not a legal IP address or range, as described later in this section.

Example: The following filters test an IP address in the payload of log entries from the log my_log:

v1 log entries

log = "my_log" AND
ip_in_net(structPayload.realClientIP, "10.1.2.0/24")

v2 log entries

logName = "projects/my_project/logs/my_log" AND
ip_in_net(jsonPayload.realClientIP, "10.1.2.0/24")

Details: If, in a log entry, [FIELD] is missing, defaulted, or it does not contain a legal IP address or range, then the function returns false. For more information on missing and defaulted fields, see Missing fields.

Examples of the supported IP addresses and ranges are listed below:

  • IPv4: 10.1.2.3
  • IPv4 subnet: 10.1.2.0/24
  • CIDR IPv6: 1234:5678:90ab:cdef:1234:5678:90ab:cdef
  • CIDR IPv6 subnet: 1:2::/48

Finding log entries quickly

When filtering or querying for log entries:

  • Search using indexed fields.
  • Minimize the number of log entries that must be searched.

Use indexed fields

Specify exact values for indexed fields. Do not use substring matches. The following log entry fields are indexed:

v1 log entries

The following fields are indexed:

v2 log entries

The following fields are indexed:

Temporary field indexes

Three additional log entry fields are partially indexed for a limited time after Stackdriver Logging receives the log entries. If you are trying to respond quickly to problems with your system, searching on these fields might be helpful. Stackdriver Logging might change how it indexes these three fields in the future, without notice:

If you routinely search for these fields in recent log entries, you should also consider limiting the search time period using the metadata.timestamp (v1) or timestamp (v2) fields.

Minimize logs, log entries, and time

Make your searches faster by reducing the number of logs, the number of log entries, or the time span of your searches. Even better, do all three.

Example: Use the right log name

Specify the log containing the log entries you're interested in. Be sure you know the real log name by inspecting one of your log entries. For example, the Logs Viewer shows that there is a log in the Compute Engine section named "activity_log". Looking closer at the activity log entries, you see that the log is actually named "compute.googleapis.com/activity_log". Following are the wrong and right ways to reference the activity log:

v1 log entries

The following comparison is incorrect. It doesn't match anything because it uses the wrong log name:

log = "activity_log"   # WRONG!

The following comparison is correct. It chooses log entries from the activity log.

log = "compute.googleapis.com/activity_log"

v2 log entries

The following comparison is incorrect. It doesn't match anything because it uses the wrong log name:

logName = "projects/my-project-id/logs/activity_log"   # WRONG!

The following comparison is correct. It chooses log entries from the activity log. You must URL-encode the log name, as shown:

logName = "projects/my-project-id/logs/compute.googleapis.com%2Factivity_log"

Example: Choose the right log entries

If you know that the log entries you want are coming from a particular VM instance, then specify it. Check for the right label names by inspecting one of the log entries that you want to search for. In the following example, resource_id is one of the v1 indexed labels:

v1 log entries

log = "compute.googleapis.com/activity_log" AND
metadata.serviceName = "compute.googleapis.com" AND
metadata.labels."compute.googleapis.com/resource_id" = "6731710280662790612"

v2 log entries

logName = "projects/my-project-id/logs/compute.googleapis.com%2Factivity_log"
resource.type = "gce_instance" AND
resource.labels.instance_id = "6731710280662790612"

You must specify a value for the service name (v1) or resource type (v2). Otherwise, the comparison of resource_id does not use the index.

Example: Choose the right time period

You should specify a time period to search in. A quick way of determining useful timestamps in RFC 3339 format is to use the Gnu/Linux date command:

$ date --rfc-3339=s
2016-06-27 17:39:00-04:00
$ date --rfc-3339=s --date="3 hours ago"
2016-06-27 14:40:00-04:00
$ date --rfc-3339=s --date="5 hours ago"
2016-06-27 12:40:00-04:00

Use the values of these timestamps in the following filters. To create a timestamp acceptable to Stackdriver Logging, replace the space between the date and time with the letter T:

v1 log entries

Search within the last three hours:

metadata.timestamp >= "2016-06-27T14:40:00-04:00"

Search between three and five hours ago:

metadata.timestamp >= "2016-06-27T12:40:00-04:00" AND
metadata.timestamp <= "2016-06-27T14:40:00-04:00"

v2 log entries

Search within the last three hours:

timestamp >= "2016-06-27T14:40:00-04:00"

Search between three and five hours ago:

timestamp >= "2016-06-27T12:40:00-04:00" AND
timestamp <= "2016-06-27T14:40:00-04:00"

See Temporary field indexes for another example of using timestamps.

Minimize global and substring searches

Avoid the temptation to take shortcuts when typing logging filters.

Example: Don't use substrings in indexed fields

You are searching for a v2 log entry from the Apache2 web server. You know that the Apache logs are named apache-access and apache-error. What do you do?

v1 log entries

  • Don't use a substring match to save yourself some typing:

    log:apache   # THIS CAUSES A SLOW SEARCH!
    
  • Do use exact-matches when searching indexed fields:

    log = ("apache-access" OR "apache-error")
    

v2 log entries

  • Don't use a substring match to save yourself some typing:

    logName:apache   # THIS CAUSES A SLOW SEARCH!
    
  • Do use exact-matches when searching indexed fields:

    logName = ("projects/my-project-id/logs/apache-access" OR
               "projects/my-project-id/logs/apache-error")
    

An indexed field loses all of its quickness when you do a substring search.

Example: Don't use global searches

You're searching for a log entry with "Hello, Kitty" in the payload:

v1 log entries

  • Don't use a global search. One reason is they are all substring searches:

    "Hello, Kitty"   # THIS CAUSES A SLOW SEARCH!
    
  • Do limit the search to a single field, even if you must keep the substring search:

    textPayload:"Hello, Kitty"
    
  • Do use an equality test if you can:

    textPayload = "Hello, Kitty"
    
  • Do reference individual fields in a payload, if your log entries have structured payloads:

    structPayload.my_favorite_cat = "Hello, Kitty"
    
  • Do use an indexed field to restrict the search:

    log = "somelog" AND
    structPayload.my_favorite_cat = "Hello, Kitty"
    

v2 log entries

  • Don't use a global search. One reason is they are all substring searches:

    "Hello, Kitty"   # THIS CAUSES A SLOW SEARCH!
    
  • Do limit the search to a single field, even if you must keep the substring search:

    textPayload:"Hello, Kitty"
    
  • Do use an equality test if you can:

    textPayload = "Hello, Kitty"
    
  • Do reference individual fields in a payload, if your log entries have structured payloads:

    jsonPayload.my_favorite_cat = "Hello, Kitty"
    
  • Do use an indexed field to restrict the search:

    logName = "projects/my-project_id/logs/somelog" AND
    jsonPayload.my_favorite_cat = "Hello, Kitty"
    

Troubleshooting

If you have problems with your advanced filters, check the following:

  • Your filter obeys the syntax rules, with matched parentheses and quotation marks.

  • Your log entry field names are correctly spelled.

  • You are using field names for the appropriate log entry version—v1 or v2.

  • Boolean operations are in upper-case letters (AND, OR, NOT).

  • Boolean expressions as global restrictions or as the right-hand side of comparisons should be parenthesized for clarity. For example, the two filters below look the same, but are not:

    insertId = "ABC-1" OR "ABC-2"  # ERROR!?
    insertId = ("ABC-1" OR "ABC-2)
    
  • Unquoted text must not contain any special characters. When in doubt, add double quotation marks. For example, the first comparison below is illegal because of the embedded substring operator (:). The comparison must be written with quotation marks:

    insertId = abc:def  # ILLEGAL!
    insertId = "abc:def"
    
  • Search expressions in the Logs Viewer's basic filters are different from the search expressions in advanced filters. For more information, see Differences in basic and advanced filters.

Monitor your resources on the go

Get the Google Cloud Console app to help you manage your projects.

Send feedback about...

Stackdriver Logging