UDM Search Best Practices

This document describes Google's recommended best practices for conducting searches using UDM Search. UDM searches can require substantial computational resources to complete if they are not constructed carefully. Performance also varies depending on the size and complexity of the data in your Google Security Operations instance.

Each condition must be in the form of udm-field operator value.

For example: principal.hostname = "win-server"

Always try to narrow your time range to the minimum necessary. Google Security Operations can ingest a tremendous amount of data, so limiting the breadth of that data while conducting a search can substantially improve search performance.

You can use regular expressions when conducting UDM searches:

  • Use AND, OR, and NOT.
  • AND is assumed in the absence of the other operators.
  • Use parentheses to modify the order of precedence.
  • Depending on the field type, field operators can include: = != >= > < <=

Using nocase as a search modifier

nocase can be used as a modifier to ignore capitalization.

For example, the following search is invalid:

target.user.userid = "TIM.SMITH" nocase

Regular expressions do not work for enumerated fields

You cannot use regular expressions for enumerated fields (fields with a range of predefined values) like metadata.event_type or network.ip_protocol.

For example, the following search is invalid:

metadata.eventtype = /NETWORK*/

However, the following search is valid (and approximates what was attempted above):

(metadata.event_type = "NETWORK_CONNECTION" or metadata.event_type = "NETWORK_DHCP")

Using any and all operators in the Events field

In UDM search, some fields are labeled as repeated, which means they are lists of values or other types of messages. Unlike YARA-L, repeated fields in UDM search are always treated with the any operator by default, with no option to specify all in your search.

When the any operator is used, the predicate is evaluated as true if any value in the repeated field satisfies the condition. For example, if you search for principal.ip != "1.2.3.4" and events in your search include both principal.ip = "1.2.3.4" and principal.ip = "5.6.7.8", the search will generate matches. This expands your search to include results that match any of the operators instead of matching with all of them.

Each element in the repeated field is treated individually. If the repeated field is found in events in the search, the events are evaluated for each element in the field. This can cause unexpected behavior, especially when searching using the != operator.

When using the any operator, the predicate is evaluated as true if any value in the repeated field satisfies the condition.

Timestamps use Unix epoch time

Timestamp fields are matched using Unix epoch time (number of seconds that have passed since Thursday 1 January 1970 00:00:00).

When searching for a specific timestamp, the following (in epoch time) is valid:

metadata.ingested_timestamp.seconds = 1660784400

The following timestamp is invalid:

metadata.ingested_timestamp = "2022-08-18T01:00:00Z"

There are certain fields that are excluded from filters, including the following:

  • metadata.id
  • metadata.product_log_id
  • *.timestamp

Since these fields tend to have unique values, displaying them creates more "noise" than value in the UDM Search interface.

Conjunctive Normal Form

UDM Search uses conjunctive normal form, an approach to Boolean logic that expresses formulas as conjunctions of clauses with an AND or OR. Each clause connected by a conjunction (AND), must be either a literal or contain a disjunction, (OR).

For example:

  • (A OR B) AND (C OR D)
  • (A OR B) AND (NOT C OR B)

The clauses can also be literals:

  • A OR B
  • A AND B

You cannot use A OR (B AND C) in conjunctive normal form, but you can use (A OR B) AND (A OR C).

The following example would generate an error in UDM Search:

principal.hostname = "win-server" nocase OR (principal.hostname = "win-adfs" nocase AND metadata.event_type = "NETWORK_CONNECTION")

The following is a valid example using conjunctive normal form in a UDM Search:

(principal.hostname = "win-server" nocase OR principal.hostname = "win-adfs" nocase) AND (principal.hostname = "win-server" nocase OR metadata.event_type = "NETWORK_CONNECTION")

You cannot use NOT (A OR B) in conjunctive normal form, but you can use NOT A AND NOT B.

The following is invalid:

principal.hostname = "win-server" nocase AND NOT(metadata.event_type = "PROCESS_TERMINATION" OR metadata.event_type = "USER_RESOURCE_ACCESS")

The following is valid:

principal.hostname = "win-server" nocase AND NOT metadata.event_type = "PROCESS_TERMINATION" AND NOT metadata.event_type = "USER_RESOURCE_ACCESS"

You cannot use A AND (B OR (C AND D)) in conjunctive normal form, but you can use A AND (B OR C) AND (B OR D).

The following is invalid:

principal.hostname = "win-server" nocase AND (metadata.event_type = "PROCESS_LAUNCH" OR (metadata.event_type = "NETWORK_CONNECTION" AND target.ip = "10.128.0.21")

The following is valid:

principal.hostname = "win-server" nocase AND (metadata.event_type = "PROCESS_LAUNCH" OR metadata.event_type = "NETWORK_CONNECTION") AND (metadata.event_type = "PROCESS_LAUNCH" OR target.ip = "10.128.0.21")