Running a rule against live data
When you create a rule, it does not initially search for detections based on events received in your Google Security Operations account in real time. However, you set the rule to search for detections in real time by setting the Live Rule toggle to enabled.
To set a rule to live, complete the following steps:
Navigate to the Rules Dashboard.
Click the Rules option icon for a rule and toggle the Live Rule to enabled.
Live Rule
You can view detections generated by a live rule by choosing View Rule Detections.
Rules quota
Click the capacity button to display the limits to the number of rules that can be enabled as live. It is located in the upper right corner of the Rules Dashboard.
Google Security Operations imposes the following rule limits:
- Multiple Events Rules Quota—Shows the current count of Multiple Event rules enabled as live and the maximum number of rules that can be enabled as live. More information about the difference between Single Event and Multiple Event Rules can be found here.
- Total Rules Quota—Shows the current total count of rules enabled as live across all rule types and the maximum number of rules that can be enabled as live.
More information on the different types of rules can be found here.
Rules executions
Live rules executions for a given event time bucket will be triggered with decreasing frequency. There will be a final cleanup run, after which no more execution will be started.
Each execution runs over the latest versions of reference lists used in the rules, as well as against the latest event and entity data enrichment.
This means that some detections can be retrospectively generated if they are detected only by the later executions. For example, the last execution might use the latest version of the reference list, which now detects more events, and events and entity data can be reprocessed due to new enrichments.
Detection latencies
Various factors determine how long it takes for a detection to be generated from a live rule. The following list includes the different factors that contribute to detection delays:
- Rule types
- Run frequency
- Ingestion delay
- Contextual joins
- Enriched UDM data
- Timezone issues
- Reference lists
Rule types
- Single event rules are executed near-real time in a streaming fashion. Use these rules, when possible, to minimize latency.
- Multi-event rules are executed in a scheduled fashion which leads to higher latency due to the time between scheduled executions.
Run frequency
To achieve faster detections, use a shorter run frequency and smaller match window. Using shorter match windows (under one hour) enables more frequent runs.
Ingestion delay
Make sure the data is sent to Google Security Operations as soon as the event occurs. When reviewing a detection, pay close attention to the UDM event and ingestion timestamps.
Contextual joins
Multi-event rules that with contextual data, such as UEBA or Entity Graph, might have higher delays. The contextual data must first be generated by Google SecOps.
Enriched UDM data
Google SecOps enriches events with data from other events. To identify if a rule is evaluating an enriched field, review the Event Viewer. If the rule is evaluating an enriched field, the detection might be delayed.
Timezone issues
Rules execute more frequently for real-time data. Data might arrive in real-time, but Google SecOps might still treat it as late-arriving data if the event time is incorrect due to timezone issues. The Google SecOps SIEM default timezone is UTC. If the original data has an event timestamp set to another timezone besides UTC, update the data timezone. If updating the timezone at the log source is not possible, then reach out to Support, so that the timezone can be overridden.
Non-existence rules
Rules that check for non-existence (for example, rules that contain !$e or #e=0) are executed with at least a one hour delay to ensure that data has time to arrive.
Reference lists
Rule executions always use the most up-to-date version of a reference list. If the reference list was recently updated, a new detection might appear late because the detection might be included with new contents of the updated list during later executions of the scheduled rule.
To achieve lower detection latencies, we recommend doing the following:
- Send log data to Google Security Operations as soon as the event occurs.
- Audit rules to see if it is necessary to use non-existence or context-enriched data.
- Configure a smaller run frequency.
Rule status
Live rules can have one of the following statuses:
Enabled: Rule is active and working normally as a live rule.
Disabled: Rule is disabled.
Limited: Live rules can be placed in this status when they exhibit abnormally high resource usage. Limited rules are isolated from the other live rules in the system to maintain the stability of Google Security Operations.
For Limited live rules, successful rule executions are not guaranteed. However, if the rule execution succeeds, detections are retained and available for you to review. Limited live rules always generate an error message, which includes information about how to improve the performance of the rule.
If the performance of a Limited rule does not improve within 3 days, its status is changed to Paused.
Paused: Live rules enter this status when they have been in Limited status for 3 days and haven't shown any performance improvement. Executions for this rule have paused and error messages containing information on how to improve the performance of the rule are returned.
To return any live rule to the Enabled status, follow YARA-L best practices to improve the performance of your rule and save it. After the rule has been saved, it will be reset to the Enabled state and it will take at least an hour before the rule will reach the Limited status again.
You can potentially resolve the performance issues with a rule by configuring it to run less frequently. For example, you could reconfigure a rule from running every 10 minutes to running once an hour or once every 24 hours. However, changing the execution frequency of a rule won't change its status back to Enabled. If you make a small modification to the rule and save it, you can automatically reset its status Enabled.
Rule statuses are displayed in the Rules Dashboard and are also accessible through the Detection Engine API. Errors generated by rules in the Limited or Paused status are available using the ListErrors API method. The error will state that the rule is in the Limited or Paused status and directs you to documentation on how to resolve the issue.