Running a rule against live data

When you create a rule, it does not initially search for detections based on events received in your Google Security Operations account in real time. However, you set the rule to search for detections in real time by setting the Live Rule toggle to enabled.

To set a rule to live, complete the following steps:

  1. Navigate to the Rules Dashboard.

  2. Click the Rules option icon for a rule and toggle the Live Rule to enabled.

    Live rule

    Live Rule

  3. You can view detections generated by a live rule by choosing View Rule Detections.

Rules quota

Click the capacity button to display the limits to the number of rules that can be enabled as live. It is located in the upper right corner of the Rules Dashboard.

Google Security Operations imposes the following rule limits:

  • Multiple Events Rules Quota—Shows the current count of Multiple Event rules enabled as live and the maximum number of rules that can be enabled as live. More information about the difference between Single Event and Multiple Event Rules can be found here.
  • Total Rules Quota—Shows the current total count of rules enabled as live across all rule types and the maximum number of rules that can be enabled as live.

More information on the different types of rules can be found here.

Rules executions

Live rules executions for a given event time bucket will be triggered with decreasing frequency. There will be a final cleanup run, after which no more execution will be started.

Each execution runs over the latest versions of reference lists used in the rules, as well as against the latest event and entity data enrichment.

This means that some detections can be retrospectively generated if they are detected only by the later executions. For example, the last execution might use the latest version of the reference list, which now detects more events, and events and entity data can be reprocessed due to new enrichments.

Detection latencies

How long it takes for a detection to be generated from a live rule is determined by various factors. For example:

  • The ingestion time of the original log data.
  • Whether the rule uses any context-enriched data. Detections might be delayed due to the enrichments.
  • Whether the rule is non-existence. For non-existence rules (rules that contain !$e or #e = 0 in the condition section), the detection engine adds at least 1 hour delay to the expected latency (based on the rule's run frequency) to allow for late arriving data.

To achieve lower detection latencies, we recommend doing the following:

  • Send log data to Google Security Operations as soon as the event occurs.
  • Audit rules to see if it is necessary to use non-existence or context-enriched data.
  • Configure a smaller run frequency.

Rule status

Live rules can have one of the following statuses:

  • Enabled: Rule is active and working normally as a live rule.

  • Disabled: Rule is disabled.

  • Limited: Live rules can be placed in this status when they exhibit abnormally high resource usage. Limited rules are isolated from the other live rules in the system to maintain the stability of Google Security Operations.

    For Limited live rules, successful rule executions are not guaranteed. However, if the rule execution succeeds, detections are retained and available for you to review. Limited live rules always generate an error message, which includes information about how to improve the performance of the rule.

    If the performance of a Limited rule does not improve within 3 days, its status is changed to Paused.

  • Paused: Live rules enter this status when they have been in Limited status for 3 days and haven't shown any performance improvement. Executions for this rule have paused and error messages containing information on how to improve the performance of the rule are returned.

To return any live rule to the Enabled status, follow YARA-L best practices to improve the performance of your rule and save it. After the rule has been saved, it will be reset to the Enabled state and it will take at least an hour before the rule will reach the Limited status again.

You can potentially resolve the performance issues with a rule by configuring it to run less frequently. For example, you could reconfigure a rule from running every 10 minutes to running once an hour or once every 24 hours. However, changing the execution frequency of a rule won't change its status back to Enabled. If you make a small modification to the rule and save it, you can automatically reset its status Enabled.

Rule statuses are displayed in the Rules Dashboard and are also accessible through the Detection Engine API. Errors generated by rules in the Limited or Paused status are available using the ListErrors API method. The error will state that the rule is in the Limited or Paused status and directs you to documentation on how to resolve the issue.