Using Cloud Monitoring for ingestion notifications

Supported in:

Google SecOps SIEM

This document describes how to use Cloud Monitoring to receive ingestion notifications. Google SecOps uses Cloud Monitoring to send the ingestion notifications. Using this feature, you can proactively address the issues. You can integrate email notifications in the existing workflows. Notifications are triggered when the ingestion values reach certain predefined levels. In the Cloud Monitoring documentation, notifications are referred to as alerts.

Before you begin

• Make sure you are familiar with Cloud Monitoring.

• Configure a Google Cloud project for Google SecOps.

• Make sure that your Identity and Access Management role includes the permissions in the role roles/monitoring.alertPolicyEditor. For more information about roles, see Access control.

• Make sure you are familiar with creating alerting policies in Cloud Monitoring. For information about these steps, see Create alerts.

• Configure the notification channel as email to receive ingestion notifications. For information about these steps, see Manage notification channels.

Set up ingestion notification for health metrics

To set up notifications that monitor ingestion health metrics specific to Google SecOps, do the following:

1. In the Google Cloud console, select Monitoring.

2. In the navigation pane, select Alerting and then click Create policy.

3. On the Select a metric page, click Select a metric.

4. In the Select a metric menu, click any of the following:

   • Active toggle to filter and display only resource and metrics with data from the last 25 hours. If you don't select this, all resource and metric types are listed.
   • Org/folder level toggle to monitor resources and metrics, such as consumer quota usage or BigQuery slot allocation, for your organization and folders.

5. Select any of the following metrics:

   • Select Chronicle Collector > Ingestion and then select either Total ingested log count or Total ingested log size.

   • Select Chronicle Collector > Normalizer and then select either Total record count or Total event count.

   • Select Chronicle Log Type > Outofband and then select either Total ingested log count (Feeds) or Total ingested log size (Feeds).

6. Click Apply.

7. To add a filter, on the Select a metric page, click Add Filter. In the filter dialog, select the collector_id label, a comparator, and then the filter value.

   • Select one or more of the following filters:

     • project_id: The identifier of the Google Cloud project associated with this resource.

     • location: The physical location of the cluster that contains the collector object. We recommend that you don't use this field. If you leave this field empty, then Google Security Operations can use information that it already has to automatically determine where to store the data.

     • collector_id: The ID of the collector.

     • log_type: The name of the log type.

     • Metric label > namespace: The namespace of the log.

     • Feed_name: The name of the feed.

     • LogType: The type of log.

     • Metric label > event_type: The event type determines which fields are included with the event. The event type includes values such as PROCESS_OPEN, FILE_CREATION, USER_CREATION, and NETWORK_DNS.

     • Metric label > state: The final status of the event or log. The status is one of the following:

       • parsed. The log is successfully parsed.
       • validated. The log is successfully validated.
       • failed_parsing. The log has parsing errors.
       • failed_validation. The log has validation errors.
       • failed_indexing. The log has batch indexing errors.

     • Metric label > drop_reason_code: This field is populated if the ingestion source is the Google SecOps forwarder and indicates the reason why a log was dropped during normalization.

     • Metric label > ingestion_source: The ingestion source present in the ingestion label when the logs are ingested using the ingestion API.

   • Select a special collector ID. Collector ID can also be a forwarder ID or a special ID based on the ingestion method.

     • aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa: represents all feeds created using the Feed Management API or page. For more information about feed management, see Feed management and Feed management API.

     • bbbbbbbb-bbbb-bbbb-bbbb-bbbbbbbbbbbb: represents all ingestion sources that use the Ingestion API unstructuredlogentries method. For more information about ingestion API, see Google SecOps Ingestion API.

     • cccccccc-cccc-cccc-cccc-cccccccccccc: represents all ingestion sources that use the Ingestion API udmevents method.

     • dddddddd-dddd-dddd-dddd-dddddddddddd: represents Google Cloud log ingestion.

     • eeeeeeee-eeee-eeee-eeee-eeeeeeeeeeee: represents the collector ID used for CreateEntities.

     • aaaa1111-aaaa-1111-aaaa-1111aaaa1111: represents collection agent.

8. In the Transform data section, do the following:

   1. Set the Time series aggregation field to sum.
   2. Set the Time series group by field to project_id.

9. Optional: Set up an alert policy with multiple conditions. To create ingestion notifications with multiple conditions within an alert policy, see Policies with multiple conditions.

Google SecOps forwarder metrics and associated filters

The following table describes the available Google SecOps forwarder metrics and the associated filters.

Google SecOps forwarder metric	Filter
Container memory used	log_type, collector_id
Container disk used	log_type, collector_id
Container cpu_used	log_type, collector_id
Log drop_count	log_type, collector_id, input_type, reason
buffer_used	log_type, collector_id, buffer_type, input_type
last_heartbeat	log_type, collector_id, input_type

Set up a sample policy to detect silent Google SecOps forwarders

The following sample policy detects all the Google SecOps forwarders and sends alerts if the Google SecOps forwarders don't send logs for 60 minutes. This may not be useful for all the Google SecOps forwarders which you want to monitor. For example, you can monitor a single log source across one or many Google SecOps forwarders with a different threshold or exclude Google SecOps forwarders based upon their frequency of reporting.

1. In the Google Cloud console, select Monitoring.
   Go to Cloud Monitoring
   

2. Click Create Policy.

3. On the Select a metric page, select Chronicle Collector > Ingestion > Total ingested log count.

4. Click Apply.

5. In the Transform data section, do the following:

   1. Set the Rolling window to 1 hour.
   2. Set the Rolling window function to mean.
   3. Set the Time series aggregation to mean.
   4. Set the Time series group by to collector_id. If this is not set to group by collector_id, then an alert is triggered for each log source.

6. Click Next.

7. Select Metric absence and do the following:

   1. Set Alert trigger to Any time series violates.
   2. Set Trigger absence time to 1 hour.
   3. Enter a name for the condition and then click Next.

8. In the Notifications and name section, do the following:

   1. Select a notification channel in the Use notification channel box. We recommend that you configure multiple notification channels for redundancy purposes.
   2. Configure notifications on incident closure.
   3. Set policy user labels to an appropriate level. This is used for setting the severity level of the alert for a policy.
   4. Enter any documentation that you would like to be sent as part of the alert.
   5. Enter a name for the alert policy.

Add exclusions to a catch-all policy

It may be necessary to exclude certain Google SecOps forwarders from a catch-all policy because they may just have low traffic volumes, or require a more custom alert policy.

1. In the Google Cloud console, select Monitoring.

2. In the navigation page, select Alerting and then in the Policies section select the policy you want to edit.

3. On the Policy details page, click Edit.

4. On the Edit alerting policy page, under the Add filters section, select Add a filter and do the following:

   1. Select the collector_id label and the collector you want to exclude from the policy.
   2. Set the comparator to != and the value to the collector_id you want to exclude, and then click Done.
   3. Repeat for each collector that needs to be excluded. You can also use a regular expression to exclude multiple collectors with only a single filter if you want to use the following format:

   (?:aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa|bbbbbbbb-bbbb-bbbb-bbbb-bbbbbbbbbbbb|cccccccc-cccc-cccc-cccc-cccccccccccc)

5. Click Save Policy.

Set up a sample policy to detect silent Google SecOps collection agents

The following sample policy detects all the Google SecOps collection agents and sends alerts if the Google SecOps collection agents don't send logs for 60 minutes. This sample might not be useful for all the Google SecOps collection agents which you want to monitor. For example, you can monitor a single log source across one or many Google SecOps collection agents with a different threshold or exclude Google SecOps collection agents based upon their frequency of reporting.

1. In the Google Cloud console, select Monitoring.
   Go to Cloud Monitoring
   

2. Click Create Policy.

3. On the Select a metric page, select Chronicle Collector > Agent > Exporter Accepted Spans Count.

4. Click Apply.

5. In the Transform data section, do the following:

   1. Set the Rolling window to 1 hour.
   2. Set the Rolling window function to mean.
   3. Set the Time series aggregation to mean.
   4. Set the Time series group by to collector_id. If this is not set to group by collector_id, then an alert is triggered for each log source.

6. Click Next.

7. Select Metric absence and do the following:

   1. Set Alert trigger to Any time series violates.
   2. Set Trigger absence time to 1 hour.
   3. Enter a name for the condition and then click Next.

8. In the Notifications and name section, do the following:

   1. Select a notification channel in the Use notification channel box. We recommend that you configure multiple notification channels for redundancy purposes.
   2. Configure notifications on incident closure.
   3. Set policy user labels to an appropriate level. This is used for setting the severity level of the alert for a policy.
   4. Enter any documentation that you would like to be sent as part of the alert.
   5. Enter a name for the alert policy.