Using Cloud Monitoring for ingestion notifications

This document describes how to use Cloud Monitoring to receive ingestion notifications. Google Security Operations uses Cloud Monitoring to send the ingestion notifications. Using this feature, you can proactively address the issues. You can integrate email notifications in the existing workflows. Notifications are triggered when the ingestion values reach certain predefined levels. In the Cloud Monitoring documentation, notifications are referred to as alerts.

Before you begin

• Make sure you are familiar with Cloud Monitoring.

• Configure a Google Cloud project for Google Security Operations.

• Make sure that your Identity and Access Management role includes the permissions in the role roles/monitoring.alertPolicyEditor. For more information about roles, see Access control.

• Make sure you are familiar with creating alerting policies in Cloud Monitoring. For information about these steps, see Create alerts.

• Configure the notification channel as email to receive ingestion notifications. For information about these steps, see Manage notification channels.

Set up ingestion notification for health metrics

To set up notifications that monitor ingestion health metrics specific to Google Security Operations, do the following:

1. In the Google Cloud console, select Monitoring.

2. In the navigation pane, select Alerting and then click Create policy.

3. On the Select a metric page, do any of the following:

   • Select Chronicle Collector > Ingestion and then select either Total ingested log count or Total ingested log size.

   • Select Chronicle Collector > Normalizer and then select either Total record count or Total event count.

   • Select Chronicle Log Type > Outofband and then select either Total ingested log count (Feeds) or Total ingested log size (Feeds).

4. Click Apply.

5. To add a filter, on the Select a metric page, click Add Filter. In the filter dialog, select the collector_id label, a comparator, and then the filter value.

   • Select one or more of the following filters:

     • project_id: The identifier of the Google Cloud project associated with this resource.

     • location: The physical location of the cluster that contains the collector object.

     • collector_id: The ID of the collector.

     • log_type: The name of the log type.

     • Metric label > namespace: The namespace of the log.

     • Feed_name: The name of the feed.

     • LogType: The type of log.

     • Metric label > event_type: The event type determines which fields are included with the event. The event type includes values such as PROCESS_OPEN, FILE_CREATION, USER_CREATION, and NETWORK_DNS.

     • Metric label > state: The final status of the event or log. The status is one of the following:

       • parsed. The log is successfully parsed.
       • validated. The log is successfully validated.
       • failed_parsing. The log has parsing errors.
       • failed_validation. The log has validation errors.

     • Metric label > drop_reason_code: This field is populated if the ingestion source is the Google Security Operations forwarder and indicates the reason why a log was dropped during normalization.

     • Metric label > ingestion_source: The ingestion source present in the ingestion label when the logs are ingested using the ingestion API.

   • Select a special collector ID. Collector ID can also be a forwarder ID or a special ID based on the ingestion method.

     • aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa: represents all feeds created using the Feed Management API or page. For more information about feed management, see Feed management and Feed management API.

     • bbbbbbbb-bbbb-bbbb-bbbb-bbbbbbbbbbbb: represents all ingestion sources that use the Ingestion API unstructuredlogentries method. For more information about ingestion API, see Google Security Operations Ingestion API.

     • cccccccc-cccc-cccc-cccc-cccccccccccc: represents all ingestion sources that use the Ingestion API udmevents method.

     • dddddddd-dddd-dddd-dddd-dddddddddddd: represents Google Cloud log ingestion.

     • eeeeeeee-eeee-eeee-eeee-eeeeeeeeeeee: represents the collector ID used for CreateEntities.

6. In the Transform data section, do the following:

   1. Set the Time series aggregation field to sum.
   2. Set the Time series group by field to project_id.

7. Optional: Set up an alert policy with multiple conditions. To create ingestion notifications with multiple conditions within an alert policy, see Policies with multiple conditions.

Google Security Operations forwarder metrics and associated filters

The following table describes the available Google Security Operations forwarder metrics and the associated filters.

Google Security Operations forwarder metric	Filter
Container memory used	log_type, collector_id
Container disk used	log_type, collector_id
Container cpu_used	log_type, collector_id
Log drop_count	log_type, collector_id, input_type, reason
buffer_used	log_type, collector_id, buffer_type, input_type
last_heartbeat	log_type, collector_id, input_type

Set up a sample policy to detect silent Google Security Operations forwarders

The following sample policy detects all the Google Security Operations forwarders and sends alerts if the Google Security Operations forwarders do not send logs for 60 minutes. This may not be useful for all the Google Security Operations forwarders which you want to monitor. For example, you can monitor a single log source across one or many Google Security Operations forwarders with a different threshold or exclude Google Security Operations forwarders based upon their frequency of reporting.

1. In the Google Cloud console, select Monitoring.
   Go to Google Security Operations
   

2. Click Create Policy.

3. On the Select a metric page, select Chronicle Collector > Ingestion > Total ingested log count.

4. Click Apply.

5. In the Transform data section, do the following:

   1. Set the Rolling window to 1 hour.
   2. Set the Rolling window function to mean.
   3. Set the Time series aggregation to mean.
   4. Set the Time series group by to collector_id. If this is not set to group by collector_id, then an alert is triggered for each log source.

6. Click Next.

7. Select Metric absence and do the following:

   1. Set Alert trigger to Any time series violates.
   2. Set Trigger absence time to 1 hour.
   3. Enter a name for the condition and then click Next.

8. In the Notifications and name section, do the following:

   1. Select a notification channel in the Use notification channel box. We recommend that you configure multiple notification channels for redundancy purposes.
   2. Configure notifications on incident closure.
   3. Set policy user labels to an appropriate level. This is used for setting the severity level of the alert for a policy.
   4. Enter any documentation that you would like to be sent as part of the alert.
   5. Enter a name for the alert policy.

Add exclusions to a catch-all policy

It may be necessary to exclude certain Google Security Operations forwarders from a catch-all policy because they may just have low traffic volumes, or require a more custom alert policy.

1. In the Google Cloud console, select Monitoring.

2. In the navigation page, select Alerting and then in the Policies section select the policy you want to edit.

3. On the Policy details page, click Edit.

4. On the Edit alerting policy page, under the Add filters section, select Add a filter and do the following:

   1. Select the collector_id label and the collector you want to exclude from the policy.
   2. Set the comparator to != and the value to the collector_id you want to exclude, and then click Done.
   3. Repeat for each collector that needs to be excluded. You can also use a regular expression to exclude multiple collectors with only a single filter if you want to use the following format:

   (?:aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa|bbbbbbbb-bbbb-bbbb-bbbb-bbbbbbbbbbbb|cccccccc-cccc-cccc-cccc-cccccccccccc)

5. Click Save Policy.