Collect Google Cloud IDS logs

This document describes how you can collect Google Cloud IDS logs by enabling Google Cloud telemetry ingestion to Google Security Operations and how log fields of Google Cloud IDS logs map to Google Security Operations Unified Data Model (UDM) fields.

For more information, see Data ingestion to Google Security Operations.

A typical deployment consists of Google Cloud IDS logs enabled for ingestion to Google Security Operations. Each customer deployment might differ from this representation and might be more complex.

The deployment contains the following components:

Google Cloud: The Google Cloud services and products from which you collect logs.
Google Cloud IDS logs: The Google Cloud IDS logs that are enabled for ingestion to Google Security Operations.
Google Security Operations: Google Security Operations retains and analyzes the logs from Google Cloud IDS.

An ingestion label identifies the parser which normalizes raw log data to structured UDM format. The information in this document applies to the parser with the GCP_IDS ingestion label.

Before you begin

Ensure that all systems in the deployment architecture are configured in the UTC time zone.

Configure Google Cloud to ingest Google Cloud IDS logs

To ingest Google Cloud IDS logs to Google Security Operations, follow the steps on the Ingest Google Cloud logs to Google Security Operations page.

If you encounter issues when you ingest Google Cloud IDS logs, contact Google Security Operations support.

Field mapping reference

Field mapping reference: GCP_IDS

The following table lists the log fields of the GCP_IDS log type and their corresponding UDM fields.

Log field	UDM mapping	Logic
`insertId`	`metadata.product_log_id`
`jsonPayload.alert_severity`	`security_result.severity`
`jsonPayload.alert_time`	`metadata.event_timestamp`
`jsonPayload.application`	`principal.application`	If the `jsonPayload.direction` log field value is equal to `server-to-client`, then the `jsonPayload.application` log field is mapped to the `principal.application` UDM field.
`jsonPayload.application`	`target.application`	If the `jsonPayload.direction` log field value is equal to `client-to-server` or the `logName` log field value matches the regular expression pattern `traffic`, then the `jsonPayload.application` log field is mapped to the `target.application` UDM field.
`jsonPayload.category`	`security_result.category_details`
`jsonPayload.cves`	`extensions.vulns.vulnerabilities.cve_id`	If the `jsonPayload.cves` log field value is not empty, then the `jsonPayload.cves` log field is mapped to the `extensions.vulns.vulnerabilities.cve_id` UDM field.
`jsonPayload.destination_ip_address`	`target.ip`
`jsonPayload.destination_port`	`target.port`
`jsonPayload.details`	`extensions.vulns.vulnerabilities.description`	If the `jsonPayload.cves` log field value is not empty, then the `jsonPayload.details` log field is mapped to the `extensions.vulns.vulnerabilities.description` UDM field.
`jsonPayload.direction`	`network.direction`	If the `jsonPayload.direction` log field value is equal to `client-to-server`, then the `network.direction` UDM field is set to `OUTBOUND`. Else, if the `jsonPayload.direction` log field value is equal to `server-to-client`, then the `network.direction` UDM field is set to `INBOUND`.
`jsonPayload.elapsed_time`	`network.session_duration.seconds`
`jsonPayload.ip_protocol`	`network.ip_protocol`	If the `jsonPayload.ip_protocol` log field value contains one of the following values, then the `network.ip_protocol` UDM field is set to `ICMP`. `1` `ICMP` `ICMPV6` `58` `1.0` `58.0` Else, if the `jsonPayload.ip_protocol` log field value contains one of the following values, then the `network.ip_protocol` UDM field is set to `IGMP`. `2` `IGMP` `2.0` Else, if the `jsonPayload.ip_protocol` log field value contains one of the following values, then the `network.ip_protocol` UDM field is set to `TCP`. `6` `TCP` `6.0` Else, if the `jsonPayload.ip_protocol` log field value contains one of the following values, then the `network.ip_protocol` UDM field is set to `UDP`. `17` `UDP` `17.0` Else, if the `jsonPayload.ip_protocol` log field value contains one of the following values, then the `network.ip_protocol` UDM field is set to `IP6IN4`. `41` `IP6IN4` `41.0` Else, if the `jsonPayload.ip_protocol` log field value contains one of the following values, then the `network.ip_protocol` UDM field is set to `GRE`. `47` `GRE` `47.0` Else, if the `jsonPayload.ip_protocol` log field value contains one of the following values, then the `network.ip_protocol` UDM field is set to `ESP`. `50` `ESP` `50.0` Else, if the `jsonPayload.ip_protocol` log field value contains one of the following values, then the `network.ip_protocol` UDM field is set to `EIGRP`. `88` `EIGRP` `88.0` Else, if the `jsonPayload.ip_protocol` log field value contains one of the following values, then the `network.ip_protocol` UDM field is set to `ETHERIP`. `97` `ETHERIP` `97.0` Else, if the `jsonPayload.ip_protocol` log field value contains one of the following values, then the `network.ip_protocol` UDM field is set to `PIM`. `103` `PIM` `103.0` Else, if the `jsonPayload.ip_protocol` log field value contains one of the following values, then the `network.ip_protocol` UDM field is set to `VRRP`. `112` `VRRP` `112.0`
`jsonPayload.name`	`security_result.threat_name`
`jsonPayload.network`	`target.resource.name`	If the `jsonPayload.direction` log field value is equal to `client-to-server` or the `logName` log field value matches the regular expression pattern `traffic`, then the `jsonPayload.network` log field is mapped to the `target.resource.name` UDM field.
`jsonPayload.network`	`principal.resource.name`	If the `jsonPayload.direction` log field value is equal to `server-to-client`, then the `jsonPayload.network` log field is mapped to the `principal.resource.name` UDM field.
	`target.resource.resource_type`	If the `jsonPayload.direction` log field value is equal to `client-to-server` or the `logName` log field value matches the regular expression pattern `traffic`, then the `target.resource.resource_type` UDM field is set to `VPC_NETWORK`.
	`principal.resource.resource_type`	If the `jsonPayload.direction` log field value is equal to `server-to-client`, then the `principal.resource.resource_type` UDM field is set to `VPC_NETWORK`.
`jsonPayload.repeat_count`	`security_result.detection_fields[repeat_count]`
`jsonPayload.session_id`	`network.session_id`
`jsonPayload.source_ip_address`	`principal.ip`
`jsonPayload.source_port`	`principal.port`
`jsonPayload.start_time`	`about.labels[start_time]`
`jsonPayload.threat_id`	`security_result.threat_id`
`jsonPayload.total_bytes`	`about.labels[total_bytes]`
`jsonPayload.total_packets`	`about.labels[total_packets]`
`jsonPayload.type`	`security_result.detection_fields[type]`
`jsonPayload.uri_or_filename`	`target.file.full_path`
`logName`	`security_result.category_details`
`receiveTimestamp`	`metadata.collected_timestamp`
`resource.labels.id`	`observer.resource.product_object_id`
`resource.labels.location`	`observer.location.name`
`resource.labels.resource_container`	`observer.resource.name`
`resource.type`	`observer.resource.resource_subtype`
`timestamp`	`metadata.event_timestamp`	If the `logName` log field value matches the regular expression pattern `traffic`, then the `timestamp` log field is mapped to the `metadata.event_timestamp` UDM field.
	`observer.resource.resource_type`	The `observer.resource.resource_type` UDM field is set to `CLOUD_PROJECT`.
	`observer.resource.attribute.cloud.environment`	The `observer.resource.attribute.cloud.environment` UDM field is set to `GOOGLE_CLOUD_PLATFORM`.
	`idm.is_alert`	If the `jsonPayload.alert_severity` log field value is equal to `CRITICAL`, then the `idm.is_alert` UDM field is set to `true`.
	`idm.is_significant`	If the `jsonPayload.alert_severity` log field value is equal to `CRITICAL`, then the `idm.is_significant` UDM field is set to `true`.
	`security_result.category`	If the `jsonPayload.category` log field value is equal to `dos`, then the `security_result.category` UDM field is set to `NETWORK_DENIAL_OF_SERVICE`. Else, if the `jsonPayload.category` log field value is equal to `info-leak`, then the `security_result.category` UDM field is set to `NETWORK_SUSPICIOUS`. Else, if the `jsonPayload.category` log field value is equal to `protocol-anomaly`, then the `security_result.category` UDM field is set to `NETWORK_MALICIOUS`. Else, if the `jsonPayload.category` log field value contains one of the following values, then the `security_result.category` UDM field is set to `SOFTWARE_MALICIOUS`. `backdoor` `spyware` `trojan`
	`extensions.vulns.vulnerabilities.vendor`	if the `jsonPayload.cves` log field value is not empty, then the `extensions.vulns.vulnerabilities.vendor` UDM field is set to `GCP_IDS`.
	`metadata.product_name`	The `metadata.product_name` UDM field is set to `GCP_IDS`.
	`metadata.vendor_name`	The `metadata.vendor_name` UDM field is set to `Google Cloud Platform`.
	`metadata.event_type`	If the `jsonPayload.cves` log field value is not empty, then the `metadata.event_type` UDM field is set to `SCAN_VULN_NETWROK`. Else, if the `jsonPayload.source_ip_address` log field value is not empty, then the `metadata.event_type` UDM field is set to `SCAN_NETWORK`. Else, the `metadata.event_type` UDM field is set to `GENERIC_EVENT`.

What's next

Data ingestion to Google Security Operations