Collect Cloud Compute logs

Supported in:

Google SecOps SIEM

Note: This feature is covered by Pre-GA Offerings Terms of the Google Security Operations Service Specific Terms. Pre-GA features might have limited support, and changes to pre-GA features might not be compatible with other pre-GA versions. For more information, see the Google Security Operations Technical Support Service guidelines and the Google Security Operations Service Specific Terms.

This document explains how to configure Google Cloud Compute logs export to Google Security Operations using Cloud Storage. The parser extracts fields, normalizes the message field, and maps the extracted data to the Unified Data Model (UDM) schema for consistent security event representation. It handles various log formats, including syslog-like messages and key-value pairs, and categorizes events based on extracted fields like type and action.

Before You Begin

• Ensure that you have a Google SecOps instance.
• Ensure that Compute is set up and active in your Google Cloud environment.
• Ensure that you have privileged access to Google Cloud.

Create a Google Cloud Storage Bucket

1. Sign in to the Google Cloud console.

2. Go to the Cloud Storage Buckets page.

   Go to Buckets

3. Click Create.

4. On the Create a bucket page, enter your bucket information. After each of the following steps, click Continue to proceed to the next step:

   1. In the Get started section, do the following:

      1. Enter a unique name that meets the bucket name requirements; for example, compute-logs.

      2. To enable hierarchical namespace, click the expander arrow to expand the Optimize for file oriented and data-intensive workloads section, and then select Enable Hierarchical namespace on this bucket.

         Note: You cannot enable hierarchical namespace in an existing bucket.
         1. To add a bucket label, click the expander arrow to expand the Labels section.
         2. Click Add label, and specify a key and a value for your label.

   2. In the Choose where to store your data section, do the following:

      1. Select a Location type.

         1. Use the location type menu to select a Location where object data within your bucket will be permanently stored.
         Note: If you select the dual-region location type, you can also choose to enable turbo replication by using the relevant checkbox.
         1. To set up cross-bucket replication, expand the Set up cross-bucket replication section.

   3. In the Choose a storage class for your data section, either select a default storage class for the bucket, or select Autoclass for automatic storage class management of your bucket's data.

   4. In the Choose how to control access to objects section, select not to enforce public access prevention, and select an access control model for your bucket's objects.

      Note: If public access prevention is already enforced by your project's organization policy, the Prevent public access checkbox is locked.

   5. In the Choose how to protect object data section, do the following:

      1. Select any of the options under Data protection that you want to set for your bucket.
      2. To choose how your object data will be encrypted, click the expander arrow labeled Data encryption, and select a Data encryption method.

5. Click Create.

Note: Be sure to provide your Google SecOps Service Account with permissions to Read or Read & Write to the newly created bucket.

Configure Google Cloud Compute Logs Export

1. In the Google Cloud Console, go to Logging > Log Router.
2. Click Create Sink.

3. Provide the following details:

   • Sink Name: Provide a meaningful name; for example, Compute-Logs-Sink.
   • Sink Destination: Select Cloud Storage .
   • Sink Destination: Select Cloud Storage and enter the bucket URI; for example, gs://<your-bucket-name>/compute-logs.

   • Log Filter: Set filters to capture Google Cloud Compute logs as follows:

     • Log name and type:

       logName="*compute*"
       

     • Network-related fields (like IP addresses, ports):

       jsonPayload.connection.dest_ip="*" OR jsonPayload.connection.src_ip="*"
       

     • Instance details:

       jsonPayload.dest_instance.project_id="*"
       jsonPayload.src_instance.project_id="*"
       

     • Security-related details:

       jsonPayload.rule_details.action="ALLOW" OR jsonPayload.rule_details.action="BLOCK"
       

4. Click Create.

Configure Permissions for Cloud Storage

1. Go to IAM > IAM & Admin > Service Accounts.
2. Find the Cloud Logging service account; for example, [e-mail].
3. Provide it with roles/storage.admin role on the bucket.

Configure a feed in Google SecOps to ingest Google Cloud Compute logs

1. Go to SIEM Settings > Feeds.
2. Click Add new.
3. In the Feed name field, enter a name for the feed; for example, Google Cloud Compute Logs.
4. Select Google Cloud Storage as the Source type.
5. Select GCP Compute as the Log type.
6. Click Get Service Account as the Chronicle Service Account.
7. Click Next.

8. Specify values for the following input parameters:

   • Storage Bucket URI: Google Cloud storage bucket URL in gs://my-bucket/<value> format.
   • URI Is A: Select Directory which includes subdirectories.

   • Source deletion options: select deletion option according to your preference.

     Note: If you select the Delete transferred files or Delete transferred files and empty directories option, make sure that you granted appropriate permissions to the service account.

   • Asset namespace: the asset namespace.

   • Ingestion labels: the label applied to the events from this feed.

9. Click Next.

10. Review your new feed configuration in the Finalize screen, and then click Submit.

UDM Mapping Table

Log field	UDM mapping	Logic
addr	read_only_udm.principal.ip	Merged into the principal IP address list if the field is not empty or "?".
jsonPayload.connection.dest_ip	read_only_udm.target.ip	Merged into the target IP address list if the field exists.
jsonPayload.connection.dest_port	read_only_udm.target.port	Converted to string, then to integer and mapped if no errors occur during conversion.
jsonPayload.connection.protocol	read_only_udm.network.ip_protocol	Converted to string, then to integer. Used to determine the IP protocol (TCP, UDP, etc.) using a lookup table and mapped if no errors occur during conversion.
jsonPayload.connection.src_ip	read_only_udm.principal.ip	Merged into the principal IP address list if the field exists.
jsonPayload.connection.src_port	read_only_udm.principal.port	Converted to string, then to integer and mapped if no errors occur during conversion.
jsonPayload.dest_instance.project_id	read_only_udm.target.resource.product_object_id	Conditionally mapped if jsonPayload.dest_vpc.project_id exists.
jsonPayload.dest_instance.region	read_only_udm.target.location.name	Conditionally mapped if jsonPayload.dest_vpc.project_id exists.
jsonPayload.dest_instance.vm_name	read_only_udm.target.resource.attribute.cloud.project.name	Conditionally mapped if jsonPayload.dest_vpc.project_id exists.
jsonPayload.dest_instance.zone	read_only_udm.target.resource.attribute.cloud.availability_zone	Conditionally mapped if jsonPayload.dest_vpc.project_id exists.
jsonPayload.dest_vpc.project_id	read_only_udm.target.cloud.vpc.product_object_id	Used as a condition to map related fields.
jsonPayload.dest_vpc.subnetwork_name	read_only_udm.target.cloud.vpc.name	Conditionally mapped if jsonPayload.dest_vpc.project_id exists.
jsonPayload.instance.project_id	read_only_udm.target.resource.product_object_id	Conditionally mapped if jsonPayload.instance.project_id exists.
jsonPayload.instance.region	read_only_udm.target.location.name	Conditionally mapped if jsonPayload.instance.project_id exists.
jsonPayload.instance.vm_name	read_only_udm.target.resource.attribute.cloud.project.name	Conditionally mapped if jsonPayload.instance.project_id exists.
jsonPayload.instance.zone	read_only_udm.target.resource.attribute.cloud.availability_zone	Conditionally mapped if jsonPayload.instance.project_id exists.
jsonPayload.message	read_only_udm.metadata.product_event_type, read_only_udm.principal.application, read_only_udm.target.process.pid, read_only_udm.target.user.userid, read_only_udm.principal.hostname, read_only_udm.target.process.command_line, read_only_udm.security_result.description, read_only_udm.principal.process.file.full_path	Parsed and mapped to different fields based on grok patterns and conditional logic.
jsonPayload.rule_details.action	read_only_udm.security_result.action	Used to determine the security result action (ALLOW/BLOCK) and mapped.
jsonPayload.rule_details.direction	read_only_udm.network.direction	Used to determine the network direction (INBOUND/OUTBOUND/UNKNOWN_DIRECTION) and mapped.
jsonPayload.rule_details.priority	read_only_udm.security_result.priority_details	Converted to string and mapped if no errors occur during conversion.
jsonPayload.rule_details.reference	read_only_udm.security_result.rule_labels.value	Mapped to the rule label value.
jsonPayload.src_instance.project_id	read_only_udm.principal.resource.product_object_id	Conditionally mapped if jsonPayload.src_vpc.project_id exists.
jsonPayload.src_instance.region	read_only_udm.principal.location.name	Conditionally mapped if jsonPayload.src_vpc.project_id exists.
jsonPayload.src_instance.vm_name	read_only_udm.principal.resource.attribute.cloud.project.name	Conditionally mapped if jsonPayload.src_vpc.project_id exists.
jsonPayload.src_instance.zone	read_only_udm.principal.resource.attribute.cloud.availability_zone	Conditionally mapped if jsonPayload.src_vpc.project_id exists.
jsonPayload.src_vpc.project_id	read_only_udm.principal.cloud.vpc.product_object_id	Used as a condition to map related fields.
jsonPayload.src_vpc.subnetwork_name	read_only_udm.principal.cloud.vpc.name	Conditionally mapped if jsonPayload.src_vpc.project_id exists.
jsonPayload.vpc.project_id	read_only_udm.target.cloud.vpc.product_object_id	Conditionally mapped if jsonPayload.vpc.project_id exists.
jsonPayload.vpc.subnetwork_name	read_only_udm.target.cloud.vpc.name	Conditionally mapped if jsonPayload.vpc.project_id exists.
logName	read_only_udm.security_result.category_details	Mapped directly.
resource.labels.instance_id	read_only_udm.principal.resource.product_object_id, read_only_udm.principal.asset_id	Conditionally mapped. If type is "PROCTITLE", it's used to construct the asset ID.
resource.labels.location	read_only_udm.principal.location.name	Conditionally mapped if the field exists.
resource.labels.project_id	read_only_udm.metadata.product_deployment_id	Conditionally mapped if the field exists.
resource.labels.zone	read_only_udm.principal.resource.attribute.cloud.availability_zone	Conditionally mapped if the field exists.
resource.type	read_only_udm.metadata.event_type	Used to determine the event type and mapped.
timestamp	read_only_udm.metadata.event_timestamp	Mapped directly.
type	read_only_udm.metadata.product_event_type, read_only_udm.metadata.event_type, read_only_udm.extensions.auth.type	Used to determine the event type, product event type, and authentication type and mapped accordingly.
	read_only_udm.metadata.event_type	The logic sets the event type based on the "type" field and other conditions. If no specific match is found, it defaults to "GENERIC_EVENT".
	read_only_udm.metadata.log_type	Constant value "GCP_COMPUTE".
	read_only_udm.metadata.vendor_name	Constant value "Google Cloud Platform".
	read_only_udm.metadata.product_name	Constant value "Google Cloud Platform".
	read_only_udm.security_result.rule_labels.key	Constant value "Reference".
	read_only_udm.target.cloud.vpc.resource_type	Conditionally set to "VPC_NETWORK" if jsonPayload.instance.project_id or jsonPayload.dest_vpc.project_id exists.
	read_only_udm.target.resource.attribute.cloud.environment	Conditionally set to "GOOGLE_CLOUD_PLATFORM" if jsonPayload.instance.project_id, jsonPayload.dest_vpc.project_id, or jsonPayload.src_vpc.project_id exists.
	read_only_udm.principal.administrative_domain	Mapped from the "Account Domain" field extracted from the "kv_data" field.
	read_only_udm.principal.user.user_display_name	Mapped from the "Account Name" field extracted from the "kv_data" field.
	read_only_udm.target.resource.name	Mapped from the "Object Name" field extracted from the "kv_data" field.
	read_only_udm.target.resource.type	Mapped from the "Object Type" field extracted from the "kv_data" field.
	read_only_udm.principal.process.pid	Mapped from the "Process ID" field extracted from the "kv_data" field.
	read_only_udm.target.user.windows_sid	Mapped from the "Security ID" field extracted from the "kv_data" field.
	read_only_udm.network.session_id	Mapped from the "auid" field.

Changes

2024-06-18

Enhancement:

• Mapped "file" to "principal.file.names".
• Mapped "function" to "principal.resource.attribute.labels".
• Mapped "line" to "principal.resource.attribute.labels".
• Mapped "timestamp" to "event_timestamp".

2023-02-24

Bug Fix:

• Added mapping for "asset_id" to facilitate search in UI: "asset:resource.labels.instance_id" is mapped to "principal.asset_id"

2022-06-16

Enhancement:

• Added mapping for the following new fields: jsonPayload.Message as syslog.
• Process Name to principal.application.
• Process ID to principal.process.pid.
• Account Domain to principal.administrative_domain.
• Account Name to principal.user.user_display_name.
• Object Name to target.resource.name.
• Object Type to target.resource.type.
• Security ID to target.user.windows_sid.
• addr to principal.ip.
• auid to network.session_id.
• Mapped "LINUX - %{type}" for linux logs and "WINDOWS event log" for windows log to metadata.product_event_type.
• pid to target.process.pid.
• acct to target.user.userid.
• exe to target.process.command_line.
• file_path to principal.process.file.full_path.
• Changed mapping of desc from metadata.description to security_result.description.

2022-05-23

Enhancement:

• Added mapping for the following new fields:
• jsonPayload.message as syslog.
• resource.labels.zone to principal.resource.attribute.cloud.availability_zone.
• resource.labels.location to principal.location.name.
• resource.labels.project_id to metadata.product_deployment_id.
• resource.labels.instance_id to principal.resource.product_object_id.

Need more help? Get answers from Community members and Google SecOps professionals.