Collect Cloud VPC Flow Logs

Supported in:

This document explains how to export Cloud VPC Flow Logs to Google Security Operations using Google Cloud. The parser transforms the logs from their built-in JSON format into the Google Security Operations UDM. It extracts relevant fields like source and destination IP, port, protocol, and bytes sent, then maps them to corresponding UDM fields, taking into account network direction and special cases for accurate representation in Google SecOps.

Before You Begin

  • Ensure that you have a Google SecOps instance.
  • Ensure that VPC Flow is set up and active in your Google Cloud environment.
  • Ensure that you have privileged access to Google Cloud.

Create a Google Cloud Storage Bucket

  1. Sign in to the Google Cloud console.
  2. Go to the Cloud Storage Buckets page.

    Go to Buckets

  3. Click Create.

  4. On the Create a bucket page, enter your bucket information. After each of the following steps, click Continue to proceed to the next step:

    1. In the Get started section, do the following:

      1. Enter a unique name that meets the bucket name requirements; for example, vpcflow-logs.
      2. To enable hierarchical namespace, click the expander arrow to expand the Optimize for file oriented and data-intensive workloads section, and then select Enable Hierarchical namespace on this bucket.

      3. To add a bucket label, click the expander arrow to expand the Labels section.

      4. Click Add label, and specify a key and a value for your label.

    2. In the Choose where to store your data section, do the following:

      1. Select a Location type.
      2. Use the location type menu to select a Location where object data within your bucket will be permanently stored.

      3. To set up cross-bucket replication, expand the Set up cross-bucket replication section.

    3. In the Choose a storage class for your data section, either select a default storage class for the bucket, or select Autoclass for automatic storage class management of your bucket's data.

    4. In the Choose how to control access to objects section, select not to enforce public access prevention, and select an access control model for your bucket's objects.

    5. In the Choose how to protect object data section, do the following:

      1. Select any of the options under Data protection that you want to set for your bucket.
      2. To choose how your object data will be encrypted, click the expander arrow labeled Data encryption, and select a Data encryption method.
  5. Click Create.

Configure Log Export in Google Cloud VPC Flow

  1. Sign in to Google Cloud account using your privileged account.
  2. On Welcome page, click VPC Networks.
  3. Click Default and a subnet page should appear.
  4. Select all logs.
  5. Click Flow Logs > Configure.
  6. Select Aggregation Interval; for example, 30 SEC.
  7. Provide Sample Rate; for example, 50%.
  8. Click Save
  9. Search Logging in the search bar and click Enter.
  10. In Log Explorer, filter the logs by choosing VPC_flows in the Log Name and click Apply.
  11. Click More Actions.
  12. Click Create Sink.
  13. Provide the following configurations:
    1. Sink Details: enter a name and description.
    2. Click Next.
    3. Sink Destination: select Cloud Storage Bucket.
    4. Cloud Storage Bucket: select the bucket created earlier or create a new bucket.
    5. Click Next.
    6. Choose Logs to include in Sink: a default log is populated when you select an option in Cloud Storage Bucket.
    7. Click Next.
    8. Optional: Choose Logs to filter out of Sink: select the logs that you would like not to sink.
  14. Click Create Sink.

Configure a feed in Google SecOps to ingest Google Cloud VPC Flow Logs

  1. Go to SIEM Settings > Feeds.
  2. Click Add new.
  3. In the Feed name field, enter a name for the feed; for example, Google Cloud VPC Flow Logs.
  4. Select Google Cloud Storage as the Source type.
  5. Select GCP VPC Flow as the Log type.
  6. Click Get Service Account as the Chronicle Service Account.
  7. Click Next.
  8. Specify values for the following input parameters:

    • Storage Bucket URI: Google Cloud storage bucket URL in gs://my-bucket/<value> format.
    • URI Is A: Select Directory which includes subdirectories.
    • Source deletion options: select deletion option according to your preference.

    • Asset namespace: the asset namespace.

    • Ingestion labels: the label applied to the events from this feed.

  9. Click Next.

  10. Review your new feed configuration in the Finalize screen, and then click Submit.

UDM Mapping Table

Log field UDM mapping Logic
connection.dest_ip target.asset.ip
target.ip
Direct mapping when network.direction is OUTBOUND.
Mapped from principal.ip when network.direction is INBOUND.
connection.dest_port target.port Converted to integer if greater than -1.
connection.protocol network.ip_protocol Converted to string, then mapped to integer.
Based on the integer value, mapped to IP protocol name (e.g., TCP, UDP, ICMP).
connection.src_ip principal.ip Direct mapping.
connection.src_port principal.port Converted to integer.
dest_instance.region target.location.name Direct mapping.
dest_instance.vm_name target.asset.hostname Direct mapping.
dest_location.city target.location.city Direct mapping.
dest_location.country target.location.country_or_region Direct mapping.
dest_location.region target.location.state Direct mapping.
dest_vpc.project_id target.namespace Used with dest_vpc.vpc_name to form the target.namespace.
dest_vpc.vpc_name target.namespace Used with dest_vpc.project_id to form the target.namespace.
insertId metadata.product_log_id Direct mapping.
jsonPayload.bytes_sent network.sent_bytes Renamed to network.sent_bytes and converted to uinteger.
jsonPayload.packets_sent network.sent_packets Converted to integer.
labels.tunnel_id additional.fields Merged into additional.fields with the key Tunnel Id and type string_value.
logName security_result.category_details Direct mapping.
resource.labels.project_id target.resource.name Used to construct the target.resource.name with the format //cloudresourcemanager.googleapis.com/projects/{resource.labels.project_id}.
resource.labels.region target.location.country_or_region Direct mapping.
resource.labels.subnetwork_id target.user.attribute.labels Merged into target.user.attribute.labels with the key subnetwork_id.
resource.type metadata.product_event_type Direct mapping.
severity security_result.severity Mapped to LOW if the value is DEBUG.
src_gke_details.cluster.cluster_location principal.resource.attribute.labels Merged into principal.resource.attribute.labels with the key cluster_location.
src_gke_details.cluster.cluster_name principal.resource.attribute.labels Merged into principal.resource.attribute.labels with the key cluster_name.
src_gke_details.pod.pod_name principal.resource.attribute.labels Merged into principal.resource.attribute.labels with the key pod_name.
src_gke_details.pod.pod_namespace principal.resource.attribute.labels Merged into principal.resource.attribute.labels with the key pod_namespace.
src_instance.region principal.location.name Direct mapping.
src_instance.vm_name principal.asset.hostname Direct mapping.
src_location.city principal.location.city Direct mapping.
src_location.country principal.location.country_or_region Direct mapping.
src_location.region principal.location.state Direct mapping.
src_vpc.project_id principal.namespace Used with src_vpc.vpc_name to form the principal.namespace.
src_vpc.vpc_name principal.namespace Used with src_vpc.project_id to form the principal.namespace.
textPayload additional.fields Merged into additional.fields with the key Textpayload and type string_value.
timestamp metadata.event_timestamp Used to populate event_timestamp if jsonPayload.end_time is empty.
metadata.description A description of the network flow, including the reporter (SRC or DEST) and the direction (INBOUND or OUTBOUND), is generated based on the 'reporter' field.
metadata.event_type Set to NETWORK_CONNECTION for VPC flow logs and USER_RESOURCE_ACCESS for other log types.
metadata.log_type Set to GCP_VPC_FLOW.
metadata.product_name Set to GCP VPC Flow Logs.
metadata.product_version Set to 1.0.
metadata.vendor_name Set to Google Cloud.
network.direction Determined based on the target.port. If the port is a well-known or reserved port, it's considered INBOUND; otherwise, it's considered OUTBOUND.
security_result.severity Set to LOW by default.
target.resource.attribute.cloud.environment Set to GOOGLE_CLOUD_PLATFORM.
target.resource.resource_type Set to CLOUD_PROJECT.

Changes

2024-10-24

Enhancement:

  • Interchanged mapping of principal.ip, principal.port with target.ip and target.port respectively.

2024-03-15

Enhancement:

  • Mapped jsonPayload.src_gke_details.pod.pod_namespace, jsonPayload.src_gke_details.pod.pod_name, jsonPayload.src_gke_details.cluster.cluster_name, jsonPayload.src_gke_details.cluster.cluster_location to principal.resource.attribute.labels.
  • Mapped jsonPayload.dest_gke_details.pod.pod_namespace, jsonPayload.dest_gke_details.pod.pod_name, jsonPayload.dest_gke_details.cluster.cluster_name, jsonPayload.dest_gke_details.cluster.cluster_location to target.resource.attribute.labels.

2023-05-23

Enhancement:

  • Mapped 'metadata.event_type' to 'USER_RESOURCE_ACCESS' when field 'logName' does not contain 'vpc_flows'.
  • Mapped 'timestamp' to 'events.timestamp'.
  • Mapped 'textPayload', 'labels.tunnel_id' to 'additional.fields'.
  • Mapped 'resource.labels.region' to 'target.location.country_or_region'.
  • Added null checks for various fields wherever required.

2023-04-10

Enhancement:

  • Set target.resource.attribute.cloud.environment to GOOGLE_CLOUD_PLATFORM.
  • Set target.resource.name to the full resource name value.

2022-07-22

Enhancement:

  • Mapped resource.labels.location to principal.location.name.
  • Mapped resource.labels.subnetwork_id to target.user.attribute.labels.
  • Mapped logName to security_result.category_details.

Need more help? Get answers from Community members and Google SecOps professionals.