Collect AWS VPC Flow logs
This document describes how you can collect AWS VPC Flow Logs by using a Google Security Operations forwarder.
For more information, see Data ingestion to Google Security Operations.
An ingestion label identifies the parser which normalizes raw log data
to structured UDM format. The information in this document applies to the parser with the AWS_VPC_FLOW
ingestion label.
Before you begin
Ensure that the Amazon S3 bucket is created. For more information, see Create your first S3 bucket.
Ensure that the Amazon CloudWatch log group is created. For more information, see Working with log groups and log streams.
Configure AWS VPC Flow
Configure AWS VPC Flow based on whether you forward the logs to Amazon S3 or to Amazon CloudWatch.
For information about forwarding logs to the Amazon S3 bucket, see the Configure flow logs to forward logs to Amazon S3 section of this document.
For information about forwarding logs to Amazon CloudWatch, see the Configure flow logs to Amazon CloudWatch section of this document.
Configure flow logs to forward logs to Amazon S3
After you create and configure the Amazon S3 bucket, you can create flow logs for your network interfaces, subnets, and VPCs networks.
Create a flow log for a network interface
- Sign in to the Amazon EC2 console.
- In the navigation pane, select Network Interfaces.
- Select one or more network interfaces.
- Select Actions > Create flow log.
- Configure the flow log settings. For more information, see Configure flow log settings section of this document.
Create a flow log for a subnet
- Sign in to the Amazon VPC console.
- In the navigation pane, select Subnets.
- Select one or more subnets.
- Select Actions > Create flow log.
- Configure the flow log settings. For more information, see Configure flow log settings section of this document.
Create a flow log for a VPC
- Sign in to the Amazon VPC console.
- In the navigation pane, select VPCs.
- Select one or more VPCs.
- Select Actions > Create flow log.
- Configure the flow log settings. For more information, see Configure flow log settings section of this document.
Configure flow log settings
In the Filter section, specify the IP traffic to log:
Accept: log only accepted traffic.
Reject: log only rejected traffic.
All: log accepted and rejected traffic.
In the Maximum aggregation interval section, select 1 minute.
In the Destination section, select Send to an Amazon S3 bucket.
In the S3 bucket ARN section, specify the ARN of an Amazon S3 bucket.
In the Log record format section, specify the following formats for the flow log record:
- To use the default flow log record format, select AWS default format.
- To create a custom format, select Custom format.
Configure the VPC log flow with the custom AWS log format to use MSS true IP features.
In the Log format list, select all the attributes.
In the Format preview section, review the custom format.
In the Log file format section, select Text (default).
In the Hive-compatible S3 prefix section, keep the Enable checkbox unchecked.
In the Partition logs by time section, select Every 1 hour (60 mins).
To add a tag to the flow log, select Add new tag and specify the tag key and value.
Select Create flow log. For more information, see Publish flow logs to Amazon S3.
Configure flow logs to the Amazon CloudWatch
You can configure flow log either from VPCs, subnets, or network interfaces.
In the Filter section, specify the type of IP traffic to log:
Accept: log only accepted traffic.
Reject: log only rejected traffic.
All: log accepted and rejected traffic.
In the Maximum aggregation interval section, select 1 minute.
In the Destination section, select Send to CloudWatch Logs.
In the Destination log group section, provide the destination log group name that you created.
In the IAM role list, select the role name. The selected role name has permissions to publish logs to CloudWatch logs.
The IAM role must include the following permissions:
{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "logs:CreateLogGroup", "logs:CreateLogStream", "logs:PutLogEvents", "logs:DescribeLogGroups", "logs:DescribeLogStreams" ], "Resource": "*" } ] }
In the Log record format section, select Custom format for the flow log record.
To add a tag to the flow log, select Add new tag and specify the tag key and value.
Select Create flow log. For more information, see Publish flow logs to Amazon S3.
Amazon S3 can be configured to send the event notifications to Amazon SQS. For more information, see Configuring a bucket for notifications (SNS topic or SQS queue).
IAM user policies are required for Amazon S3 and Amazon SQS if using Amazon SQS (Amazon S3 using Amazon SQS) as a log collection method. For more information, see Using IAM policies with AWS KMS.
Based on the service and region, identify the endpoints for connectivity by referring to the following AWS documentation:
For information about any logging sources, see AWS Identity and Access Management endpoints and quotas.
For information about Amazon S3 logging sources, see Amazon Simple Storage Service endpoints and quotas.
For information about Amazon SQS logging sources, see Amazon Simple Queue Service endpoints and quotas.
For information about Amazon CloudWatch logging sources, see Amazon CloudWatch logs endpoints and quotas.
Configure Google Security Operations forwarder and syslog to ingest AWS VPC Flow logs
- From the Google Security Operations menu, select Settings.
- Click Forwarders.
- Click Add new forwarder.
- In the Forwarder name field, type a name.
- Click Submit. The forwarder is added and the Add collector configuration window appears.
- In the Collector name field, type a name.
- Select AWS VPC Flow as the Log type.
- Select Syslog as the Collector type.
- Configure the following input parameters:
- Protocol: specify the connection protocol the collector will use to listen for syslog data.
- Address: specify the target IP address or hostname where the collector resides and addresses to the syslog data.
- Port: specify the target port where the collector resides and listens for syslog data.
- Click Submit.
For more information about Google Security Operations forwarders, see Google Security Operations forwarders documentation. For information about requirements for each forwarder type, see Forwarder configuration by type.
If you encounter issues when you create forwarders, contact Google Security Operations support.
What's next
UDM mapping
This parser code takes raw AWS CloudTrail logs in JSON format describing EC2 VPC events and transforms them into a structured UDM format. It extracts relevant fields, renames them to match the UDM schema, and enriches the data with additional context like resource type, cloud environment, and labels for easier analysis.
UDM Mapping table for AWS EC2 VPC Parser
Log Field (Ascending) | UDM Mapping | Logic |
---|---|---|
CidrBlock | event.idm.entity.entity.resource.attribute.labels.cidr_block | Directly mapped from the "CidrBlock" field in the raw log. |
CidrBlock | event.idm.entity.entity.network.ip_subnet_range | Directly mapped from the "CidrBlock" field in the raw log. |
CidrBlockAssociation.AssociationID | event.idm.entity.entity.resource.attribute.labels.cidr_block_association_association_id | Directly mapped from the "AssociationID" field within the "CidrBlockAssociation" array in the raw log. |
CidrBlockAssociation.CidrBlockState.State | event.idm.entity.entity.resource.attribute.labels.cidr_block_association_cidr_block_state_state | Directly mapped from the "State" field within the "CidrBlockState" object of the "CidrBlockAssociation" array in the raw log. |
CidrBlockAssociation.CidrBlockState.StatusMessage | event.idm.entity.entity.resource.attribute.labels.cidr_block_association_cidr_block_state_status_message | Directly mapped from the "StatusMessage" field within the "CidrBlockState" object of the "CidrBlockAssociation" array in the raw log. |
DhcpOptionsID | event.idm.entity.entity.resource.attribute.labels.dhcp_options_id | Directly mapped from the "DhcpOptionsID" field in the raw log. |
ID | event.idm.entity.entity.resource.product_object_id | Directly mapped from the "ID" field in the raw log, which is renamed to "VpcID" in the parser. |
ID | event.idm.entity.metadata.product_entity_id | Directly mapped from the "ID" field in the raw log, which is renamed to "VpcID" in the parser. |
InstanceTenancy | event.idm.entity.entity.resource.attribute.labels.instance_tenancy | Directly mapped from the "InstanceTenancy" field in the raw log. |
IsDefault | event.idm.entity.entity.resource.attribute.labels.is_default | Directly mapped from the "IsDefault" field in the raw log. |
Ipv6CidrBlockAssociationSet.AssociationID | event.idm.entity.entity.resource.attribute.labels.ipv6_cidr_block_association_set_association_id | Directly mapped from the "AssociationID" field within the "Ipv6CidrBlockAssociationSet" array in the raw log. |
Ipv6CidrBlockAssociationSet.Ipv6CidrBlock | event.idm.entity.entity.resource.attribute.labels.ipv6_cidr_block_association_set_ipv6_cidr_block | Directly mapped from the "Ipv6CidrBlock" field within the "Ipv6CidrBlockAssociationSet" array in the raw log. |
Ipv6CidrBlockAssociationSet.Ipv6CidrBlockState.State | event.idm.entity.entity.resource.attribute.labels.ipv6_cidr_block_association_set_ipv6_cidr_block_state_state | Directly mapped from the "State" field within the "Ipv6CidrBlockState" object of the "Ipv6CidrBlockAssociationSet" array in the raw log. |
Ipv6CidrBlockAssociationSet.Ipv6CidrBlockState.StatusMessage | event.idm.entity.entity.resource.attribute.labels.ipv6_cidr_block_association_set_ipv6_cidr_block_state_status_message | Directly mapped from the "StatusMessage" field within the "Ipv6CidrBlockState" object of the "Ipv6CidrBlockAssociationSet" array in the raw log. |
Ipv6CidrBlockAssociationSet.Ipv6Pool | event.idm.entity.entity.resource.attribute.labels.ipv6_cidr_block_association_set_ipv6_pool | Directly mapped from the "Ipv6Pool" field within the "Ipv6CidrBlockAssociationSet" array in the raw log. |
Ipv6CidrBlockAssociationSet.NetworkBorderGroup | event.idm.entity.entity.resource.attribute.labels.ipv6_cidr_block_association_set_network_border_group | Directly mapped from the "NetworkBorderGroup" field within the "Ipv6CidrBlockAssociationSet" array in the raw log. |
OwnerID | event.idm.entity.entity.resource.attribute.labels.owner_id | Directly mapped from the "OwnerID" field in the raw log. |
State | event.idm.entity.entity.resource.attribute.labels.state | Directly mapped from the "State" field in the raw log. |
TagSet.Key | event.idm.entity.entity.resource.attribute.labels.key | Directly mapped from the "Key" field within the "TagSet" array in the raw log. This creates a new label for each tag in the "TagSet". |
TagSet.Value | event.idm.entity.entity.resource.attribute.labels.value | Directly mapped from the "Value" field within the "TagSet" array in the raw log. This populates the value for each corresponding label created from the "Key" field. |
N/A | event.idm.entity.entity.resource.attribute.cloud.environment | Hardcoded to "AMAZON_WEB_SERVICES" in the parser code. |
N/A | event.idm.entity.entity.resource.resource_type | Hardcoded to "VPC_NETWORK" in the parser code. |
N/A | event.idm.entity.metadata.collected_timestamp | Populated with the event timestamp, which is derived from the "collection_time" field in the raw log. |
N/A | event.idm.entity.metadata.entity_type | Hardcoded to "RESOURCE" in the parser code. |
N/A | event.idm.entity.metadata.product_name | Hardcoded to "Amazon VPC" in the parser code. |
N/A | event.idm.entity.metadata.vendor_name | Hardcoded to "AWS" in the parser code. |
N/A | events.timestamp | Populated with the event timestamp, which is derived from the "collection_time" field in the raw log. |