Collect AWS Elastic Load Balancing logs

Supported in:

Google SecOps SIEM

Note: This feature is covered by Pre-GA Offerings Terms of the Google Security Operations Service Specific Terms. Pre-GA features might have limited support, and changes to pre-GA features might not be compatible with other pre-GA versions. For more information, see the Google Security Operations Technical Support Service guidelines and the Google Security Operations Service Specific Terms.

This document explains how to collect AWS Elastic Load Balancing logs by setting up a Google Security Operations feed. The parser converts the logs into UDM format. It uses grok patterns to extract fields from both CEF and non-CEF formatted messages, mapping them to UDM fields and handling various data transformations, including specific logic for HTTP, TLS, and security-related fields. It also performs conditional processing based on the presence or format of certain fields to ensure accurate UDM representation.

Before you begin

• Ensure that you have a Google SecOps instance.
• Ensure that you have privileged access to AWS.

Configure AWS Elastic Load Balancing

• Enable access logging to send Access Logs to an S3 storage bucket.
• Create an Amazon Simple Queue Service (SQS) and attach it to an S3 storage bucket.

Configure Amazon S3 bucket

1. Sign in to the AWS console.
2. Create an Amazon S3 bucket following this user guide: Creating a bucket
3. Save the bucket Name (for example, elb-logs) and Region for later use.
4. Create a user following this user guide: Creating an IAM user.
5. Select the created User.
6. Select the Security credentials tab.
7. Click Create Access Key in the Access Keys section.
8. Select Third-party service as the Use case.
9. Click Next.
10. Optional: add a description tag.
11. Click Create access key.
12. Click Download CSV file to save the Access Key and Secret Access Key for later use.
13. Click Done.
14. Select the Permissions tab.
15. Click Add permissions in the Permissions policies section.
16. Select Add permissions.
17. Select Attach policies directly.
18. Search for and select the AmazonS3FullAccess policy.
19. Click Next.
20. Click Add permissions.

Configure AWS Elastic Load Balancer to Enable Access Logs

1. Sign in to the AWS Management Console.
2. Search for and select EC2.
3. Select Load balancers in the navigation menu.
4. Select the load balancer for which you want to enable logging.
5. In the Description tab, scroll to Attributes.
6. Click Edit attributes.
7. Enable Access logs by selecting Enable.
8. Select the S3 bucket created earlier (for example, elb-logs).
9. Optional: set the Log Prefix for easier log identification (for example, elb/access-logs/).
10. Click Save.

Configure a feed in Google SecOps to ingest the AWS Elastic Load Balancer logs

1. Go to SIEM Settings > Feeds.
2. Click Add new.
3. In the Feed name field, enter a name for the feed (for example, AWS Elastic Load Balancer Logs).
4. Select Amazon S3 as the Source type.
5. Select AWS Elastic Load Balancer as the Log type.
6. Click Next.

7. Specify values for the following input parameters:

   • Region: the region where the Amazon S3 bucket is located.
   • S3 URI: the bucket URI.
     • s3:/BUCKET_NAME
       • Replace BUCKET_NAME with the actual name of the bucket.
   • URI is a: select the URI TYPE according to the log stream configuration (Single file | Directory | Directory which includes subdirectories).
   • Source deletion options: select the deletion option according to your preference.
   Note: If you select the Delete transferred files or Delete transferred files and empty directories option, make sure that you granted appropriate permissions to the service account.
   • Access Key ID: the User access key with access to the S3 bucket.
   • Secret Access Key: the User secret key with access to the S3 bucket.
   • Asset namespace: the asset namespace.
   • Ingestion labels: the label to be applied to the events from this feed.

8. Click Next.

9. Review your new feed configuration in the Finalize screen, and then click Submit.

UDM Mapping Table

Log Field	UDM Mapping	Logic
actions_executed	security_result.action	If actions_executed is "waf,forward" or "waf,redirect", set to "ALLOW". If actions_executed is "waf", set to "BLOCK".
chosen_cert_arn	principal.user.attribute.labels	If chosen_cert_arn contains "session", map its value to security_result.description. Otherwise, create a label with key "ARN" and value as chosen_cert_arn and add it to the principal.user.attribute.labels array.
chosen_cert_arn	security_result.description	If chosen_cert_arn contains "session", map its value to this field.
client_ip	principal.asset.ip	Directly mapped.
client_ip	principal.ip	Directly mapped.
client_port	principal.port	Directly mapped.
classification	security_result.rule_name	Directly mapped if not empty or "-".
classification_reason	security_result.summary	Directly mapped if not empty or "-".
Customer (CEF)	principal.user.user_display_name	Directly mapped from the CEF field.
data	Various	Parsed using grok patterns to extract multiple fields. See other rows for specific mappings.
data.act (CEF)	security_result.action_details	Directly mapped from the CEF field.
data.app (CEF)	principal.application	Directly mapped from the CEF field.
data.ccode (CEF)	principal.location.country	Directly mapped from the CEF field.
data.cicode (CEF)	principal.location.city	Directly mapped from the CEF field.
data.cn1 (CEF)	network.http.response_code	Directly mapped from the CEF field.
data.cpt (CEF)	principal.port	Directly mapped from the CEF field.
data.cs1Label (CEF)	additional.fields	Creates a key-value pair with key "Cap Support" and value from cs1Label.
data.cs2Label (CEF)	additional.fields	Creates a key-value pair with key "Javascript Support" and value from cs2Label.
data.cs3Label (CEF)	additional.fields	Creates a key-value pair with key "CO Support" and value from cs3Label.
data.cs4Label (CEF)	additional.fields	Creates a key-value pair with key "VID" and value from cs4Label.
data.cs5Label (CEF)	additional.fields	Creates a key-value pair with key "clappsig" and value from cs5Label.
data.cs6Label (CEF)	additional.fields	Creates a key-value pair with key "clapp" and value from cs6Label.
data.cs7Label (CEF)	additional.fields	Creates a key-value pair with key "latitude" and value from cs7Label.
data.deviceExternalId (CEF)	about.asset.asset_id	Used as part of the asset ID: Incapsula.SIEMintegration:deviceExternalId.
data.deviceFacility (CEF)	principal.location.region	Directly mapped from the CEF field.
data.dproc (CEF)	target.process.command_line	Directly mapped from the CEF field.
data.dst_ip	target.asset.ip	Directly mapped.
data.dst_ip	target.ip	Directly mapped.
data.dst_port	target.port	Directly mapped.
data.elb	target.resource.id	Directly mapped.
data.fileId (CEF)	security_result.detection_fields	Creates a key-value pair with key "fileId" and value from fileId.
data.in (CEF)	network.received_bytes	Directly mapped from the CEF field.
data.request (CEF)	target.url	Directly mapped from the CEF field.
data.requestClientApplication (CEF)	network.http.user_agent	Directly mapped from the CEF field.
data.requestMethod (CEF)	network.http.method	Directly mapped from the CEF field.
data.severity (CEF)	security_result.severity	Mapped to LOW if severity is 0.
data.sip (CEF)	principal.asset.ip	Directly mapped from the CEF field.
data.sip (CEF)	principal.ip	Directly mapped from the CEF field.
data.siteid (CEF)	security_result.detection_fields	Creates a key-value pair with key "siteid" and value from siteid.
data.sourceServiceName (CEF)	principal.application	Directly mapped from the CEF field.
data.spt (CEF)	principal.port	Directly mapped from the CEF field.
data.src (CEF)	principal.ip	Directly mapped from the CEF field.
data.suid (CEF)	principal.user.userid	Directly mapped from the CEF field.
data.ver (CEF)	network.tls.version	The version part is extracted using grok and mapped.
data.ver (CEF)	network.tls.cipher	The cipher part is extracted using grok and mapped.
data.xff (CEF)	principal.ip	Directly mapped from the CEF field.
domain_name	principal.administrative_domain	Directly mapped.
http_method	network.http.method	Directly mapped.
log_type	metadata.log_type	Directly mapped.
message	Various	Parsed using grok patterns to extract multiple fields. See other rows for specific mappings.
received_bytes	network.received_bytes	Directly mapped.
redirect_url	network.application_protocol	If redirect_url starts with "http", the protocol is extracted and mapped.
redirect_url	target.asset.hostname	If redirect_url starts with "http", the hostname is extracted and mapped.
redirect_url	target.hostname	If redirect_url starts with "http", the hostname is extracted and mapped.
redirect_url	target.port	If redirect_url starts with "http", the port is extracted and mapped.
request_creation_time	metadata.collected_timestamp	Directly mapped after date parsing.
request_processing_time	security_result.detection_fields	Creates a key-value pair with key "request_processing_time" and value from this field.
response_processing_time	security_result.detection_fields	Creates a key-value pair with key "response_processing_time" and value from this field.
sent_bytes	network.sent_bytes	Directly mapped.
ssl_cipher	network.tls.cipher	Directly mapped.
ssl_protocol	network.tls.version	Directly mapped.
target_group_arn	target.group.group_display_name	Directly mapped.
target_processing_time	security_result.detection_fields	Creates a key-value pair with key "target_processing_time" and value from this field.
target_status_code	target.labels	Creates a label with key "target_status_code" and value from this field, and adds it to the target.labels array.
time	metadata.event_timestamp	Directly mapped after date parsing.
trace_id	metadata.product_log_id	Directly mapped after removing "Root=".
url	network.http.referral_url	Directly mapped.
user_agent	network.http.user_agent	Directly mapped.
(Parser)	metadata.event_type	Set to "NETWORK_HTTP" if principal and target machine IDs are present, "STATUS_UPDATE" if only principal machine ID is present, "GENERIC_EVENT" if no target IP, hostname, or destination IP is present, and "NETWORK_HTTP" otherwise.
(Parser)	metadata.product_name	Set to "AWS Elastic Load Balancer".
(Parser)	metadata.vendor_name	Set to "AMAZON".

Changes

2024-03-22

• Added new Grok pattern to support new pattern of JSON logs.
• Added support for CEF pattern logs.
• Mapped "dst_ip" to "target.ip".
• Mapped "dst_port" to "target.port".
• Mapped "sip" to "principal.ip".
• Mapped "request_processing_time", "target_processing_time", "siteid", "fileId", and "response_processing_time" to "security_result.detection.fields".
• Aligned mappings for "principal.ip" and "principal.asset.ip".
• Aligned mappings for "target.ip" and "target.asset.ip".
• Aligned mappings for "target.hostname" and "target.asset.hostname".
• Mapped "cipher" to "network.tls.cipher".
• Mapped "version" to "network.tls.version".
• Mapped "Customer" to "principal.user.user_display_name".

2022-05-27

• Enhancement - Modified the value stored in metadata.product_name to 'AWS Elastic Load Balancer'.

Need more help? Get answers from Community members and Google SecOps professionals.