Collect AWS S3 server access logs

Supported in:

Google SecOps SIEM

Note: This feature is covered by Pre-GA Offerings Terms of the Google Security Operations Service Specific Terms. Pre-GA features might have limited support, and changes to pre-GA features might not be compatible with other pre-GA versions. For more information, see the Google Security Operations Technical Support Service guidelines and the Google Security Operations Service Specific Terms.

This document explains how to collect AWS S3 server access logs by setting up a Google Security Operations feed. The parser extracts fields using grok patterns, handles potential JSON input, and maps the extracted fields to the UDM. It performs data transformations, type conversions, and conditional logic based on the presence and values of specific fields to ensure accurate UDM representation.

Before you begin

• Ensure that you have a Google SecOps instance.
• Ensure that you have privileged access to AWS.

Configure AWS S3 server access logging

Google SecOps supports log collection using Amazon S3 through Amazon SQS.

1. Sign in to the AWS Management console.
2. Access the Amazon S3 console.
3. Go to Amazon S3 > Buckets.
4. Select an existing bucket or create a new one.
5. Click Properties.
6. In the Server access logging section, click Edit.
7. Select Enable.
8. In the Target bucket field, enter a name for the new bucket to send the log record objects to or select an existing bucket as the target.

Note: Your target bucket cannot have server access logging enabled. You can deliver logs to any bucket you own in the same region as the source bucket. However, it is not recommended to use the original source bucket.

1. Click Save changes.
2. To create the SQS queue for the S3 bucket, configure an Amazon SQS instance with the S3 storage. For more information, see Configuring a bucket for notifications (SNS topic or SQS queue).

Note: IAM user and KMS key policies are required for Amazon S3, AWS KMS, and Amazon SQS.

Note: For more information, see Using IAM policies with AWS KMS.

Based on the service and region, identify the endpoints for connectivity by referring to the following AWS documentation: * For information about any logging source, see AWS Identity and Access Management endpoints and quotas. * For information about S3 logging sources, see Amazon Simple Storage Service endpoints and quotas. * For information about SQS logging sources, see Amazon Simple Queue Service endpoints and quotas.

Configure a feed in Google SecOps to ingest AWS S3 server access logs

1. In the Google SecOps menu, go to Settings > Feeds > Add new.
2. In the Source type list, select Amazon S3 or Amazon SQS.
3. In the Log type list, select AWS S3 server access.
4. Click Next.
5. Google SecOps supports log collection using an access key ID and secret method. To create the access key ID and secret, see Configure tool authentication with AWS.
6. Based on the AWS S3 server access configuration that you created, specify values for the input parameters:
   • If using Amazon S3, specify values for the following fields:
     • Region
     • S3 URI
     • URI is a
     • Source deletion option
     • Access key ID
     • Secret access key
   • If using Amazon SQS, specify values for the following fields:
     • Region
     • Queue name
     • Account number
     • Queue access key ID
     • Queue secret access key
     • Source deletion option
7. Click Next, and then click Submit.

UDM Mapping Table

Log Field	UDM Mapping	Logic
aclRequired	target.resource.attribute.labels.key: "aclRequired" target.resource.attribute.labels.value: Value of aclRequired	Directly mapped from the raw log field aclRequired.
authenticationtype	extensions.auth.auth_details	Directly mapped from the raw log field authenticationtype.
bucket	target.resource.name	Directly mapped from the raw log field bucket.
bucket	target.resource.resource_type: "STORAGE_BUCKET"	The parser sets the resource_type to "STORAGE_BUCKET" if the bucket field is present.
bucketowner	target.resource.product_object_id	Directly mapped from the raw log field bucketowner.
bytes_sent	network.sent_bytes	Directly mapped from the raw log field bytes_sent after converting it to an unsigned integer and replacing "-" with "0".
ciphersuite	network.application_protocol: "HTTPS"	The parser sets the application_protocol to "HTTPS" if the ciphersuite field is present.
ciphersuite	network.tls.cipher	Directly mapped from the raw log field ciphersuite.
errorcode	security_result.action_details	Directly mapped from the raw log field errorcode.
errorcode	security_result.action: "BLOCK"	The parser sets the action to "BLOCK" if the errorcode field contains "AccessDenied" (case-insensitive).
hostheader	target.hostname	Extracted from the raw log field hostheader, potentially removing the port number.
hostheader	target.port	Extracted from the raw log field hostheader if a port number is present.
hostid	target.resource.attribute.labels.key: "S3 Extended Request ID" target.resource.attribute.labels.value: Value of hostid	Directly mapped from the raw log field hostid.
http_capture	network.http.method	The HTTP method is extracted from the http_capture field.
http_capture	network.http.version	The HTTP version is extracted from the http_capture field.
http_capture	target.url	The target URL is constructed using hostheader and http_request_uri (extracted from http_capture), prefixed with "http://" or "https://" based on the presence of ciphersuite.
httpstatus	network.http.response_code	Directly mapped from the raw log field httpstatus after converting it to an integer.
object_version_id	target.resource.product_object_id	Directly mapped from the raw log field object_version_id.
objectsize	target.file.size	Directly mapped from the raw log field objectsize after converting it to an unsigned integer and replacing "-" with "0".
operation	metadata.product_event_type	Directly mapped from the raw log field operation.
referrer	network.http.referral_url	Directly mapped from the raw log field referrer after removing quotes.
remoteip	metadata.event_type: "USER_RESOURCE_ACCESS"	The parser sets the event_type to "USER_RESOURCE_ACCESS" if the remoteip field is empty.
remoteip	principal.ip	Directly mapped from the raw log field remoteip.
requester	target.resource.attribute.labels.key: "Access Point ARN" target.resource.attribute.labels.value: Value of requester	Directly mapped from the raw log field requester.
requester_user	principal.user.userid	Directly mapped from the raw log field requester_user.
requestid	network.session_id	Directly mapped from the raw log field requestid.
request_time_ms	network.session_duration.nanos	Directly mapped from the raw log field request_time_ms after converting it to an integer, replacing "-" with "0", and padding with zeros to represent nanoseconds.
signatureversion	target.resource.attribute.labels.key: "Signature Version" target.resource.attribute.labels.value: Value of signatureversion	Directly mapped from the raw log field signatureversion.
time	metadata.event_timestamp	Parsed from the raw log field time and converted to a timestamp.
tlsVersion	network.tls.version	Directly mapped from the raw log field tlsVersion.
useragent	network.http.user_agent	Directly mapped from the raw log field useragent after removing quotes.
(Parser Logic)	metadata.event_type: "NETWORK_HTTP"	The parser sets the default event_type to "NETWORK_HTTP".
(Parser Logic)	metadata.log_type: "AWS_S3_SERVER_ACCESS"	The parser sets the log_type to "AWS_S3_SERVER_ACCESS".
(Parser Logic)	metadata.product_name: "AWS S3 Server Access"	The parser sets the product_name to "AWS S3 Server Access".
(Parser Logic)	metadata.product_version: "HTTP/http_version"	The parser sets the product_version using the extracted http_version.
(Parser Logic)	metadata.vendor_name: "AMAZON"	The parser sets the vendor_name to "AMAZON".
(Parser Logic)	network.application_protocol: "HTTP"	The parser sets the application_protocol to "HTTP" if the ciphersuite field is not present.
(Parser Logic)	timestamp	The parser sets the event timestamp to the current time when the event is processed.

Changes

2023-07-19

• Bug-Fix -
• Modified Grok pattern to handle a hyphen (-) when data is not present.
• Mapped "aclRequired" to "target.resource.attribute.labels".

2023-05-04

• Enhancement-
• Added support for JSON format logs.

2022-07-21

• Newly Created Parser.

Need more help? Get answers from Community members and Google SecOps professionals.