Collect AWS Aurora logs

Supported in:

Google SecOps SIEM

Note: This feature is covered by Pre-GA Offerings Terms of the Google Security Operations Service Specific Terms. Pre-GA features might have limited support, and changes to pre-GA features might not be compatible with other pre-GA versions. For more information, see the Google Security Operations Technical Support Service guidelines and the Google Security Operations Service Specific Terms.

This document explains how to ingest AWS Aurora logs to Google Security Operations. AWS Aurora is a managed relational database service that offers high performance, scalability, and availability. In this integration, you will configure AWS Aurora to forward logs to Google SecOps for analysis, monitoring, and threat detection.

Before You Begin

• Ensure that you have a Google SecOps instance.
• Ensure that you have privileged access to AWS.
• Ensure that your AWS Aurora database cluster is set up and running.

Configure Amazon S3 bucket

1. Create an Amazon S3 bucket following this user guide: Creating a bucket
2. Save the bucket Name and Region for later use.
3. Create a user following this user guide: Creating an IAM user.
4. Select the created User.
5. Select the Security credentials tab.
6. Click Create Access Key in the Access Keys section.
7. Select Third-party service as the Use case.
8. Click Next.
9. Optional: add a description tag.
10. Click Create access key.
11. Click Download CSV file to save the Access Key and Secret Access Key for later use.
12. Click Done.
13. Select the Permissions tab.
14. Click Add permissions in the Permissions policies section.
15. Select Add permissions.
16. Select Attach policies directly.
17. Search for and select AmazonS3FullAccess and CloudWatchLogsFullAccess policies.
18. Click Next.
19. Click Add permissions.

Configure Enhanced Monitoring

1. Sign in to the AWS Management Console.
2. In the search bar, type RDS and select RDS from the services list.
3. In the RDS Dashboard, select Databases from the navigation pane.
4. Select the Aurora cluster you want to monitor.
5. Under the Logs & monitoring section, click Modify.
6. Go to the Monitoring section and enable Enhanced Monitoring.
7. Set the Monitoring role to the appropriate IAM role that has permissions to publish to CloudWatch Logs or S3.
8. Save the changes and apply them to your Aurora cluster.

Configure AWS Aurora Audit Logs

1. In the RDS Dashboard, select Databases and click your Aurora cluster.
2. Under the Logs & Monitoring section, click Modify.
3. In the Database Options section, ensure that Enable Audit Logs is selected.
4. Under Destination, choose S3 and specify the S3 bucket where logs will be stored.
5. Click Save changes to apply the settings.

Optional: AWS Aurora Logs Configuration using CloudWatch

For additional monitoring capabilities, you can configure CloudWatch Logs to capture Aurora logs.

1. In the RDS Dashboard, select your Aurora cluster.
2. Under the Logs & Monitoring section, ensure that CloudWatch Logs integration is enabled.
3. Go to CloudWatch Logs and create a new Log Group to store the Aurora logs.
4. On the Log Groups screen, choose the name of your new Log Group.
5. Select Actions > Export data to Amazon S3.

6. On the Export data to Amazon S3 screen, under Define data export, set the time range for the data to export using From and To.

   Note: If your log group has multiple log streams, you can provide a log stream prefix to limit the log group data to a specific stream.
   Choose Advanced, and then for Stream prefix, enter the log stream prefix.

7. Choose S3 bucket, select the account associated with the Amazon S3 bucket.

8. S3 bucket name, select an Amazon S3 bucket.

9. S3 Bucket prefix, enter the randomly generated string that you specified in the bucket policy.

10. Choose Export to export your log data to Amazon S3.

11. To view the status of the log data that you exported to Amazon S3, select Actions > View all exports to Amazon S3.

Configure a feed in Google SecOps to ingest AWS Aurora logs

1. Go to SIEM Settings > Feeds.
2. Click Add new.
3. In the Feed name field, enter a name for the feed (for example, AWS Aurora Logs).
4. Select Amazon S3 as the Source type.
5. Select AWS Aurora as the Log type.
6. Click Next.

7. Specify values for the following input parameters:

   • Region: The region where the Amazon S3 bucket is located.
   • S3 URI: the bucket URI.
     • s3://your-log-bucket-name/
       • Replace your-log-bucket-name with the actual name of the bucket.
   • URI is a: select Directory or Directory which includes subdirectories.

   • Source deletion options: select the deletion option according to your preference.

     Note: If you select the Delete transferred files or Delete transferred files and empty directories option, make sure that you granted appropriate permissions to the service account.

   • Access Key ID: the User access key with access to the S3 bucket.

   • Secret Access Key: the User secret key with access to the S3 bucket.

   • Asset namespace: the asset namespace.

   • Ingestion labels: the label to be applied to the events from this feed.

8. Click Next.

9. Review your new feed configuration in the Finalize screen, and then click Submit.

UDM Mapping Table

Log Field	UDM Mapping	Logic
account	principal.group.product_object_id	Directly mapped from the account field in the raw log.
column1	timestamp_epoch	Directly mapped from the column1 field in the raw log. Used to derive metadata.event_timestamp.
column10	Varies	Can be principal.process.command_line, object or number depending on the log format.
column11	ddl or response or command_line2	Can be principal.resource.resource_subtype (ddl), security_result.outcomes.value (response) or part of principal.process.command_line (command_line2) depending on the log format.
column12	operation or response or command_line3	Can be sr.summary (operation), security_result.outcomes.value (response) or part of principal.process.command_line (command_line3) depending on the log format.
column13	database or response	Can be target.resource.name (database) or security_result.outcomes.value (response) depending on the log format.
column14	object	Directly mapped to principal.resource.product_object_id or target_data.resource.name depending on the log format.
column15	command_line	Directly mapped to principal.process.command_line.
column16	response	Directly mapped to security_result.outcomes.value.
column2	timestamp or timestamp_ms	Directly mapped from the column2 field in the raw log.
column3	ip or hostname	Can be principal.ip or principal.resource.name depending on the log format.
column4	port or userid	Can be principal.port or principal.user.userid depending on the log format.
column5	userid or ip	Can be principal.user.userid or principal.ip depending on the log format.
column6	hostname or connection_id	Can be principal.resource.name or network.session_id depending on the log format.
column7	connection_id or query_id	Can be network.session_id or principal.process.pid depending on the log format.
column8	operation	Directly mapped to sr.summary or metadata.product_event_type.
column9	query_id or database	Can be principal.process.pid or target_data.resource.name depending on the log format.
command_line	principal.process.command_line	Directly mapped from the extracted command_line field.
connection_id	network.session_id	Directly mapped from the extracted connection_id field.
database	target.resource.name	Directly mapped from the extracted database field. Derived from several fields like operation, command_line, has_principal_user, and has_principal_machine through conditional logic in the parser. Can be RESOURCE_DELETION, RESOURCE_CREATION, RESOURCE_READ, RESOURCE_WRITTEN, USER_RESOURCE_ACCESS, USER_UNCATEGORIZED, or GENERIC_EVENT. Hardcoded to "AWS_AURORA". Mapped from column8 or derived from parser logic. Hardcoded to "AURORA". Hardcoded to "AMAZON".
has_principal_machine	has_principal_machine	Set to "true" if principal.ip is present, otherwise initialized to "false".
has_principal_user	has_principal_user	Set to "true" if principal.user.userid is present, otherwise initialized to "false".
hostname	principal.resource.name	Directly mapped from the extracted hostname field.
ip	principal.ip	Directly mapped from the extracted ip field.
logevent.id	security_result.detection_fields.value	Nested within target.logEvents.logEvents, mapped with key "id".
logevent.message	security_result.detection_fields.value	Nested within target.logEvents.logEvents, mapped with key "message". Used to extract principal.ip, time_unix, operation, and user.
logevent.timestamp	security_result.detection_fields.value	Nested within target.logEvents.logEvents, mapped with key "timestamp".
object	target_data.resource.name or principal.resource.product_object_id	Directly mapped from the extracted object field.
operation	sr.summary	Directly mapped from the extracted operation field.
port	principal.port	Directly mapped from the extracted port field.
query_id	principal.process.pid	Directly mapped from the extracted query_id field.
response	security_result.outcomes.value	Directly mapped from the extracted response field.
service	principal.application	Directly mapped from the service field in the raw log.
src_ip	principal.ip	Extracted from logevent.message within the nested target.logEvents.logEvents structure.
target.logEvents.logGroup	target.resource.attribute.labels.value	Mapped with key "logGroup".
target.logEvents.logStream	target.resource.attribute.labels.value	Mapped with key "logStream".
target.logEvents.messageType	target.resource.attribute.labels.value	Mapped with key "messageType".
target.logEvents.owner	target.resource.attribute.labels.value	Mapped with key "owner".
timestamp_epoch	metadata.event_timestamp	Converted to metadata.event_timestamp using the date filter.
user	principal.user.userid	Extracted from logevent.message within the nested target.logEvents.logEvents structure.
userid	principal.user.userid	Directly mapped from the extracted userid field.

Changes

2024-01-12

• Mapped "logEvents.messageType", "logEvents.owner", "logEvents.logGroup", "logEvents.logStream" to "target.resource.attribute.labels".
• Mapped "logEvents.logEvents.message", "logEvents.logEvents.timestamp", "logEvents.logEvents.id" to "securit_result.detection_fields".
• Added a Grok pattern to retrieve the IP address from "logEvents.logEvents.message" and mapped "src_data" to "principal.ip".
• Mapped "user" to "principal.user.userid".

2023-11-02

• Newly created parser.

Need more help? Get answers from Community members and Google SecOps professionals.