Collect AWS CloudWatch logs

Supported in:

Google secops SIEM

This document explains how to ingest AWS CloudWatch logs to Google Security Operations using Amazon S3 or Amazon Kinesis Data Firehose. AWS CloudWatch is a monitoring and observability service, which collects operational data in the form of logs, metrics, and events. This integration lets you send these logs to Google SecOps for analysis and monitoring.

Before you begin

Make sure you have the following prerequisites:

Google SecOps instance
Privileged access to AWS

Configure CloudWatch logs export using AWS S3

This export process is required to be done regularly, in order to get the latest CloudWatch logs ingested into S3.

Create an Amazon S3 bucket

We recommend to use a bucket that was created specifically for CloudWatch logs.

Open the Amazon S3 console.
If required, you can change the Region.
- On the navigation bar, select the Region where your CloudWatch Logs reside.
Click Create Bucket.
- Bucket Name: enter a meaningful name for the bucket.
- Region: select the region where your CloudWatch Logs data resides.
- Click Create.

Create an IAM user with full access to Amazon S3 and CloudWatch Logs

Open the IAM console.
Click Users > Create user.
Enter a name in the User name field (for example, CWExport).
Select both Programmatic access and AWS Management Console access.
Select either Autogenerated password or Custom password.
Click Next: Permissions.
Choose Attach existing policies directly.
Search for and select the AmazonS3FullAccess and CloudWatchLogsFullAccess policies to the user.
Click Next: Tags.
Click Next: Review.
Click Create user.

Configure permissions on Amazon S3 bucket

In the Amazon S3 console, choose the bucket that you previously created.
Click Permissions > Bucket policy.

In the Bucket Policy Editor, add the following policy.

{             
  "Version": "2012-10-17",
  "Statement": [
      {
        "Action": "s3:GetBucketAcl",
        "Effect": "Allow",
        "Resource": "arn:aws:s3:::cw-exported-logs",
        "Principal": { "Service": "logs.us-east-1.amazonaws.com" }
      },
      {
        "Action": "s3:PutObject" ,
        "Effect": "Allow",
        "Resource": "arn:aws:s3:::my-exported-logs/random-string/*",
        "Condition": { "StringEquals": { "s3:x-amz-acl": "bucket-owner-full-control" } },
        "Principal": { "Service": "logs.us-east-1.amazonaws.com" }

      }

  ]

}

Change and update the following json variables:
- Change cw-exported-logs to the name of your S3 bucket.
- Change random-string to a randomly generated string of characters.
- Be sure to specify the correct Region endpoint for Principal.
Click Save.

Configure CloudWatch Export

Sign in as the IAM user you previously created.
Open CloudWatch console.
In the navigation menu, select Log groups.
Select the name of an existing or create a new log group.
Choose Actions > Export data to Amazon S3.
On the Export data to Amazon S3 screen, locate Define data export.
Set the time range for the data to export using From and To.

Note: If your log group has multiple log streams, you can provide a log stream prefix to limit the log group data to a specific stream. Go to Advanced and enter the prefix in the Stream prefix field to target specific streams.
Choose S3 bucket: Select the account associated with the Amazon S3 bucket.
S3 bucket name: Select an Amazon S3 bucket.
S3 Bucket prefix: Enter the randomly generated string that you specified in the bucket policy.
Choose Export to export your log data to Amazon S3.
To view the status of the log data that you exported to Amazon S3, select Actions > View all exports to Amazon S3.

Configure a feed in Google SecOps to ingest AWS CloudWatch logs

Go to SIEM Settings > Feeds.
Click Add New Feed.
In the Feed name field, enter a name for the feed (for example, AWS CloudWatch Logs).
Select Amazon S3 V2 as the Source type.
Select AWS CloudWatch as the Log type.
Click Next.
Specify values for the following input parameters:
- S3 URI: The bucket URI
- s3://your-log-bucket-name/
  - Replace your-log-bucket-name with the actual name of the bucket.
- Source deletion options: Select the deletion option according to your preference.
  
  Note: If you select the Delete transferred files or Delete transferred files and empty directories option, make sure that you granted appropriate permissions to the service account.
- Maximum File Age: Default 180 Days.
- Access Key ID: User access key with access to the S3 bucket.
- Secret Access Key: User secret key with access to the S3 bucket.
- Asset namespace: The asset namespace.
- Ingestion labels: The label to be applied to the events from this feed.
Click Next.
Review your new feed configuration in the Finalize screen, and then click Submit.

Configure Cloudwatch logs export using AWS Data Firehose

This export process is not required to be done regularly after initial setup.

Configure a feed in Google SecOps to ingest AWS CloudWatch Logs

Go to SIEM Settings > Feeds.
Click + Add New Feed.
In the Feed name field, enter a name for the feed (for example, AWS CloudWatch Logs).
Select Amazon Data Firehose as the Source type.
Select AWS CloudWatch as the Log type.
Click Next.
Specify values for the following input parameters:
- Split delimiter: Optional \n.
- Asset namespace: The asset namespace.
- Ingestion labels: The label to be applied to the events from this feed.
Click Next.
Review the feed configuration and click Submit.
Click Generate Secret Key to generate a secret key to authenticate this feed.
Copy and save the secret key as you cannot view this value again.
Go to the Details tab.
Copy the feed endpoint URL from the Endpoint Information field.
Click Done.

Create an API key for the Amazon Data Firehose feed

Go to the Google Cloud console Credentials page.
Click Create credentials, and then select API key.
Restrict the API key access to the Google SecOps API.

Specify the endpoint URL

To specify the HTTPS endpoint and access key in Amazon Data Firehose, follow these steps:

Append the API key to the feed endpoint URL and specify this URL as the HTTP endpoint URL in the following format:
```
ENDPOINT_URL?key=API_KEY
```
- Replace the following:
  - ENDPOINT_URL: the feed endpoint URL.
  - API_KEY: the API key to authenticate to Google SecOps.

For the access key, specify the secret key that you obtained when you created the Amazon Data Firehose feed.

Configure Amazon Kinesis Data Firehose to Google SecOps {:#configure-kinesis-secops}.

In the AWS Console, go to Kinesis > Data Firehose > Create delivery stream.
Provide the following configuration details:
- Source: Select Direct PUT or other sources.
- Destination: Choose HTTP endpoint.
- HTTP endpoint URL: Enter the Feed HTTPS endpoint URL from Google SecOps with the API Key.
- HTTP method: Select POST.
Under Access key, enter the following details:
- Secret key header: <HEADER_NAME_FOR_SECRET> with value <YOUR_SECRET_KEY>
- Buffering hints: set Buffer size = 1 MiB, Buffer interval = 60 seconds.
- Compression: select Disabled.
- S3 backup: select Disabled.
- Leave retry and logging settings as default.
Click Create delivery stream.

Configure IAM permissions and subscribe the log group

In the AWS Console, go to IAM > Policies > Create policy > JSON.

Paste the following policy JSON, replacing <region> and <account-id> with your AWS Region and account ID:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "firehose:PutRecord",
        "firehose:PutRecordBatch"
      ],
      "Resource": "arn:aws:firehose:<region>:<account-id>:deliverystream/cwlogs-to-secops"
    }
  ]
}

Name the policy CWLtoFirehoseWrite and click Create policy.
Go to IAM > Roles > Create role.
Select Custom trust policy and paste:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "logs.<your-region>.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

Attach the policy CWLtoFirehoseWrite to the role.
Name the role CWLtoFirehoseRole and click Create role.
Go to CloudWatch > Logs > Log groups.
Select the target log group.
Open the Subscription filters tab and click Create.
Choose Create Amazon Kinesis Data Firehose subscription filter.
Provide the following configuration details:
- Destination: Select delivery stream cwlogs-to-secops.
- Grant permission: Choose role CWLtoFirehoseRole.
- Filter name: Enter all-events.
- Leave Filter pattern empty to send all events.
Click Start streaming.

UDM Mapping Table

Log Field	UDM Mapping	Logic
`account`	`principal.user.userid`	The value of `account` from the raw log is mapped to the `principal.user.userid` field.
`account_id`	`principal.user.userid`	The value of `account_id` from the raw log is mapped to the `principal.user.userid` field.
`AlertId`	`metadata.product_log_id`	The value of `AlertId` from the raw log is mapped to the `metadata.product_log_id` field.
`arrivalTimestamp`	`metadata.event_timestamp`	The value of `arrivalTimestamp` from the raw log is converted to a timestamp and mapped to the `metadata.event_timestamp` field.
`attemptsMade`	`additional.fields`	The value of `attemptsMade` from the raw log is converted to a string and added as a key-value pair to the `additional.fields` with the key "Attempts Made".
`awsAccountId`	`principal.asset_id`	The value of `awsAccountId` from the raw log is prepended with "AWS Account id: " and mapped to the `principal.asset_id` field.
`billed_duration`	`additional.fields`	The value of `billed_duration` from the raw log is added as a key-value pair to the `additional.fields` with the key "billed_duration".
`BytesIn`	`network.received_bytes`	The value of `BytesIn` from the raw log is converted to an unsigned integer and mapped to the `network.received_bytes` field.
`cipher`	`network.tls.cipher`	The value of `cipher` from the raw log is mapped to the `network.tls.cipher` field.
`Ciphers`	`network.tls.client.supported_ciphers`	The value of `Ciphers` from the raw log is split by commas and each value is added to the `network.tls.client.supported_ciphers` array.
`cloudwatchLog`	`security_result.description`	The value of `cloudwatchLog` from the raw log is mapped to the `security_result.description` field.
`CloudAccountId`	`metadata.product_deployment_id`	The value of `CloudAccountId` from the raw log is mapped to the `metadata.product_deployment_id` field.
`CloudType`	`target.resource.attribute.cloud.environment`	The value of `CloudType` from the raw log determines the value of `target.resource.attribute.cloud.environment`. If `CloudType` is "gcp", the value is "GOOGLE_CLOUD_PLATFORM". If `CloudType` is "aws", the value is "AMAZON_WEB_SERVICES". If `CloudType` is "azure", the value is "MICROSOFT_AZURE".
`Context.Execution.Id`	`target.resource.attribute.labels`	The value of `Context.Execution.Id` from the raw log is added as a key-value pair to the `target.resource.attribute.labels` with the key "Context Id".
`Context.Execution.Name`	`target.resource.attribute.labels`	The value of `Context.Execution.Name` from the raw log is added as a key-value pair to the `target.resource.attribute.labels` with the key "Context Name".
`Context.Execution.RoleArn`	`target.resource.product_object_id`	The value of `Context.Execution.RoleArn` from the raw log is mapped to the `target.resource.product_object_id` field.
`descr`	`metadata.description`	The value of `descr` from the raw log, after removing extra whitespace, is mapped to the `metadata.description` field unless it is "-". If `descr` is empty, the value of `log` is used instead.
`destination.name`	`target.location.country_or_region`	The value of `destination.name` from the raw log is mapped to the `target.location.country_or_region` field.
`destination.properties.prefix`	`target.resource.attribute.labels`	The value of `destination.properties.prefix` from the raw log is added as a key-value pair to the `target.resource.attribute.labels` with the key "Destination properties prefix".
`detail.additionalEventData.configRuleArn`	`security_result.rule_id`	The value of `detail.additionalEventData.configRuleArn` from the raw log is mapped to the `security_result.rule_id` field.
`detail.additionalEventData.configRuleName`	`security_result.rule_name`	The value of `detail.additionalEventData.configRuleName` from the raw log is mapped to the `security_result.rule_name` field.
`detail.additionalEventData.managedRuleIdentifier`	`additional.fields`	The value of `detail.additionalEventData.managedRuleIdentifier` from the raw log is added as a key-value pair to the `additional.fields` with the key "managedRuleIdentifier".
`detail.additionalEventData.notificationJobType`	`additional.fields`	The value of `detail.additionalEventData.notificationJobType` from the raw log is added as a key-value pair to the `additional.fields` with the key "notificationJobType".
`detail.awsAccountId`	`principal.asset_id`	The value of `detail.awsAccountId` from the raw log is prepended with "AWS Account id: " and mapped to the `principal.asset_id` field.
`detail.awsRegion`	`principal.location.name`	The value of `detail.awsRegion` from the raw log is mapped to the `principal.location.name` field.
`detail.configRuleArn`	`security_result.rule_id`	The value of `detail.configRuleArn` from the raw log is mapped to the `security_result.rule_id` field.
`detail.configRuleName`	`security_result.rule_name`	The value of `detail.configRuleName` from the raw log is mapped to the `security_result.rule_name` field.
`detail.configurationItem.awsAccountId`	`principal.user.userid`	The value of `detail.configurationItem.awsAccountId` from the raw log is mapped to the `principal.user.userid` field.
`detail.configurationItem.awsRegion`	`target.location.country_or_region`	The value of `detail.configurationItem.awsRegion` from the raw log is mapped to the `target.location.country_or_region` field.
`detail.configurationItem.configuration.complianceType`	`security_result.summary`	The value of `detail.configurationItem.configuration.complianceType` from the raw log is mapped to the `security_result.summary` field.
`detail.configurationItem.configuration.targetResourceId`	`target.resource.attribute.labels`	The value of `detail.configurationItem.configuration.targetResourceId` from the raw log is added as a key-value pair to the `target.resource.attribute.labels` with the key "configurationItem configuration targetResourceId".
`detail.configurationItem.configuration.targetResourceType`	`target.resource.attribute.labels`	The value of `detail.configurationItem.configuration.targetResourceType` from the raw log is added as a key-value pair to the `target.resource.attribute.labels` with the key "configurationItem configuration targetResourceType".
`detail.configurationItem.configurationItemCaptureTime`	`_target.asset.attribute.creation_time`	The value of `detail.configurationItem.configurationItemCaptureTime` from the raw log is converted to a timestamp and mapped to the `_target.asset.attribute.creation_time` field.
`detail.configurationItem.configurationItemStatus`	`target.resource.attribute.labels`	The value of `detail.configurationItem.configurationItemStatus` from the raw log is added as a key-value pair to the `target.resource.attribute.labels` with the key "configurationItem configurationItemStatus".
`detail.configurationItem.configurationStateId`	`target.resource.attribute.labels`	The value of `detail.configurationItem.configurationStateId` from the raw log is converted to a string and added as a key-value pair to the `target.resource.attribute.labels` with the key "configurationItem configurationStateId".
`detail.configurationItem.resourceId`	`target.resource.id`	The value of `detail.configurationItem.resourceId` from the raw log is mapped to the `target.resource.id` field.
`detail.configurationItem.resourceType`	`target.resource.resource_subtype`	The value of `detail.configurationItem.resourceType` from the raw log is mapped to the `target.resource.resource_subtype` field.
`detail.configurationItemDiff.changedProperties.Configuration.configRuleList.1.updatedValue.configRuleArn`	`security_result.rule_id`	The value of `detail.configurationItemDiff.changedProperties.Configuration.configRuleList.1.updatedValue.configRuleArn` from the raw log is mapped to the `security_result.rule_id` field.
`detail.eventCategory`	`security_result.category_details`	The value of `detail.eventCategory` from the raw log is mapped to the `security_result.category_details` field.
`detail.eventID`	`metadata.product_log_id`	The value of `detail.eventID` from the raw log is mapped to the `metadata.product_log_id` field.
`detail.eventName`	`additional.fields`	The value of `detail.eventName` from the raw log is added as a key-value pair to the `additional.fields` with the key "Event Name".
`detail.eventSource`	`target.application`	The value of `detail.eventSource` from the raw log is mapped to the `target.application` field.
`detail.eventType`	`additional.fields`	The value of `detail.eventType` from the raw log is added as a key-value pair to the `additional.fields` with the key "Event Type".
`detail.eventVersion`	`metadata.product_version`	The value of `detail.eventVersion` from the raw log is mapped to the `metadata.product_version` field.
`detail.managementEvent`	`additional.fields`	The value of `detail.managementEvent` from the raw log is converted to a string and added as a key-value pair to the `additional.fields` with the key "detail managementEvent".
`detail.messageType`	`target.resource.attribute.labels`	The value of `detail.messageType` from the raw log is added as a key-value pair to the `target.resource.attribute.labels` with the key "Message Type".
`detail.newEvaluationResult.complianceType`	`security_result.summary`	The value of `detail.newEvaluationResult.complianceType` from the raw log is mapped to the `security_result.summary` field.
`detail.newEvaluationResult.configRuleInvokedTime`	`additional.fields`	The value of `detail.newEvaluationResult.configRuleInvokedTime` from the raw log is added as a key-value pair to the `additional.fields` with the key "newEvaluationResult_configRuleInvokedTime".
`detail.newEvaluationResult.evaluationResultIdentifier.evaluationResultQualifier.configRuleName`	`additional.fields`	The value of `detail.newEvaluationResult.evaluationResultIdentifier.evaluationResultQualifier.configRuleName` from the raw log is added as a key-value pair to the `additional.fields` with the key "newEvaluationResult_configRuleName".
`detail.newEvaluationResult.evaluationResultIdentifier.evaluationResultQualifier.resourceId`	`additional.fields`	The value of `detail.newEvaluationResult.evaluationResultIdentifier.evaluationResultQualifier.resourceId` from the raw log is added as a key-value pair to the `additional.fields` with the key "newEvaluationResult_resourceId".
`detail.newEvaluationResult.evaluationResultIdentifier.evaluationResultQualifier.resourceType`	`additional.fields`	The value of `detail.newEvaluationResult.evaluationResultIdentifier.evaluationResultQualifier.resourceType` from the raw log is added as a key-value pair to the `additional.fields` with the key "newEvaluationResult_resourceType".
`detail.newEvaluationResult.resultRecordedTime`	`additional.fields`	The value of `detail.newEvaluationResult.resultRecordedTime` from the raw log is added as a key-value pair to the `additional.fields` with the key "newEvaluationResult_resultRecordedTime".
`detail.oldEvaluationResult.configRuleInvokedTime`	`additional.fields`	The value of `detail.oldEvaluationResult.configRuleInvokedTime` from the raw log is added as a key-value pair to the `additional.fields` with the key "oldEvaluationResult_configRuleInvokedTime".
`detail.oldEvaluationResult.evaluationResultIdentifier.evaluationResultQualifier.configRuleName`	`additional.fields`	The value of `detail.oldEvaluationResult.evaluationResultIdentifier.evaluationResultQualifier.configRuleName` from the raw log is added as a key-value pair to the `additional.fields` with the key "oldEvaluationResult_configRuleName".
`detail.oldEvaluationResult.evaluationResultIdentifier.evaluationResultQualifier.resourceId`	`additional.fields`	The value of `detail.oldEvaluationResult.evaluationResultIdentifier.evaluationResultQualifier.resourceId` from the raw log is added as a key-value pair to the `additional.fields` with the key "oldEvaluationResult_resourceId".
`detail.oldEvaluationResult.evaluationResultIdentifier.evaluationResultQualifier.resourceType`	`additional.fields`	The value of `detail.oldEvaluationResult.evaluationResultIdentifier.evaluationResultQualifier.resourceType` from the raw log is added as a key-value pair to the `additional.fields` with the key "oldEvaluationResult_resourceType".
`detail.oldEvaluationResult.resultRecordedTime`	`additional.fields`	The value of `detail.oldEvaluationResult.resultRecordedTime` from the raw log is added as a key-value pair to the `additional.fields` with the key "oldEvaluationResult_resultRecordedTime".
`detail.readOnly`	`additional.fields`	The value of `detail.readOnly` from the raw log is converted to a string and added as a key-value pair to the `additional.fields` with the key "detail readOnly".
`detail.recipientAccountId`	`target.resource.attribute.labels`	The value of `detail.recipientAccountId` from the raw log is added as a key-value pair to the `target.resource.attribute.labels` with the key "Recipient Account Id".
`detail.recordVersion`	`metadata.product_version`	The value of `detail.recordVersion` from the raw log is mapped to the `metadata.product_version` field.
`detail.requestID`	`target.resource.attribute.labels`	The value of `detail.requestID` from the raw log is added as a key-value pair to the `target.resource.attribute.labels` with the key "Detail Request ID".
`detail.resourceType`	`target.resource.resource_subtype`	The value of `detail.resourceType` from the raw log is mapped to the `target.resource.resource_subtype` field.
`detail.s3Bucket`	`about.resource.name`	The value of `detail.s3Bucket` from the raw log is mapped to the `about.resource.name` field.
`detail.s3ObjectKey`	`target.resource.attribute.labels`	The value of `detail.s3ObjectKey` from the raw log is added as a key-value pair to the `target.resource.attribute.labels` with the key "s3ObjectKey".
`detail.userAgent`	`network.http.user_agent`	The value of `detail.userAgent` from the raw log is mapped to the `network.http.user_agent` field.
`detail.userIdentity.accessKeyId`	`target.user.userid`	The value of `detail.userIdentity.accessKeyId` from the raw log is mapped to the `target.user.userid` field.
`detail.userIdentity.accountId`	`metadata.product_deployment_id`	The value of `detail.userIdentity.accountId` from the raw log is mapped to the `metadata.product_deployment_id` field.
`detail.userIdentity.arn`	`target.user.userid`	The value of `detail.userIdentity.arn` from the raw log is mapped to the `target.user.userid` field.
`detail.userIdentity.principalId`	`principal.user.product_object_id`	The value of `detail.userIdentity.principalId` from the raw log is mapped to the `principal.user.product_object_id` field.
`detail.userIdentity.sessionContext.attributes.mfaAuthenticated`	`principal.user.attribute.labels`	The value of `detail.userIdentity.sessionContext.attributes.mfaAuthenticated` from the raw log is added as a key-value pair to the `principal.user.attribute.labels` with the key "mfaAuthenticated".
`detail.userIdentity.sessionContext.sessionIssuer.userName`	`target.user.user_display_name`	The value of `detail.userIdentity.sessionContext.sessionIssuer.userName` from the raw log is mapped to the `target.user.user_display_name` field.
`detail.userIdentity.type`	`principal.resource.type`	The value of `detail.userIdentity.type` from the raw log is mapped to the `principal.resource.type` field.
`detail-type`	`metadata.product_event_type`	The value of `detail-type` from the raw log is mapped to the `metadata.product_event_type` field.
`device`	`principal.asset.product_object_id`	The value of `device` from the raw log is mapped to the `principal.asset.product_object_id` field.
`digestPublicKeyFingerprint`	`target.file.sha1`	The value of `digestPublicKeyFingerprint` from the raw log is mapped to the `target.file.sha1` field.
`digestS3Bucket`	`principal.resource.name`	The value of `digestS3Bucket` from the raw log is mapped to the `principal.resource.name` field.
`digestS3Object`	`principal.asset.asset_id`	The value of `digestS3Object` from the raw log is prepended with "S3 Object: " and mapped to the `principal.asset.asset_id` field.
`digestSignatureAlgorithm`	`network.tls.cipher`	The value of `digestSignatureAlgorithm` from the raw log is mapped to the `network.tls.cipher` field.
`digestStartTime`	`metadata.event_timestamp`	The value of `digestStartTime` from the raw log is converted to a timestamp and mapped to the `metadata.event_timestamp` field.
`dimensions.VolumeId`	`additional.fields`	The value of `dimensions.VolumeId` from the raw log is added as a key-value pair to the `additional.fields` with the key "VolumeId".
`duration`	`additional.fields`	The value of `duration` from the raw log is added as a key-value pair to the `additional.fields` with the key "duration".
`errorCode`	`security_result.rule_name`	The value of `errorCode` from the raw log is mapped to the `security_result.rule_name` field.
`errorMessage`	`security_result.summary`	The value of `errorMessage` from the raw log is mapped to the `security_result.summary` field.
`executionId`	`principal.process.pid`	The value of `executionId` from the raw log is mapped to the `principal.process.pid` field.
`host`	`principal.hostname`, `principal.ip`	The value of `host` from the raw log, with hyphens replaced by dots, is parsed as an IP address and mapped to the `principal.ip` field if successful. Otherwise, it is mapped to the `principal.hostname` field.
`http_verb`	`network.http.method`	The value of `http_verb` from the raw log is converted to uppercase and mapped to the `network.http.method` field.
`kubernetes.container_hash`	`additional.fields`	The value of `kubernetes.container_hash` from the raw log is added as a key-value pair to the `additional.fields` with the key "container_hash".
`kubernetes.container_image`	`additional.fields`	The value of `kubernetes.container_image` from the raw log is added as a key-value pair to the `additional.fields` with the key "container_image".
`kubernetes.container_name`	`additional.fields`	The value of `kubernetes.container_name` from the raw log is added as a key-value pair to the `additional.fields` with the key "container_name".
`kubernetes.docker_id`	`principal.asset_id`	The value of `kubernetes.docker_id` from the raw log is prepended with "id: " and mapped to the `principal.asset_id` field.
`kubernetes.host`	`principal.hostname`, `principal.ip`	The value of `kubernetes.host` from the raw log, with hyphens replaced by dots, is parsed as an IP address and mapped to the `principal.ip` field if successful. Otherwise, it is mapped to the `principal.hostname` field.
`kubernetes.namespace`	`principal.namespace`	The value of `kubernetes.namespace` from the raw log is mapped to the `principal.namespace` field.
`kubernetes.namespace_name`	`principal.namespace`	The value of `kubernetes.namespace_name` from the raw log is mapped to the `principal.namespace` field.
`kubernetes.pod_id`	`principal.asset.asset_id`	The value of `kubernetes.pod_id` from the raw log is prepended with "pod_id: " and mapped to the `principal.asset.asset_id` field.
`kubernetes.pod_name`	`additional.fields`	The value of `kubernetes.pod_name` from the raw log is added as a key-value pair to the `additional.fields` with the key "pod name".
`lambdaArn`	`principal.hostname`	The value of `lambdaArn` from the raw log is mapped to the `principal.hostname` field.
`level`	`security_result.severity`	The value of `level` from the raw log determines the value of `security_result.severity`. If `level` is "Info", the value is "INFORMATIONAL". If `level` is "Error", the value is "ERROR". If `level` is "Warning", the value is "MEDIUM".
`log`	`metadata.description`	The value of `log` from the raw log is mapped to the `metadata.description` field if `descr` is empty.
`logFiles`	`about`	For each element in the `logFiles` array from the raw log, an `about` object is created with `file.full_path` set to `s3Object`, `asset.hostname` set to `s3Bucket`, and `file.sha256` set to `hashValue`.
`log_processed.cause`	`security_result.summary`	The value of `log_processed.cause` from the raw log is mapped to the `security_result.summary` field.
`log_processed.ids`	`intermediary.hostname`	For each element in the `log_processed.ids` array from the raw log, an `intermediary` object is created with `hostname` set to the element's value.
`log_processed.level`	`security_result.severity`	The value of `log_processed.level` from the raw log is mapped to the `security_result.severity` field.
`log_processed.msg`	`metadata.description`	The value of `log_processed.msg` from the raw log is mapped to the `metadata.description` field.
`log_processed.ts`	`metadata.event_timestamp`	The value of `log_processed.ts` from the raw log is converted to a timestamp and mapped to the `metadata.event_timestamp` field.
`log_type`	`metadata.log_type`	The value of `log_type` from the raw log is mapped to the `metadata.log_type` field. This is a custom field added for context.
`logevent.message`	`security_result.description`	The value of `logevent.message` from the raw log is mapped to the `security_result.description` field. It is also parsed using grok to extract additional fields.
`logGroup`	`security_result.about.resource.name`	The value of `logGroup` from the raw log is mapped to the `security_result.about.resource.name` field.
`logStream`	`security_result.about.resource.attribute.labels`	The value of `logStream` from the raw log is added as a key-value pair to the `security_result.about.resource.attribute.labels` with the key "logStream".
`memory_used`	`additional.fields`	The value of `memory_used` from the raw log is added as a key-value pair to the `additional.fields` with the key "memory_used".
`metric_name`	`additional.fields`	The value of `metric_name` from the raw log is added as a key-value pair to the `additional.fields` with the key "metric_name".
`metric_stream_name`	`additional.fields`	The value of `metric_stream_name` from the raw log is added as a key-value pair to the `additional.fields` with the key "metric_stream_name".
`namespace`	`principal.namespace`	The value of `namespace` from the raw log is mapped to the `principal.namespace` field.
`owner`	`principal.user.userid`	The value of `owner` from the raw log is mapped to the `principal.user.userid` field.
`parameters`	`additional.fields`	The value of `parameters` from the raw log is added as a key-value pair to the `additional.fields` with the key "Parameters".
`Path`	`principal.process.file.full_path`	The value of `Path` from the raw log is mapped to the `principal.process.file.full_path` field.
`pid`	`principal.process.pid`	The value of `pid` from the raw log is mapped to the `principal.process.pid` field.
`PolicyName`	`security_result.rule_name`	The value of `PolicyName` from the raw log is mapped to the `security_result.rule_name` field.
`prin_host`	`principal.hostname`	The value of `prin_host` from the raw log is mapped to the `principal.hostname` field.
`principal_hostname`	`principal.hostname`	The value of `principal_hostname` from the raw log is mapped to the `principal.hostname` field.
`process`	`principal.application`	The value of `process` from the raw log is mapped to the `principal.application` field.
`rawData`	`additional.fields`	The value of `rawData` from the raw log is added as a key-value pair to the `additional.fields` with the key "Raw Data".
`Recommendation`	`security_result.detection_fields`	The value of `Recommendation` from the raw log is added as a key-value pair to the `security_result.detection_fields` with the key "Recommendation".
`referral_url`	`network.http.referral_url`	The value of `referral_url` from the raw log is mapped to the `network.http.referral_url` field.
`region`	`principal.location.name`	The value of `region` from the raw log is mapped to the `principal.location.name` field.
`resp_code`	`network.http.response_code`	The value of `resp_code` from the raw log is converted to an integer and mapped to the `network.http.response_code` field.
`resource_url`	`network.http.referral_url`	The value of `resource_url` from the raw log is mapped to the `network.http.referral_url` field.
`ResourceType`	`target.resource.resource_subtype`	The value of `ResourceType` from the raw log is mapped to the `target.resource.resource_subtype` field.
`response_body`	`additional.fields`	The value of `response_body` from the raw log is added as a key-value pair to the `additional.fields` with the key "Response body".
`Role`	`target.resource.product_object_id`	The value of `Role` from the raw log is mapped to the `target.resource.product_object_id` field.
`s3_bucket_path`	`target.file.full_path`	The value of `s3_bucket_path` from the raw log is mapped to the `target.file.full_path` field.
`sec_result.category`	`security_result.category`	The value of `sec_result.category` is derived from the parser logic. If `descr` contains "authentication is required", the value is "AUTH_VIOLATION".
`sec_result.description`	`security_result.description`	The value of `sec_result.description` is derived from the parser logic. It is set to the value of `cloudwatchLog` if present.
`sec_result.severity`	`security_result.severity`	The value of `sec_result.severity` is derived from the parser logic. It is set based on the value of `severity` or `level`.
`sec_result.summary`	`security_result.summary`	The value of `sec_result.summary` is derived from the parser logic. It is set to the value of `log_processed.cause` or `errorMessage` if present.
`security_result`	`security_result`	The `security_result` object is constructed from various fields and parser logic.
`serverId`	`additional.fields`	The value of `serverId` from the raw log is added as a key-value pair to the `additional.fields` with the key "server_id".
`severity`	`security_result.severity`	The value of `severity` from the raw log, converted to uppercase and normalized, is mapped to the `security_result.severity` field.
`Source`	`principal.hostname`	The value of `Source` from the raw log is mapped to the `principal.hostname` field.
`source`	`principal.hostname`	The value of `source` from the raw log is mapped to the `principal.hostname` field.
`SourceIP`	`principal.ip`	The value of `SourceIP` from the raw log is mapped to the `principal.ip` field.
`src_port`	`principal.port`	If `src_port` is "80", it is converted to an integer and mapped to the `principal.port` field, and `network.application_protocol` is set to "HTTP".
`stream`	`additional.fields`	The value of `stream` from the raw log is added as a key-value pair to the `additional.fields` with the key "stream".
`subscriptionFilters`	`security_result.about.resource.attribute.labels`	For each element in the `subscriptionFilters` array from the raw log, a key-value pair is added to the `security_result.about.resource.attribute.labels` with the key "subscriptionFilter" and the value from the array.
`support_contact`	`target.resource.attribute.labels`	The value of `support_contact` from the raw log is added as a key-value pair to the `target.resource.attribute.labels` with the key "Support Contact".
`t_ip`	`target.ip`	The value of `t_ip` from the raw log, after removing hyphens, is parsed as an IP address and mapped to the `target.ip` field if successful.
`time`	`metadata.event_timestamp`	The value of `time` from the raw log is converted to a timestamp and mapped to the `metadata.event_timestamp` field.
`timestamp`	`metadata.event_timestamp`	The value of `timestamp` from the raw log is converted to a timestamp using various formats and mapped to the `metadata.event_timestamp` field.
`tls`	`network.tls.version`	The value of `tls` from the raw log is mapped to the `network.tls.version` field.
`transferDetails.serverId`	`additional.fields`	The value of `transferDetails.serverId` from the raw log is added as a key-value pair to the `additional.fields` with the key "server_id".
`transferDetails.sessionId`	`network.session_id`	The value of `transferDetails.sessionId` from the raw log is mapped to the `network.session_id` field.
`transferDetails.username`	`principal.user.user_display_name`	The value of `transferDetails.username` from the raw log is mapped to the `principal.user.user_display_name` field.
`ts`	`metadata.event_timestamp`	The value of `ts` from the raw log, combined with the timezone if available, is converted to a timestamp and mapped to the `metadata.event_timestamp` field.
`type`	`metadata.product_event_type`	The value of `type` from the raw log is mapped to the `metadata.product_event_type` field.
`unit`	`additional.fields`	The value of `unit` from the raw log is added as a key-value pair to the `additional.fields` with the key "unit".
`url`	`target.url`	The value of `url` from the raw log is mapped to the `target.url` field.
`url_back_to_product`	`metadata.url_back_to_product`	The value of `url_back_to_product` from the raw log is mapped to the `metadata.url_back_to_product` field.
`User`	`principal.user.userid`	The value of `User` from the raw log is mapped to the `principal.user.userid` field.
`user`	`target.user.userid`, `metadata.event_type`, `extensions.auth.mechanism`	If `user` is present, `metadata.event_type` is set to "USER_LOGIN", `extensions.auth.mechanism` is set to "NETWORK", and the value of `user` is mapped to `target.user.userid`.
`value.count`	`additional.fields`	The value of `value.count` from the raw log is converted to a string and added as a key-value pair to the `additional.fields` with the key "count".
`value.max`	`additional.fields`	The value of `value.max` from the raw log is converted to a string and added as a key-value pair to the `additional.fields` with the key "max".
`value.min`	`additional.fields`	The value of `value.min` from the raw log is converted to a string and added as a key-value pair to the `additional.fields` with the key "min".
`value.sum`	`additional.fields`	The value of `value.sum` from the raw log is converted to a string and added as a key-value pair to the `additional.fields` with the key "sum".
`workflowId`	`additional.fields`	The value of `workflowId` from the raw log is added as a key-value pair to the `additional.fields` with the key "workflowId".

Need more help? Get answers from Community members and Google SecOps professionals.