Collect AWS CloudWatch logs

Supported in:

Google SecOps SIEM

This document explains how to ingest AWS CloudWatch logs to Google Security Operations. AWS CloudWatch is a monitoring and observability service, which collects operational data in the form of logs, metrics, and events. This integration lets you send these logs to Google SecOps for analysis and monitoring.

Before You Begin

Ensure that you have a Google SecOps instance.
Ensure that you have privileged access to AWS.

Create an Amazon S3 bucket

It's recommended to use a bucket that was created specifically for CloudWatch logs.

Open the Amazon S3 console.
If required, you can change the Region.
- From the navigation bar, select the Region where your CloudWatch Logs reside.
Click Create Bucket.
- Bucket Name: enter a meaningful name for the bucket.
- Region: select the region where your CloudWatch Logs data resides.
- Click Create.

Create an IAM user with full access to Amazon S3 and CloudWatch Logs

Open the IAM console.
Click Users > Add user.
Enter a user name (for example, CWExport).
Select Programmatic access and AWS Management Console access.
Select either Autogenerated password or Custom password.
Click Next: Permissions.
Choose Attach existing policies directly.
Search for and select the AmazonS3FullAccess and CloudWatchLogsFullAccess policies to the user.
Click Next: Tags.
Click Next: Review.
Click Create user.

Configure permissions on Amazon S3 bucket

In the Amazon S3 console, choose the bucket that you previously created.
Click Permissions > Bucket policy.

In the Bucket Policy Editor, add the following policy.

{             
  "Version": "2012-10-17",
  "Statement": [
      {
        "Action": "s3:GetBucketAcl",
        "Effect": "Allow",
        "Resource": "arn:aws:s3:::cw-exported-logs",
        "Principal": { "Service": "logs.us-east-1.amazonaws.com" }
      },
      {
        "Action": "s3:PutObject" ,
        "Effect": "Allow",
        "Resource": "arn:aws:s3:::my-exported-logs/random-string/*",
        "Condition": { "StringEquals": { "s3:x-amz-acl": "bucket-owner-full-control" } },
        "Principal": { "Service": "logs.us-east-1.amazonaws.com" }

      }

  ]

}

Change and update the following json variables:
- Change cw-exported-logs to the name of your S3 bucket.
- Change random-string to a randomly generated string of characters.
- Be sure to specify the correct Region endpoint for Principal.
Click Save to set the policy that you just added as the access policy on your bucket.

Configure CloudWatch Export

Sign in as the IAM user you previously created.
Open CloudWatch console.
In the navigation menu, select Log groups.
Select the name of an existing log group or create a new log group.
Choose Actions > Export data to Amazon S3.
On the Export data to Amazon S3 screen, locate Define data export.
Set the time range for the data to export using From and To.

Choose S3 bucket: select the account associated with the Amazon S3 bucket.
S3 bucket name: select an Amazon S3 bucket.
S3 Bucket prefix: enter the randomly generated string that you specified in the bucket policy.
Choose Export to export your log data to Amazon S3.
To view the status of the log data that you exported to Amazon S3, select Actions > View all exports to Amazon S3.

Configure a feed in Google SecOps to ingest AWS CloudWatch logs

Go to SIEM Settings > Feeds.
Click Add new.
In the Feed name field, enter a name for the feed (for example, AWS CloudWatch Logs).
Select Amazon S3 as the Source type.
Select AWS CloudWatch as the Log type.
Click Next.
Specify values for the following input parameters:
- Region: the region where the Amazon S3 bucket is located.
- S3 URI: the bucket URI.
  - s3://your-log-bucket-name/
    - Replace your-log-bucket-name with the actual name of the bucket.
- URI is a: select Directory or Directory which includes subdirectories.
- Source deletion options: select the deletion option according to your preference.
  
  Note: If you select the Delete transferred files or Delete transferred files and empty directories option, make sure that you granted appropriate permissions to the service account.
- Access Key ID: the User access key with access to the S3 bucket.
- Secret Access Key: the User secret key with access to the S3 bucket.
- Asset namespace: the asset namespace.
- Ingestion labels: the label to be applied to the events from this feed.
Click Next.
Review your new feed configuration in the Finalize screen, and then click Submit.

UDM Mapping Table

Log Field	UDM Mapping	Logic
`account`	`principal.user.userid`	The value of `account` from the raw log is mapped to the `principal.user.userid` field.
`account_id`	`principal.user.userid`	The value of `account_id` from the raw log is mapped to the `principal.user.userid` field.
`AlertId`	`metadata.product_log_id`	The value of `AlertId` from the raw log is mapped to the `metadata.product_log_id` field.
`arrivalTimestamp`	`metadata.event_timestamp`	The value of `arrivalTimestamp` from the raw log is converted to a timestamp and mapped to the `metadata.event_timestamp` field.
`attemptsMade`	`additional.fields`	The value of `attemptsMade` from the raw log is converted to a string and added as a key-value pair to the `additional.fields` with the key "Attempts Made".
`awsAccountId`	`principal.asset_id`	The value of `awsAccountId` from the raw log is prepended with "AWS Account id: " and mapped to the `principal.asset_id` field.
`billed_duration`	`additional.fields`	The value of `billed_duration` from the raw log is added as a key-value pair to the `additional.fields` with the key "billed_duration".
`BytesIn`	`network.received_bytes`	The value of `BytesIn` from the raw log is converted to an unsigned integer and mapped to the `network.received_bytes` field.
`cipher`	`network.tls.cipher`	The value of `cipher` from the raw log is mapped to the `network.tls.cipher` field.
`Ciphers`	`network.tls.client.supported_ciphers`	The value of `Ciphers` from the raw log is split by commas and each value is added to the `network.tls.client.supported_ciphers` array.
`cloudwatchLog`	`security_result.description`	The value of `cloudwatchLog` from the raw log is mapped to the `security_result.description` field.
`CloudAccountId`	`metadata.product_deployment_id`	The value of `CloudAccountId` from the raw log is mapped to the `metadata.product_deployment_id` field.
`CloudType`	`target.resource.attribute.cloud.environment`	The value of `CloudType` from the raw log determines the value of `target.resource.attribute.cloud.environment`. If `CloudType` is "gcp", the value is "GOOGLE_CLOUD_PLATFORM". If `CloudType` is "aws", the value is "AMAZON_WEB_SERVICES". If `CloudType` is "azure", the value is "MICROSOFT_AZURE".
`Context.Execution.Id`	`target.resource.attribute.labels`	The value of `Context.Execution.Id` from the raw log is added as a key-value pair to the `target.resource.attribute.labels` with the key "Context Id".
`Context.Execution.Name`	`target.resource.attribute.labels`	The value of `Context.Execution.Name` from the raw log is added as a key-value pair to the `target.resource.attribute.labels` with the key "Context Name".
`Context.Execution.RoleArn`	`target.resource.product_object_id`	The value of `Context.Execution.RoleArn` from the raw log is mapped to the `target.resource.product_object_id` field.
`descr`	`metadata.description`	The value of `descr` from the raw log, after removing extra whitespace, is mapped to the `metadata.description` field unless it is "-". If `descr` is empty, the value of `log` is used instead.
`destination.name`	`target.location.country_or_region`	The value of `destination.name` from the raw log is mapped to the `target.location.country_or_region` field.
`destination.properties.prefix`	`target.resource.attribute.labels`	The value of `destination.properties.prefix` from the raw log is added as a key-value pair to the `target.resource.attribute.labels` with the key "Destination properties prefix".
`detail.additionalEventData.configRuleArn`	`security_result.rule_id`	The value of `detail.additionalEventData.configRuleArn` from the raw log is mapped to the `security_result.rule_id` field.
`detail.additionalEventData.configRuleName`	`security_result.rule_name`	The value of `detail.additionalEventData.configRuleName` from the raw log is mapped to the `security_result.rule_name` field.
`detail.additionalEventData.managedRuleIdentifier`	`additional.fields`	The value of `detail.additionalEventData.managedRuleIdentifier` from the raw log is added as a key-value pair to the `additional.fields` with the key "managedRuleIdentifier".
`detail.additionalEventData.notificationJobType`	`additional.fields`	The value of `detail.additionalEventData.notificationJobType` from the raw log is added as a key-value pair to the `additional.fields` with the key "notificationJobType".
`detail.awsAccountId`	`principal.asset_id`	The value of `detail.awsAccountId` from the raw log is prepended with "AWS Account id: " and mapped to the `principal.asset_id` field.
`detail.awsRegion`	`principal.location.name`	The value of `detail.awsRegion` from the raw log is mapped to the `principal.location.name` field.
`detail.configRuleArn`	`security_result.rule_id`	The value of `detail.configRuleArn` from the raw log is mapped to the `security_result.rule_id` field.
`detail.configRuleName`	`security_result.rule_name`	The value of `detail.configRuleName` from the raw log is mapped to the `security_result.rule_name` field.
`detail.configurationItem.awsAccountId`	`principal.user.userid`	The value of `detail.configurationItem.awsAccountId` from the raw log is mapped to the `principal.user.userid` field.
`detail.configurationItem.awsRegion`	`target.location.country_or_region`	The value of `detail.configurationItem.awsRegion` from the raw log is mapped to the `target.location.country_or_region` field.
`detail.configurationItem.configuration.complianceType`	`security_result.summary`	The value of `detail.configurationItem.configuration.complianceType` from the raw log is mapped to the `security_result.summary` field.
`detail.configurationItem.configuration.targetResourceId`	`target.resource.attribute.labels`	The value of `detail.configurationItem.configuration.targetResourceId` from the raw log is added as a key-value pair to the `target.resource.attribute.labels` with the key "configurationItem configuration targetResourceId".
`detail.configurationItem.configuration.targetResourceType`	`target.resource.attribute.labels`	The value of `detail.configurationItem.configuration.targetResourceType` from the raw log is added as a key-value pair to the `target.resource.attribute.labels` with the key "configurationItem configuration targetResourceType".
`detail.configurationItem.configurationItemCaptureTime`	`_target.asset.attribute.creation_time`	The value of `detail.configurationItem.configurationItemCaptureTime` from the raw log is converted to a timestamp and mapped to the `_target.asset.attribute.creation_time` field.
`detail.configurationItem.configurationItemStatus`	`target.resource.attribute.labels`	The value of `detail.configurationItem.configurationItemStatus` from the raw log is added as a key-value pair to the `target.resource.attribute.labels` with the key "configurationItem configurationItemStatus".
`detail.configurationItem.configurationStateId`	`target.resource.attribute.labels`	The value of `detail.configurationItem.configurationStateId` from the raw log is converted to a string and added as a key-value pair to the `target.resource.attribute.labels` with the key "configurationItem configurationStateId".
`detail.configurationItem.resourceId`	`target.resource.id`	The value of `detail.configurationItem.resourceId` from the raw log is mapped to the `target.resource.id` field.
`detail.configurationItem.resourceType`	`target.resource.resource_subtype`	The value of `detail.configurationItem.resourceType` from the raw log is mapped to the `target.resource.resource_subtype` field.
`detail.configurationItemDiff.changedProperties.Configuration.configRuleList.1.updatedValue.configRuleArn`	`security_result.rule_id`	The value of `detail.configurationItemDiff.changedProperties.Configuration.configRuleList.1.updatedValue.configRuleArn` from the raw log is mapped to the `security_result.rule_id` field.
`detail.eventCategory`	`security_result.category_details`	The value of `detail.eventCategory` from the raw log is mapped to the `security_result.category_details` field.
`detail.eventID`	`metadata.product_log_id`	The value of `detail.eventID` from the raw log is mapped to the `metadata.product_log_id` field.
`detail.eventName`	`additional.fields`	The value of `detail.eventName` from the raw log is added as a key-value pair to the `additional.fields` with the key "Event Name".
`detail.eventSource`	`target.application`	The value of `detail.eventSource` from the raw log is mapped to the `target.application` field.
`detail.eventType`	`additional.fields`	The value of `detail.eventType` from the raw log is added as a key-value pair to the `additional.fields` with the key "Event Type".
`detail.eventVersion`	`metadata.product_version`	The value of `detail.eventVersion` from the raw log is mapped to the `metadata.product_version` field.
`detail.managementEvent`	`additional.fields`	The value of `detail.managementEvent` from the raw log is converted to a string and added as a key-value pair to the `additional.fields` with the key "detail managementEvent".
`detail.messageType`	`target.resource.attribute.labels`	The value of `detail.messageType` from the raw log is added as a key-value pair to the `target.resource.attribute.labels` with the key "Message Type".
`detail.newEvaluationResult.complianceType`	`security_result.summary`	The value of `detail.newEvaluationResult.complianceType` from the raw log is mapped to the `security_result.summary` field.
`detail.newEvaluationResult.configRuleInvokedTime`	`additional.fields`	The value of `detail.newEvaluationResult.configRuleInvokedTime` from the raw log is added as a key-value pair to the `additional.fields` with the key "newEvaluationResult_configRuleInvokedTime".
`detail.newEvaluationResult.evaluationResultIdentifier.evaluationResultQualifier.configRuleName`	`additional.fields`	The value of `detail.newEvaluationResult.evaluationResultIdentifier.evaluationResultQualifier.configRuleName` from the raw log is added as a key-value pair to the `additional.fields` with the key "newEvaluationResult_configRuleName".
`detail.newEvaluationResult.evaluationResultIdentifier.evaluationResultQualifier.resourceId`	`additional.fields`	The value of `detail.newEvaluationResult.evaluationResultIdentifier.evaluationResultQualifier.resourceId` from the raw log is added as a key-value pair to the `additional.fields` with the key "newEvaluationResult_resourceId".
`detail.newEvaluationResult.evaluationResultIdentifier.evaluationResultQualifier.resourceType`	`additional.fields`	The value of `detail.newEvaluationResult.evaluationResultIdentifier.evaluationResultQualifier.resourceType` from the raw log is added as a key-value pair to the `additional.fields` with the key "newEvaluationResult_resourceType".
`detail.newEvaluationResult.resultRecordedTime`	`additional.fields`	The value of `detail.newEvaluationResult.resultRecordedTime` from the raw log is added as a key-value pair to the `additional.fields` with the key "newEvaluationResult_resultRecordedTime".
`detail.oldEvaluationResult.configRuleInvokedTime`	`additional.fields`	The value of `detail.oldEvaluationResult.configRuleInvokedTime` from the raw log is added as a key-value pair to the `additional.fields` with the key "oldEvaluationResult_configRuleInvokedTime".
`detail.oldEvaluationResult.evaluationResultIdentifier.evaluationResultQualifier.configRuleName`	`additional.fields`	The value of `detail.oldEvaluationResult.evaluationResultIdentifier.evaluationResultQualifier.configRuleName` from the raw log is added as a key-value pair to the `additional.fields` with the key "oldEvaluationResult_configRuleName".
`detail.oldEvaluationResult.evaluationResultIdentifier.evaluationResultQualifier.resourceId`	`additional.fields`	The value of `detail.oldEvaluationResult.evaluationResultIdentifier.evaluationResultQualifier.resourceId` from the raw log is added as a key-value pair to the `additional.fields` with the key "oldEvaluationResult_resourceId".
`detail.oldEvaluationResult.evaluationResultIdentifier.evaluationResultQualifier.resourceType`	`additional.fields`	The value of `detail.oldEvaluationResult.evaluationResultIdentifier.evaluationResultQualifier.resourceType` from the raw log is added as a key-value pair to the `additional.fields` with the key "oldEvaluationResult_resourceType".
`detail.oldEvaluationResult.resultRecordedTime`	`additional.fields`	The value of `detail.oldEvaluationResult.resultRecordedTime` from the raw log is added as a key-value pair to the `additional.fields` with the key "oldEvaluationResult_resultRecordedTime".
`detail.readOnly`	`additional.fields`	The value of `detail.readOnly` from the raw log is converted to a string and added as a key-value pair to the `additional.fields` with the key "detail readOnly".
`detail.recipientAccountId`	`target.resource.attribute.labels`	The value of `detail.recipientAccountId` from the raw log is added as a key-value pair to the `target.resource.attribute.labels` with the key "Recipient Account Id".
`detail.recordVersion`	`metadata.product_version`	The value of `detail.recordVersion` from the raw log is mapped to the `metadata.product_version` field.
`detail.requestID`	`target.resource.attribute.labels`	The value of `detail.requestID` from the raw log is added as a key-value pair to the `target.resource.attribute.labels` with the key "Detail Request ID".
`detail.resourceType`	`target.resource.resource_subtype`	The value of `detail.resourceType` from the raw log is mapped to the `target.resource.resource_subtype` field.
`detail.s3Bucket`	`about.resource.name`	The value of `detail.s3Bucket` from the raw log is mapped to the `about.resource.name` field.
`detail.s3ObjectKey`	`target.resource.attribute.labels`	The value of `detail.s3ObjectKey` from the raw log is added as a key-value pair to the `target.resource.attribute.labels` with the key "s3ObjectKey".
`detail.userAgent`	`network.http.user_agent`	The value of `detail.userAgent` from the raw log is mapped to the `network.http.user_agent` field.
`detail.userIdentity.accessKeyId`	`target.user.userid`	The value of `detail.userIdentity.accessKeyId` from the raw log is mapped to the `target.user.userid` field.
`detail.userIdentity.accountId`	`metadata.product_deployment_id`	The value of `detail.userIdentity.accountId` from the raw log is mapped to the `metadata.product_deployment_id` field.
`detail.userIdentity.arn`	`target.user.userid`	The value of `detail.userIdentity.arn` from the raw log is mapped to the `target.user.userid` field.
`detail.userIdentity.principalId`	`principal.user.product_object_id`	The value of `detail.userIdentity.principalId` from the raw log is mapped to the `principal.user.product_object_id` field.
`detail.userIdentity.sessionContext.attributes.mfaAuthenticated`	`principal.user.attribute.labels`	The value of `detail.userIdentity.sessionContext.attributes.mfaAuthenticated` from the raw log is added as a key-value pair to the `principal.user.attribute.labels` with the key "mfaAuthenticated".
`detail.userIdentity.sessionContext.sessionIssuer.userName`	`target.user.user_display_name`	The value of `detail.userIdentity.sessionContext.sessionIssuer.userName` from the raw log is mapped to the `target.user.user_display_name` field.
`detail.userIdentity.type`	`principal.resource.type`	The value of `detail.userIdentity.type` from the raw log is mapped to the `principal.resource.type` field.
`detail-type`	`metadata.product_event_type`	The value of `detail-type` from the raw log is mapped to the `metadata.product_event_type` field.
`device`	`principal.asset.product_object_id`	The value of `device` from the raw log is mapped to the `principal.asset.product_object_id` field.
`digestPublicKeyFingerprint`	`target.file.sha1`	The value of `digestPublicKeyFingerprint` from the raw log is mapped to the `target.file.sha1` field.
`digestS3Bucket`	`principal.resource.name`	The value of `digestS3Bucket` from the raw log is mapped to the `principal.resource.name` field.
`digestS3Object`	`principal.asset.asset_id`	The value of `digestS3Object` from the raw log is prepended with "S3 Object: " and mapped to the `principal.asset.asset_id` field.
`digestSignatureAlgorithm`	`network.tls.cipher`	The value of `digestSignatureAlgorithm` from the raw log is mapped to the `network.tls.cipher` field.
`digestStartTime`	`metadata.event_timestamp`	The value of `digestStartTime` from the raw log is converted to a timestamp and mapped to the `metadata.event_timestamp` field.
`dimensions.VolumeId`	`additional.fields`	The value of `dimensions.VolumeId` from the raw log is added as a key-value pair to the `additional.fields` with the key "VolumeId".
`duration`	`additional.fields`	The value of `duration` from the raw log is added as a key-value pair to the `additional.fields` with the key "duration".
`errorCode`	`security_result.rule_name`	The value of `errorCode` from the raw log is mapped to the `security_result.rule_name` field.
`errorMessage`	`security_result.summary`	The value of `errorMessage` from the raw log is mapped to the `security_result.summary` field.
`executionId`	`principal.process.pid`	The value of `executionId` from the raw log is mapped to the `principal.process.pid` field.
`host`	`principal.hostname`, `principal.ip`	The value of `host` from the raw log, with hyphens replaced by dots, is parsed as an IP address and mapped to the `principal.ip` field if successful. Otherwise, it is mapped to the `principal.hostname` field.
`http_verb`	`network.http.method`	The value of `http_verb` from the raw log is converted to uppercase and mapped to the `network.http.method` field.
`kubernetes.container_hash`	`additional.fields`	The value of `kubernetes.container_hash` from the raw log is added as a key-value pair to the `additional.fields` with the key "container_hash".
`kubernetes.container_image`	`additional.fields`	The value of `kubernetes.container_image` from the raw log is added as a key-value pair to the `additional.fields` with the key "container_image".
`kubernetes.container_name`	`additional.fields`	The value of `kubernetes.container_name` from the raw log is added as a key-value pair to the `additional.fields` with the key "container_name".
`kubernetes.docker_id`	`principal.asset_id`	The value of `kubernetes.docker_id` from the raw log is prepended with "id: " and mapped to the `principal.asset_id` field.
`kubernetes.host`	`principal.hostname`, `principal.ip`	The value of `kubernetes.host` from the raw log, with hyphens replaced by dots, is parsed as an IP address and mapped to the `principal.ip` field if successful. Otherwise, it is mapped to the `principal.hostname` field.
`kubernetes.namespace`	`principal.namespace`	The value of `kubernetes.namespace` from the raw log is mapped to the `principal.namespace` field.
`kubernetes.namespace_name`	`principal.namespace`	The value of `kubernetes.namespace_name` from the raw log is mapped to the `principal.namespace` field.
`kubernetes.pod_id`	`principal.asset.asset_id`	The value of `kubernetes.pod_id` from the raw log is prepended with "pod_id: " and mapped to the `principal.asset.asset_id` field.
`kubernetes.pod_name`	`additional.fields`	The value of `kubernetes.pod_name` from the raw log is added as a key-value pair to the `additional.fields` with the key "pod name".
`lambdaArn`	`principal.hostname`	The value of `lambdaArn` from the raw log is mapped to the `principal.hostname` field.
`level`	`security_result.severity`	The value of `level` from the raw log determines the value of `security_result.severity`. If `level` is "Info", the value is "INFORMATIONAL". If `level` is "Error", the value is "ERROR". If `level` is "Warning", the value is "MEDIUM".
`log`	`metadata.description`	The value of `log` from the raw log is mapped to the `metadata.description` field if `descr` is empty.
`logFiles`	`about`	For each element in the `logFiles` array from the raw log, an `about` object is created with `file.full_path` set to `s3Object`, `asset.hostname` set to `s3Bucket`, and `file.sha256` set to `hashValue`.
`log_processed.cause`	`security_result.summary`	The value of `log_processed.cause` from the raw log is mapped to the `security_result.summary` field.
`log_processed.ids`	`intermediary.hostname`	For each element in the `log_processed.ids` array from the raw log, an `intermediary` object is created with `hostname` set to the element's value.
`log_processed.level`	`security_result.severity`	The value of `log_processed.level` from the raw log is mapped to the `security_result.severity` field.
`log_processed.msg`	`metadata.description`	The value of `log_processed.msg` from the raw log is mapped to the `metadata.description` field.
`log_processed.ts`	`metadata.event_timestamp`	The value of `log_processed.ts` from the raw log is converted to a timestamp and mapped to the `metadata.event_timestamp` field.
`log_type`	`metadata.log_type`	The value of `log_type` from the raw log is mapped to the `metadata.log_type` field. This is a custom field added for context.
`logevent.message`	`security_result.description`	The value of `logevent.message` from the raw log is mapped to the `security_result.description` field. It is also parsed using grok to extract additional fields.
`logGroup`	`security_result.about.resource.name`	The value of `logGroup` from the raw log is mapped to the `security_result.about.resource.name` field.
`logStream`	`security_result.about.resource.attribute.labels`	The value of `logStream` from the raw log is added as a key-value pair to the `security_result.about.resource.attribute.labels` with the key "logStream".
`memory_used`	`additional.fields`	The value of `memory_used` from the raw log is added as a key-value pair to the `additional.fields` with the key "memory_used".
`metric_name`	`additional.fields`	The value of `metric_name` from the raw log is added as a key-value pair to the `additional.fields` with the key "metric_name".
`metric_stream_name`	`additional.fields`	The value of `metric_stream_name` from the raw log is added as a key-value pair to the `additional.fields` with the key "metric_stream_name".
`namespace`	`principal.namespace`	The value of `namespace` from the raw log is mapped to the `principal.namespace` field.
`owner`	`principal.user.userid`	The value of `owner` from the raw log is mapped to the `principal.user.userid` field.
`parameters`	`additional.fields`	The value of `parameters` from the raw log is added as a key-value pair to the `additional.fields` with the key "Parameters".
`Path`	`principal.process.file.full_path`	The value of `Path` from the raw log is mapped to the `principal.process.file.full_path` field.
`pid`	`principal.process.pid`	The value of `pid` from the raw log is mapped to the `principal.process.pid` field.
`PolicyName`	`security_result.rule_name`	The value of `PolicyName` from the raw log is mapped to the `security_result.rule_name` field.
`prin_host`	`principal.hostname`	The value of `prin_host` from the raw log is mapped to the `principal.hostname` field.
`principal_hostname`	`principal.hostname`	The value of `principal_hostname` from the raw log is mapped to the `principal.hostname` field.
`process`	`principal.application`	The value of `process` from the raw log is mapped to the `principal.application` field.
`rawData`	`additional.fields`	The value of `rawData` from the raw log is added as a key-value pair to the `additional.fields` with the key "Raw Data".
`Recommendation`	`security_result.detection_fields`	The value of `Recommendation` from the raw log is added as a key-value pair to the `security_result.detection_fields` with the key "Recommendation".
`referral_url`	`network.http.referral_url`	The value of `referral_url` from the raw log is mapped to the `network.http.referral_url` field.
`region`	`principal.location.name`	The value of `region` from the raw log is mapped to the `principal.location.name` field.
`resp_code`	`network.http.response_code`	The value of `resp_code` from the raw log is converted to an integer and mapped to the `network.http.response_code` field.
`resource_url`	`network.http.referral_url`	The value of `resource_url` from the raw log is mapped to the `network.http.referral_url` field.
`ResourceType`	`target.resource.resource_subtype`	The value of `ResourceType` from the raw log is mapped to the `target.resource.resource_subtype` field.
`response_body`	`additional.fields`	The value of `response_body` from the raw log is added as a key-value pair to the `additional.fields` with the key "Response body".
`Role`	`target.resource.product_object_id`	The value of `Role` from the raw log is mapped to the `target.resource.product_object_id` field.
`s3_bucket_path`	`target.file.full_path`	The value of `s3_bucket_path` from the raw log is mapped to the `target.file.full_path` field.
`sec_result.category`	`security_result.category`	The value of `sec_result.category` is derived from the parser logic. If `descr` contains "authentication is required", the value is "AUTH_VIOLATION".
`sec_result.description`	`security_result.description`	The value of `sec_result.description` is derived from the parser logic. It is set to the value of `cloudwatchLog` if present.
`sec_result.severity`	`security_result.severity`	The value of `sec_result.severity` is derived from the parser logic. It is set based on the value of `severity` or `level`.
`sec_result.summary`	`security_result.summary`	The value of `sec_result.summary` is derived from the parser logic. It is set to the value of `log_processed.cause` or `errorMessage` if present.
`security_result`	`security_result`	The `security_result` object is constructed from various fields and parser logic.
`serverId`	`additional.fields`	The value of `serverId` from the raw log is added as a key-value pair to the `additional.fields` with the key "server_id".
`severity`	`security_result.severity`	The value of `severity` from the raw log, converted to uppercase and normalized, is mapped to the `security_result.severity` field.
`Source`	`principal.hostname`	The value of `Source` from the raw log is mapped to the `principal.hostname` field.
`source`	`principal.hostname`	The value of `source` from the raw log is mapped to the `principal.hostname` field.
`SourceIP`	`principal.ip`	The value of `SourceIP` from the raw log is mapped to the `principal.ip` field.
`src_port`	`principal.port`	If `src_port` is "80", it is converted to an integer and mapped to the `principal.port` field, and `network.application_protocol` is set to "HTTP".
`stream`	`additional.fields`	The value of `stream` from the raw log is added as a key-value pair to the `additional.fields` with the key "stream".
`subscriptionFilters`	`security_result.about.resource.attribute.labels`	For each element in the `subscriptionFilters` array from the raw log, a key-value pair is added to the `security_result.about.resource.attribute.labels` with the key "subscriptionFilter" and the value from the array.
`support_contact`	`target.resource.attribute.labels`	The value of `support_contact` from the raw log is added as a key-value pair to the `target.resource.attribute.labels` with the key "Support Contact".
`t_ip`	`target.ip`	The value of `t_ip` from the raw log, after removing hyphens, is parsed as an IP address and mapped to the `target.ip` field if successful.
`time`	`metadata.event_timestamp`	The value of `time` from the raw log is converted to a timestamp and mapped to the `metadata.event_timestamp` field.
`timestamp`	`metadata.event_timestamp`	The value of `timestamp` from the raw log is converted to a timestamp using various formats and mapped to the `metadata.event_timestamp` field.
`tls`	`network.tls.version`	The value of `tls` from the raw log is mapped to the `network.tls.version` field.
`transferDetails.serverId`	`additional.fields`	The value of `transferDetails.serverId` from the raw log is added as a key-value pair to the `additional.fields` with the key "server_id".
`transferDetails.sessionId`	`network.session_id`	The value of `transferDetails.sessionId` from the raw log is mapped to the `network.session_id` field.
`transferDetails.username`	`principal.user.user_display_name`	The value of `transferDetails.username` from the raw log is mapped to the `principal.user.user_display_name` field.
`ts`	`metadata.event_timestamp`	The value of `ts` from the raw log, combined with the timezone if available, is converted to a timestamp and mapped to the `metadata.event_timestamp` field.
`type`	`metadata.product_event_type`	The value of `type` from the raw log is mapped to the `metadata.product_event_type` field.
`unit`	`additional.fields`	The value of `unit` from the raw log is added as a key-value pair to the `additional.fields` with the key "unit".
`url`	`target.url`	The value of `url` from the raw log is mapped to the `target.url` field.
`url_back_to_product`	`metadata.url_back_to_product`	The value of `url_back_to_product` from the raw log is mapped to the `metadata.url_back_to_product` field.
`User`	`principal.user.userid`	The value of `User` from the raw log is mapped to the `principal.user.userid` field.
`user`	`target.user.userid`, `metadata.event_type`, `extensions.auth.mechanism`	If `user` is present, `metadata.event_type` is set to "USER_LOGIN", `extensions.auth.mechanism` is set to "NETWORK", and the value of `user` is mapped to `target.user.userid`.
`value.count`	`additional.fields`	The value of `value.count` from the raw log is converted to a string and added as a key-value pair to the `additional.fields` with the key "count".
`value.max`	`additional.fields`	The value of `value.max` from the raw log is converted to a string and added as a key-value pair to the `additional.fields` with the key "max".
`value.min`	`additional.fields`	The value of `value.min` from the raw log is converted to a string and added as a key-value pair to the `additional.fields` with the key "min".
`value.sum`	`additional.fields`	The value of `value.sum` from the raw log is converted to a string and added as a key-value pair to the `additional.fields` with the key "sum".
`workflowId`	`additional.fields`	The value of `workflowId` from the raw log is added as a key-value pair to the `additional.fields` with the key "workflowId".

Changes

2024-02-12

Enhancement:
Mapped timestamp to UNIX_MS.

2023-09-02

Enhancement:
Added a "kv block" to parse key-value format logs.
Mapped "SourceIP" to "principal.ip".
Mapped "prin_host" to "principal.hostname".
Mapped "User" to "principal.user.userid".
Mapped "Ciphers" to "network.tls.client.supported_ciphers".
Mapped "executionId" to "principal.process.pid".
Mapped "transferDetails.sessionId" to "network.session_id".
Mapped "transferDetails.username" to "principal.user.user_display_name".
Mapped "transferDetails.serverId", "workflowId", "details.input.initialFileLocation.etag", "details.input.initialFileLocation.backingStore", "details.input.initialFileLocation.bucket", "details.input.initialFileLocation.key",
"Mode", "Kex" to "additional.fields".
Mapped "BytesIn" to "network.received_bytes".
Mapped "Role" to "target.resource.product_object_id".

2023-08-18

Enhancement:
Added a Grok pattern to parse the unparsed raw logs.

2023-07-07

Enhancement:
Added support for 'logEvents'-related JSON logs.

2022-12-17

Mapped "CloudType" to "target.resource.attribute.cloud.environment".
Mapped "AlertId" to "metadata.product_log_id".
Mapped "ResourceType" to "target.resource.resource_subtype".
Mapped "ResourceRegion" to "target.location.country_or_region".
Mapped "Recommendation" to "security_result.detection_fields".
Mapped "PolicyName","detail.additionalEventData.configRuleName" to "security_result.rule_name".
Mapped "detail-type" to "metadata.product_event_type".
Mapped "region","detail.awsRegion" to "principal.location.name".
Mapped "detail.eventSource" to "target.application".
Mapped "detail.requestID" to "target.resource.attribute.labels".
Mapped "detail.userAgent" to "network.http.user_agent".
Mapped "detail.eventVersion" to "metadata.product_version".
Mapped "detail.userIdentity.accountId" to "metadata.product_deployment_id".
Mapped "detail.userIdentity.accessKeyId" to "target.user.userid".
Mapped "detail.userIdentity.type" to "principal.resource.type".
Mapped "detail.userIdentity.principalId" to "principal.user.product_object_id".
Mapped "detail.user.arn" to "target.user.userid".
Mapped "detail.user.sessionContext.sessionIssuer.userName" to "target.user.user_display_name".
Mapped "detail.user.mfaAuthenticated" to "principal.user.attribute.labels".
Mapped "detail.recipientAccountId" to "target.resource.attribute.labels".
Mapped "detail.managementEvent", "detail.eventType", "detail.readOnly", "detail.eventName", "detail.additionalEventData.notificationJobType", "detail.additionalEventData.managedRuleIdentifier", "duration", "billed_duration", "memory_used" to "additional.fields".
Mapped "detail.eventCategory" to "security_result.category_details".
Mapped "detail.eventID" to "metadata.product_log_id".
Mapped "detail.additionalEventData.configRuleArn" to "security_result.rule_id".
Mapped "level" to "security_result.severity".
Mapped "src_port" to "principal.port".
Mapped "request_id" to "target.resource.attribute.labels".
Mapped "url" to "target.url".

2022-09-03

Enhancement:
Added grok to parse newly ingested logs.
Mapped "package" to "event.idm.read_only_udm.principal.process.command_line".
Mapped "session_id" to "event.idm.read_only_udm.network.session_id".
Mapped "network_dir" to "event.idm.read_only_udm.network.direction".
Mapped "port" to "event.idm.read_only_udm.target.port".
Remapped "digestPublicKeyFingerprint" from "additional.fields" to "event.idm.read_only_udm.target.file.sha1".
Added other log levels like "AUDIT", "TRACE", "DEBUG", "NOTICE", "ERROR" for severity mapping.
Duplicated the value in "target.ip" to "principal.ip" to set event_type as "STATUS_UPDATE" thereby reducing generic percentage.
Added conditions for "event_type" "USER_UNCATEGORIZED", "NETWORK_HTTP", "NETWORK_CONNECTION", "STATUS_UPADTE" to reduce generic percentage.

2022-08-11

Bug Fix - Remapped "digestS3Bucket" to "principal.resource.name".
Remapped "kubernetes.pod_name" to "additional.fields".

2022-05-27

Enhancement:
Modified the value stored in metadata.product_name to 'AWS CloudWatch' and metadata.vendor_name to 'AMAZON'.

Need more help? Get answers from Community members and Google SecOps professionals.