Collect Cloudflare WAF logs

Supported in:

Google SecOps SIEM

Note: This feature is covered by Pre-GA Offerings Terms of the Google Security Operations Service Specific Terms. Pre-GA features might have limited support, and changes to pre-GA features might not be compatible with other pre-GA versions. For more information, see the Google Security Operations Technical Support Service guidelines and the Google Security Operations Service Specific Terms.

This parser extracts fields from Cloudflare Web Application Firewall (WAF) JSON logs, transforms and maps them to the Unified Data Model (UDM). It handles various Cloudflare actions, enriching the data with metadata and network information before structuring the output into the UDM format.

Before you begin

• Ensure that you have a Google SecOps instance.
• Ensure that you have privileged access to Google Cloud.
• Ensure that you have Cloudflare Enterprise plan.
• Ensure that you have privileged access to Cloudflare.

Create a Google Cloud Storage Bucket

1. Sign in to the Google Cloud console

2. Go to the Cloud Storage Buckets page.

   Go to Buckets

3. Click Create.

4. Configure the bucket:

   • Name: enter a unique name that meets the bucket name requirements (for example, cloudflare-waf).
   • Choose where to store your data: select a location.
   • Choose a storage class for your data: either select a default storage class for the bucket, or select Autoclass for automatic storage class management.
   • Choose how to control access to objects: select not to enforce public access prevention, and select an access control model for your bucket's objects.
   Note: If public access prevention is already enforced by your project's organization policy, the Prevent public access checkbox is locked.
   • Storage class: Choose based on your needs (for example, Standard).

5. Click Create.

Note: Do not set a retention policy, as the last data entry may need to be overwritten in case of a timeout.

Grant bucket permissions to Cloudflare IAM user

1. In Google Cloud, go to Storage > Browser > Bucket > Permissions.
2. Add the account [e-mail] with Storage Object Admin permission.

Create a Logpush Job for WAF Logs using Cloudflare UI

1. Sign in to Cloudflare.
2. Go to Analytics & Logs > Logpush.
3. Select Create a Logpush job.
4. In Select a destination, choose Google Cloud Storage.
5. Enter the following destination details:
   • Bucket: Google Cloud Storage bucket name
   • Path: Bucket location within the storage container
   • Select Organize logs into daily subfolders
6. Click Continue.

Note: To prove ownership, Cloudflare will send a file to your designated destination. To find the token, select the Open button in the Overview tab of the ownership challenge file, then paste it into the Cloudflare dashboard to verify your access to the bucket. Enter the Ownership Token and select Continue.

1. Select the Security (WAF) dataset to push to the storage.
2. Configure the logpush job:
   • Enter the Job name.
   • Under If logs match, you can select the events to include and/or remove from your logs. Refer to Filters for more information. Not all datasets have this option available.
   • In Send the following fields, you can choose to either push all logs to your storage destination or selectively choose which logs you want to push.
3. Click Submit.

Configure a feed in Google SecOps to ingest Cloudflare WAF logs

1. Go to SIEM Settings > Feeds.
2. Click Add new.
3. In the Feed name field, enter a name for the feed (for example, Cloudflare WAF Logs).
4. Select Google Cloud Storage as the Source type.
5. Select Cloudflare WAF as the Log type.
6. Click Get Service Account.
7. Click Next.

8. Specify values for the following input parameters:

   • GCS URI: the Cloud Storage URL.
   • URI is a: select Directory which includes subdirectories.
   • Source deletion options: select the deletion option according to your preference.
   Note: If you select the Delete transferred files or Delete transferred files and empty directories option, make sure that you granted appropriate permissions to the service account.
   • Asset namespace: the asset namespace.
   • Ingestion labels: the label to be applied to the events from this feed.

9. Click Next.

10. Review your new feed configuration in the Finalize screen, and then click Submit.

UDM Mapping Table

Log Field	UDM Mapping	Logic
Action	security_result.action_details	The value of Action from the raw log is directly assigned to this UDM field.
Action	security_result.action	The value of this field is derived from the Action field in the raw log. If Action is "allow", the UDM field is set to ALLOW. If Action is "challengeSolved", "jschallengeSolved", "managedchallengenoninteractivesolved", or "managedchallengeinteractivesolved", the UDM field is set to ALLOW_WITH_MODIFICATION. If Action is "drop", "block", or "connectionclose", the UDM field is set to BLOCK. If Action is "challengefailed" or "jschallengefailed", the UDM field is set to FAIL. Otherwise, it's set to UNKNOWN_ACTION.
ClientASN	network.asn	The value of ClientASN from the raw log is directly assigned to this UDM field after converting it to a string.
ClientASNDescription	additional.fields.key	The key is statically set to "ClientASNDescription".
ClientASNDescription	additional.fields.value.string_value	The value of ClientASNDescription from the raw log is directly assigned to this UDM field.
ClientCountry	principal.location.country_or_region	The value of ClientCountry from the raw log is directly assigned to this UDM field.
ClientIP	principal.ip	The value of ClientIP from the raw log is directly assigned to this UDM field.
ClientRefererHost	intermediary.hostname	The value of ClientRefererHost from the raw log is directly assigned to this UDM field.
ClientRefererPath	network.http.referral_url	The value of ClientRefererPath from the raw log is directly assigned to this UDM field.
ClientRequestHost	target.hostname	The value of ClientRequestHost from the raw log is directly assigned to this UDM field.
ClientRequestMethod	network.http.method	The value of ClientRequestMethod from the raw log is directly assigned to this UDM field.
ClientRequestPath	target.file.full_path	The value of ClientRequestPath from the raw log is directly assigned to this UDM field.
ClientRequestProtocol	network.application_protocol	The protocol part of ClientRequestProtocol (e.g., "HTTP" from "HTTP/1.1") is extracted using grok, converted to uppercase, and assigned to this UDM field.
ClientRequestUserAgent	network.http.user_agent	The value of ClientRequestUserAgent from the raw log is directly assigned to this UDM field.
Datetime	metadata.event_timestamp	The value of Datetime from the raw log is parsed as an RFC 3339 timestamp and assigned to this UDM field.
EdgeColoCode	additional.fields.key	The key is statically set to "EdgeColoCode".
EdgeColoCode	additional.fields.value.string_value	The value of EdgeColoCode from the raw log is directly assigned to this UDM field.
EdgeResponseStatus	network.http.response_code	The value of EdgeResponseStatus from the raw log is directly assigned to this UDM field and converted to an integer.
Kind	metadata.product_event_type	The value of Kind from the raw log is directly assigned to this UDM field.
Metadata.filter	target.resource.attribute.labels.value	The value of Metadata.filter from the raw log is assigned to the value field of a label within target.resource.attribute.labels. The key for this label is statically set to "Metadata filter".
Metadata.type	target.resource.attribute.labels.value	The value of Metadata.type from the raw log is assigned to the value field of a label within target.resource.attribute.labels. The key for this label is statically set to "Metadata type". The value of this field is derived based on the presence and values of ClientIP, ClientRequestHost, and app_protocol. See parser code for the specific logic. Statically set to "Cloudflare". Statically set to "Cloudflare log Aggregator". Statically set to "CLOUDFLARE_WAF".
RayID	metadata.product_log_id	The value of RayID from the raw log is directly assigned to this UDM field.
RuleID	security_result.rule_id	The value of RuleID from the raw log is directly assigned to this UDM field.
Source	security_result.rule_name	The value of Source from the raw log is directly assigned to this UDM field.
timestamp	metadata.event_timestamp, events.timestamp	The value of timestamp from the raw log is directly assigned to these UDM fields.

Changes

2024-08-08

• Extracted data from "ClientRequestHost" and "ClientRequestURI", merged it and mapped the result to "target.url".

2023-08-30

• Initialized field "ClientRequestPath".

2023-02-02

• Validated the 'security_result' value before its getting merged to event.

2022-09-16

• Mapped the field 'Action' to 'security_result.action_details'.
• Mapped 'security_result.action' to 'ALLOW_WITH_MODIFICATION' when action contains "challengeSolved", "jschallengeSolved", "managedchallengenoninteractivesolved", "managedchallengeinteractivesolved".
• Mapped 'security_result.action' to 'BLOCK' when action contains "drop", "block", "connectionclose".
• Mapped 'security_result.action' to 'FAIL' when action contains "challengefailed", "jschallengefailed".

2022-07-25

• Newly created parser