Collect Cloud Secure Web Proxy logs

Supported in:

Google SecOps SIEM

Note: This feature is covered by Pre-GA Offerings Terms of the Google Security Operations Service Specific Terms. Pre-GA features might have limited support, and changes to pre-GA features might not be compatible with other pre-GA versions. For more information, see the Google Security Operations Technical Support Service guidelines and the Google Security Operations Service Specific Terms.

This document explains how to export and ingest Cloud Secure Web Proxy into Google Security Operations using Cloud Storage. The parser extracts fields from JSON logs, transforming them into the Unified Data Model (UDM). It initializes UDM fields, parses the JSON payload, extracts network information, security details, resource attributes, and sets the event type based on the presence of principal and target information.

Before you begin

• Ensure that you have a Google SecOps instance.
• Ensure that Cloud Secure Web Proxy is active and configured in your Google Cloud environment.
• Ensure that you have privileged access to Google Cloud and appropriate permissions to access Secure Web Proxy logs.

Create a Cloud Storage bucket

1. Sign in to the Google Cloud console.

2. Go to the Cloud Storage Buckets page.

   Go to Buckets

3. Click Create.

4. On the Create a bucket page, enter your bucket information. After each of the following steps, click Continue to proceed to the next step:

   1. In the Get started section, do the following:

      1. Enter a unique name that meets the bucket name requirements; for example, gcp-swp-logs.

      2. To enable hierarchical namespace, click the expander arrow to expand the Optimize for file oriented and data-intensive workloads section, and then select Enable Hierarchical namespace on this bucket.

         Note: You cannot enable hierarchical namespace on an existing bucket.

      3. To add a bucket label, click the expander arrow to expand the Labels section.

      4. Click Add label, and specify a key and a value for your label.

   2. In the Choose where to store your data section, do the following:

      1. Select a Location type.

      2. Use the location type menu to select a Location where object data within your bucket will be permanently stored.

         Note: If you select the dual-region location type, you can also choose to enable turbo replication by using the relevant checkbox.

      3. To set up cross-bucket replication, expand the Set up cross-bucket replication section.

   3. In the Choose a storage class for your data section, either select a default storage class for the bucket, or select Autoclass for automatic storage class management of your bucket's data.

   4. In the Choose how to control access to objects section, select not to enforce public access prevention, and select an access control model for your bucket's objects.

      Note: If public access prevention is already enforced by your project's organization policy, the Prevent public access checkbox is locked.

   5. In the Choose how to protect object data section, do the following:

      1. Select any of the options under Data protection that you want to set for your bucket.
      2. To choose how your object data will be encrypted, click the expander arrow labeled Data encryption, and select a Data encryption method.

5. Click Create.

Note: Be sure to provide your Google SecOps service account with permissions to Read or Read & Write to the newly created bucket.

Configure Google Cloud Secure Web Proxy logs export

1. Sign in to the Google Cloud console.
2. Go to Logging > Log Router.
3. Click Create Sink.

4. Provide the following configuration parameters:

   • Sink Name: enter a meaningful name; for example, SWP-Export-Sink.
   • Sink Destination: select Cloud Storage Storage and enter the URI for your bucket; for example, gs://gcp-swp-logs.
   • Log Filter:

   logName="projects/<your-project-id>/logs/networkservices.googleapis.com/gateway_requests"
   

5. Click Create.

Configure permissions for Cloud Storage

1. Go to IAM & Admin > IAM.
2. Locate the Cloud Logging service account.
3. Grant the roles/storage.admin on the bucket.

Configure a feed in Google SecOps to ingest Google Cloud Secure Web Proxy logs

1. Go to SIEM Settings > Feeds.
2. Click Add new.
3. In the Feed name field, enter a name for the feed; for example, Google Cloud SWP Logs.
4. Select Google Cloud Storage as the Source type.
5. Select GCP Secure Web Proxy as the Log type.
6. Click Get Service Account next to the Chronicle Service Account field.
7. Click Next.

8. Specify values for the following input parameters:

   • Storage Bucket URI: Cloud Storage bucket URL; for example, gs://gcp-swp-logs.
   • URI Is A: select Directory which includes subdirectories.

   • Source deletion options: select the deletion option according to your preference.

     Note: If you select the Delete transferred files or Delete transferred files and empty directories option, make sure that you granted appropriate permissions to the service account.

   • Asset namespace: the asset namespace.

   • Ingestion labels: the label applied to the events from this feed.

9. Click Next.

10. Review your new feed configuration in the Finalize screen, and then click Submit.

UDM Mapping Table

Log Field	UDM Mapping	Logic
httpRequest.latency	additional.fields[].key: HTTPRequest Latency additional.fields[].value.string_value: 0.124462s	Directly mapped from the raw log field.
httpRequest.protocol	network.application_protocol: HTTP network.application_protocol_version: 2	The protocol and version are extracted from the httpRequest.protocol field using a grok pattern.
httpRequest.remoteIp	target.asset.ip: 1.1.0.1 target.ip: 1.1.0.1	The IP address is extracted from the httpRequest.remoteIp field using a grok pattern.
httpRequest.requestMethod	network.http.method: GET	Directly mapped from the raw log field.
httpRequest.requestSize	network.sent_bytes: 144	Directly mapped from the raw log field and converted to an integer.
httpRequest.requestUrl	target.url: https://github.com/tempuslabs/tempusutils/info/refs?service=git-upload-pack	Directly mapped from the raw log field.
httpRequest.responseSize	network.received_bytes: 225	Directly mapped from the raw log field and converted to an integer.
httpRequest.serverIp	principal.asset.ip: 1.8.1.4 principal.ip: 1.8.1.4	The IP address is extracted from the httpRequest.serverIp field using a grok pattern.
httpRequest.status	network.http.response_code: 401	Directly mapped from the raw log field and converted to an integer.
httpRequest.userAgent	network.http.user_agent: git/2.34.1 network.http.parsed_user_agent: { family: USER_DEFINED, device: git, device_version: 2.34.1 }	Directly mapped from the raw log field. The parsed_user_agent field is derived by parsing the httpRequest.userAgent field.
insertId	metadata.product_log_id: 1yh8wczer5o8n	Directly mapped from the raw log field.
jsonPayload.@type	additional.fields[].key: Log Type additional.fields[].value.string_value: type.googleapis.com/google.cloud.loadbalancing.type.LoadBalancerLogEntry	Directly mapped from the raw log field.
jsonPayload.enforcedGatewaySecurityPolicy.hostname	target.asset.hostname: github.com target.hostname: github.com	Directly mapped from the raw log field.
jsonPayload.enforcedGatewaySecurityPolicy.matchedRules[].action	security_result.action: ALLOW security_result.action_details: ALLOWED	The security_result.action is derived based on the value of jsonPayload.enforcedGatewaySecurityPolicy.matchedRules[].action. If the action is ALLOWED, the UDM field is set to ALLOW. If the action is DENIED, the UDM field is set to BLOCK.
jsonPayload.enforcedGatewaySecurityPolicy.matchedRules[].name	security_result.rule_name: projects/671807354785/locations/us-central1/gatewaySecurityPolicies/github-access-gateway-security-policy-5cec30cd/rules/github-access-gateway-security-policy-rule-5cec30cd	Directly mapped from the raw log field.
jsonPayload.enforcedGatewaySecurityPolicy.requestWasTlsIntercepted	security_result.detection_fields[].key: requestWasTlsIntercepted security_result.detection_fields[].value: true	Directly mapped from the raw log field.
logName	additional.fields[].key: Log Name additional.fields[].value.string_value: projects/rws-w6uza3pn5jzzh6z3hc3d/logs/networkservices.googleapis.com%2Fgateway_requests	Directly mapped from the raw log field.
receiveTimestamp	metadata.collected_timestamp: { seconds: 1710189647, nanos: 661101224 }	Parsed from the raw log field using the RFC 3339 date format.
resource.labels.gateway_name	security_result.detection_fields[].key: gateway-name security_result.detection_fields[].value: github-access-gateway-5cec30cd	Directly mapped from the raw log field.
resource.labels.gateway_type	security_result.detection_fields[].key: gateway-type security_result.detection_fields[].value: SECURE_WEB_GATEWAY	Directly mapped from the raw log field.
resource.labels.location	target.resource.attribute.cloud.availability_zone: us-central1	Directly mapped from the raw log field.
resource.labels.network_name	target.resource.attribute.labels[].key: rc_network_name target.resource.attribute.labels[].value: projects/rws-w6uza3pn5jzzh6z3hc3d/global/networks/rws-tr-pilot-workspace	Directly mapped from the raw log field.
resource.type	target.resource.attribute.labels[].key: Resource Type target.resource.attribute.labels[].value: networkservices.googleapis.com/Gateway	Directly mapped from the raw log field.
severity	security_result.severity: MEDIUM	Mapped from the raw log field. The value is translated to a UDM severity level. In this case, WARNING is mapped to MEDIUM.
timestamp	metadata.event_timestamp: { seconds: 1710189639, nanos: 952848000 }	Parsed from the raw log field using the RFC 3339 date format.
(Parser Logic)	metadata.event_type: NETWORK_HTTP	Determined by parser logic based on the presence of has_principal, has_target, and a protocol matching http.
(Parser Logic)	metadata.log_type: GCP_SWP	Hardcoded value based on the product.

Changes

2024-04-15

• Newly created parser.

Need more help? Get answers from Community members and Google SecOps professionals.