Ingest Google Cloud data to Google Security Operations

This page shows how to enable and disable the ingestion of your Google Cloud data into Google Security Operations. Google Security Operations lets you to store, search and examine the aggregated security information for your enterprise going back for months or longer in accordance to your data retention period.

Overview

There are two options to send Google Cloud data to Google Security Operations. Choosing the right option depends on log type.

Option 1: Direct ingestion

A special Cloud Logging filter can be configured in Google Cloud to send specific log types to Google Security Operations in real-time. These logs are generated by Google Cloud services.

Google Security Operations receives logs even if they're excluded at the project level in Google Cloud, but included in both the log export filter and organization-level Google Cloud logging. To exclude logs from Google Security Operations, you must update the Google Security Operations log export filter in Google Cloud.

Available log types include:

• Cloud Audit Logs
• Cloud NAT
• Cloud DNS
• Cloud Next Generation Firewall
• Cloud Intrusion Detection System
• Cloud Load Balancing
• Cloud SQL
• Windows Event logs
• Linux syslog
• Linux Sysmon
• Zeek
• Google Kubernetes Engine
• Audit Daemon (auditd)
• Apigee
• reCAPTCHA Enterprise
• Cloud Run logs (GCP_RUN)
• Cloud Next Generation Firewall

To collect Compute Engine or Google Kubernetes Engine (GKE) application logs (such as Apache, Nginx, or IIS), use Option 2. Additionally, raise a Support ticket with Google Security Operations to provide feedback for future consideration to support as a log type using Option 1.

For specific log filters and more ingestion details, see Exporting Google Cloud Logs to Google Security Operations.

Additional Google Cloud metadata to be used as context for enrichment purposes can also be sent to Google Security Operations. See Exporting Google Cloud Asset Metadata to Google Security Operations for more details.

Note: At this time, the direct ingestion method does not support Namespace or Label assignment.

Option 2: Google Cloud Storage

Cloud Logging can route logs to Cloud Storage to be fetched by Google Security Operations on a scheduled basis.

Note: This option will generate additional costs in your Google Cloud instance for the use of Cloud Storage.

For details on how to configure Cloud Storage for Google Security Operations, see Feed Management: Cloud Storage.

Before you begin

Before you can ingest your Google Cloud data into your Google Security Operations instance, you must complete the following steps:

1. Contact your Google Security Operations representative and obtain the one-time access code you need to ingest your Google Cloud data.

2. Grant the following IAM roles required for you to access the Google Security Operations section:

   • Chronicle Service Admin (roles/chroniclesm.admin): IAM role for performing all activities.
   • Chronicle Service Viewer (roles/chroniclesm.viewer): IAM role to only view the state of ingestion.
   • Security Center Admin Editor (roles/securitycenter.adminEditor): Required to enable the ingestion of Cloud Asset Metadata.

3. If you plan to enable Cloud Asset Metadata, you must also enable either the Security Command Center Standard tier or Security Command Center Premium tier Google Cloud service. See Activate Security Command Center for an organization for more information.

Granting IAM Roles

You can grant the required IAM roles using either the Google Cloud console or using the gcloud CLI.

To grant IAM roles using Google Cloud console, complete the following steps:

1. Log on to the Google Cloud Organization you want to connect to and navigate to the IAM screen using Products > IAM & Admin > IAM.

2. From the IAM screen, select the user and click Edit Member.

   Note: If you are not in the Organization view of IAM, the Edit Member button is disabled and you need to navigate to the organization's IAM screen.

3. In the Edit Permissions screen, click Add Another Role and search for Google Security Operations to find the IAM roles.

4. Once you have assigned the roles, click Save.

To grant IAM roles using the Google Cloud CLI, complete the following steps:

1. Ensure you are logged into the correct organization. Verify this by running the gcloud init command.

2. To grant the Chronicle Service Admin IAM role using gcloud, run the following command:

   gcloud organizations add-iam-policy-binding ORGANIZATION_ID \
   --member "user:USER_EMAIL" \
   --role roles/chroniclesm.admin
   

   Replace the following:

   • ORGANIZATION_ID: the numeric organization ID.
   • USER_EMAIL: the admin user's email address.

3. To grant the Google Security Operations Service Viewer IAM role using gcloud, run the following command:

   gcloud organizations add-iam-policy-binding ORGANIZATION_ID \
   --member "user:USER_EMAIL" \
   --role roles/chroniclesm.viewer
   

4. To grant the Security Center Admin Editor role using gcloud, run the following command:

    gcloud organizations add-iam-policy-binding ORGANIZATION_ID \
    --member "user:USER_EMAIL" \
    --role roles/securitycenter.adminEditor`
   

Enabling Google Cloud data ingestion

Google Cloud data is ingested using a private internal API between Security Command Center and Google Security Operations. The ingestion never reaches the external network and never uses IP addresses. Google accesses the Google Cloud logs directly based on the authentication done using the one-time code provided by Google Security Operations for Security Command Center.

To enable data ingestion from your Google Cloud organization into your Google Security Operations instance, complete the following steps:

1. Navigate to the Google Security Operations page for the Google Cloud console.
   Go to the Google Security Operations page
   

2. Enter your one-time access code in the 1-time Google Security Operations access code field.

3. To consent to Google Security Operations usage, check the box labeled I consent to the terms and conditions of Chronicle's usage of my Google Cloud data.

4. Click Connect Google Security Operations.

Your Google Cloud data is now going to be sent to Google Security Operations. You can use Google Security Operations's analysis features to investigate any security related issues. The sections that follow describe ways to adjust the types of Google Cloud data that are going to be sent to Google Security Operations

Exporting Google Cloud Logs to Google Security Operations

You can export the following types of Google Cloud data to your Google Security Operations instance:

Note: However you configure your export filters, you can only export the following Google Cloud logs.

• GCP_CLOUDAUDIT:
  • log_id("cloudaudit.googleapis.com/activity") (exported by the default filter)
  • log_id("cloudaudit.googleapis.com/system_event") (exported by the default filter)
  • log_id("cloudaudit.googleapis.com/policy")
  • log_id("cloudaudit.googleapis.com/access_transparency")
• GCP_CLOUD_NAT:
  • log_id("compute.googleapis.com/nat_flows")
• GCP_DNS:
  • log_id("dns.googleapis.com/dns_queries") (exported by the default filter)
• GCP_FIREWALL:
  • log_id("compute.googleapis.com/firewall")
• GCP_IDS:
  • log_id("ids.googleapis.com/threat")
  • log_id("ids.googleapis.com/traffic")
• GCP_LOADBALANCING:
  • log_id("requests")
  • log_id("loadbalancing.googleapis.com/external_regional_requests")
  • log_id("networksecurity.googleapis.com/network_dos_attack_mitigations")
  • log_id("networksecurity.googleapis.com/dos_attack")
  • This includes logs from Google Cloud Armor and Cloud Load Balancing
• GCP_CLOUDSQL:
  • log_id("cloudsql.googleapis.com/mysql-general.log")
  • log_id("cloudsql.googleapis.com/mysql.err")
  • log_id("cloudsql.googleapis.com/postgres.log")
  • log_id("cloudsql.googleapis.com/sqlagent.out")
  • log_id("cloudsql.googleapis.com/sqlserver.err")
• NIX_SYSTEM:
  • log_id("syslog")
  • log_id("authlog")
  • log_id("securelog")
• LINUX_SYSMON:
  • log_id("sysmon.raw")
• WINEVTLOG:
  • log_id("winevt.raw")
  • log_id("windows_event_log")
• BRO_JSON:
  • log_id("zeek_json_streaming_conn")
  • log_id("zeek_json_streaming_dhcp")
  • log_id("zeek_json_streaming_dns")
  • log_id("zeek_json_streaming_http")
  • log_id("zeek_json_streaming_ssh")
  • log_id("zeek_json_streaming_ssl")
• KUBERNETES_NODE:
  • log_id("events")
  • log_id("stdout")
  • log_id("stderr")
• AUDITD:
  • log_id("audit_log")
• GCP_APIGEE_X:
  • log_id("apigee-logs")
  • logName =~ "^projects/[\w\-]+/logs/apigee\.googleapis\.com[\w\-]*$"
  • Adjust the log filter's regular expression as needed
• GCP_RECAPTCHA_ENTERPRISE:
  • log_id("recaptchaenterprise.googleapis.com/assessment")
  • log_id("recaptchaenterprise.googleapis.com/annotation")
• GCP_RUN:
  • log_id("run.googleapis.com/stderr")
  • log_id("run.googleapis.com/stdout")
  • log_id("run.googleapis.com/requests")
  • log_id("run.googleapis.com/varlog/system")
• GCP_NGFW_ENTERPRISE:
  • log_id("networksecurity.googleapis.com/firewall_threat")

To export your Google Cloud logs to Google Security Operations, set the Google Cloud logs toggle to enabled. All of the Google Cloud log types listed above will be exported to your Google Security Operations instance.

For best practices on which log filters to use, see Security log analytics in Google Cloud.

Export Filter Settings

By default, your Cloud Audit logs (admin activity and system event) and Cloud DNS logs are sent to your Google Security Operations account. However, you can customize the export filter to include or exclude specific types of logs. The export filter is based on the Google logging query language.

To define a custom filter for your logs, complete the following steps:

1. Define your filter by creating a custom filter for your logs using the logging query language. The following documentation describes how to define this type of filter: /logging/docs/view/logging-query-language

2. Navigate to the Logs Explorer using the link provided on the Export Filter Settings tab, copy your new query into the Query field and click Run Query to test it.

   Verify that the matched logs displayed in the Logs Explorer are exactly what you intend to export to Google Security Operations.

Complete the following steps from the Export Filter Settings tab:

1. When the filter is ready, click the edit icon and paste the filter in the Export filter field.

2. Click Save Custom Filter. Your new custom filter works against all new logs exported to your Google Security Operations account.

3. You can reset the export filter to the default version by clicking Reset to Default. Be sure to save a copy of your custom filter first.

Tuning Cloud Audit log filters

Cloud Audit Data Access logs can produce a large volume of logs without much threat detection value. If you choose to send these logs to Google Security Operations, you should filter out logs that are generated by routine activities.

The following export filter captures data access logs and excludes high volume events such as Read and List operations of Cloud Storage and Cloud SQL:

  ( `log_id("cloudaudit.googleapis.com/data_access")`
  AND NOT protoPayload.methodName =~ "^storage\.(buckets|objects)\.(get|list)$"
  AND NOT protoPayload.request.cmd = "select" )

For more information on tuning Cloud Audit Data Access logs, see here.

Export Filter Examples

The following export filter examples illustrate how you can include or exclude certain types of logs from being exported to your Google Security Operations account.

Note: However you configure your export filters, you can only export the Google Cloud logs listed here.

Export Filter Example 1: Include additional log types

The following export filter exports access transparency logs in addition to the default logs:

log_id("dns.googleapis.com/dns_queries") OR
log_id("cloudaudit.googleapis.com/activity") OR
log_id("cloudaudit.googleapis.com/system_event") OR
log_id("cloudaudit.googleapis.com/access_transparency")

Export Filter Example 2: Include additional logs from a specific project

The following export filter exports access transparency logs from a specific project, in addition to the default logs:

log_id("dns.googleapis.com/dns_queries") OR
log_id("cloudaudit.googleapis.com/activity") OR
log_id("cloudaudit.googleapis.com/system_event") OR
logName = "projects/my-project-id/logs/cloudaudit.googleapis.com%2Faccess_transparency"

Export Filter Example 3: Include additional logs from a specific folder

The following export filter exports access transparency logs from a specific folder, in addition to the default logs:

log_id("dns.googleapis.com/dns_queries") OR
log_id("cloudaudit.googleapis.com/activity") OR
log_id("cloudaudit.googleapis.com/system_event") OR
logName = "folders/my-folder-id/logs/cloudaudit.googleapis.com%2Faccess_transparency"

Export Filter Example 4: Exclude logs from a specific project

The following export filter exports the default logs from the entire Google Cloud organization with the exception of a specific project:

(log_id("dns.googleapis.com/dns_queries") OR
log_id("cloudaudit.googleapis.com/activity") OR
log_id("cloudaudit.googleapis.com/system_event")) AND
(NOT logName =~ "^projects/my-project-id/logs/.*$")

Exporting Google Cloud Asset Metadata to Google Security Operations

You can export your Google Cloud Asset metadata to Google Security Operations. This asset metadata is drawn from your Google Cloud Asset Inventory and consists of information about your assets, resources, and identities including the following:

• Environment
• Location
• Zone
• Hardware models
• Access control relationships between resources and identities

The following are the specific types of Google Cloud Asset metadata that will be exported to your Google Security Operations account:

• GCP_BIGQUERY_CONTEXT
• GCP_CLOUD_FUNCTIONS_CONTEXT
• GCP_COMPUTE_CONTEXT
• GCP_IAM_CONTEXT
• GCP_IAM_ANALYSIS
• GCP_KUBERNETES_CONTEXT
• GCP_NETWORK_CONNECTIVITY_CONTEXT
• GCP_RESOURCE_MANAGER_CONTEXT
• GCP_SQL_CONTEXT
• GCP_STORAGE_CONTEXT

The following are some examples of Google Cloud Asset metadata:

• Application name—Google-iamSample/0.1
• Project name—projects/my-project

Examples of Resource Manager context log fields include assetType, resource.data.name, and resource.version.

For more information about resource types, see Cloud Asset Inventory supported resource types.

Note: You must have the Security Command Center Standard tier or the Security Command Center Premium tier enabled to export your Google Cloud Asset metadata to Google Security Operations.

To export your Google Cloud Asset metadata to Google Security Operations, set the Cloud Asset Metadata toggle to Enabled.

Field mapping reference and supported resource types

The following table lists the context parsers that Google Security Operations supports, the corresponding ingestion label, and the supported resource types.

To view the mapping reference documentation of the context parser, click the corresponding context parser name in the table.

Service name	Ingestion label	Supported resource types
BigQuery	GCP_BIGQUERY_CONTEXT	bigquery.googleapis.com/Dataset bigquery.googleapis.com/Model bigquery.googleapis.com/Table
Resource Manager	GCP_RESOURCE_MANAGER_CONTEXT	cloudresourcemanager.googleapis.com/Folder cloudresourcemanager.googleapis.com/Organization cloudresourcemanager.googleapis.com/Project cloudresourcemanager.googleapis.com/TagBinding cloudresourcemanager.googleapis.com/TagKey cloudresourcemanager.googleapis.com/TagValue
Cloud SQL	GCP_SQL_CONTEXT	sqladmin.googleapis.com/BackupRun sqladmin.googleapis.com/Instance
Cloud Functions	GCP_CLOUD_FUNCTIONS_CONTEXT	cloudfunctions.googleapis.com/CloudFunction cloudfunctions.googleapis.com/Function
Identity and Access Management	GCP_IAM_CONTEXT	iam.googleapis.com/ServiceAccount iam.googleapis.com/Role iam.googleapis.com/ServiceAccountKey
Network Connectivity Center	GCP_NETWORK_CONNECTIVITY_CONTEXT	networkconnectivity.googleapis.com/Hub networkconnectivity.googleapis.com/PolicyBasedRoutes networkconnectivity.googleapis.com/Spoke
Google Kubernetes Engine	GCP_KUBERNETES_CONTEXT	apps.k8s.io/Deployment apps.k8s.io/ReplicaSet batch.k8s.io/Job container.googleapis.com/Cluster container.googleapis.com/NodePool extensions.k8s.io/Ingress k8s.io/Namespace k8s.io/Node k8s.io/Pod k8s.io/Service networking.k8s.io/Ingress networking.k8s.io/NetworkPolicy rbac.authorization.k8s.io/ClusterRole rbac.authorization.k8s.io/ClusterRoleBinding rbac.authorization.k8s.io/Role rbac.authorization.k8s.io/RoleBinding
Compute Engine	GCP_COMPUTE_CONTEXT	compute.googleapis.com/Autoscaler compute.googleapis.com/Address compute.googleapis.com/BackendBucket compute.googleapis.com/BackendService compute.googleapis.com/Commitment compute.googleapis.com/Disk compute.googleapis.com/ExternalVpnGateway compute.googleapis.com/Firewall compute.googleapis.com/FirewallPolicy compute.googleapis.com/ForwardingRule compute.googleapis.com/GlobalAddress compute.googleapis.com/GlobalForwardingRule compute.googleapis.com/HealthCheck compute.googleapis.com/HttpHealthCheck compute.googleapis.com/HttpsHealthCheck compute.googleapis.com/Image compute.googleapis.com/Instance compute.googleapis.com/InstanceGroup compute.googleapis.com/InstanceGroupManager compute.googleapis.com/InstanceTemplate compute.googleapis.com/Interconnect compute.googleapis.com/InterconnectAttachment compute.googleapis.com/License compute.googleapis.com/MachineImage compute.googleapis.com/Network compute.googleapis.com/NetworkEndpointGroup compute.googleapis.com/NodeGroup compute.googleapis.com/NodeTemplate compute.googleapis.com/PacketMirroring compute.googleapis.com/Project compute.googleapis.com/PublicDelegatedPrefix compute.googleapis.com/RegionBackendService compute.googleapis.com/RegionDisk compute.googleapis.com/Reservation compute.googleapis.com/ResourcePolicy compute.googleapis.com/Route compute.googleapis.com/Router compute.googleapis.com/SecurityPolicy compute.googleapis.com/Snapshot compute.googleapis.com/SslCertificate compute.googleapis.com/SslPolicy compute.googleapis.com/Subnetwork compute.googleapis.com/ServiceAttachment compute.googleapis.com/TargetHttpProxy compute.googleapis.com/TargetHttpsProxy compute.googleapis.com/TargetInstance compute.googleapis.com/TargetPool compute.googleapis.com/TargetTcpProxy compute.googleapis.com/TargetSslProxy compute.googleapis.com/TargetVpnGateway compute.googleapis.com/UrlMap compute.googleapis.com/VpnGateway compute.googleapis.com/VpnTunnel

Exporting Security Command Center findings to Google Security Operations

You can export Security Command Center Premium Event Threat Detection (ETD) findings and all other findings to Google Security Operations.

For more information about Event Threat Detection findings, see the Security Command Center overview.

Note: You must have the Security Command Center Premium tier enabled to export your Premium tier findings to Google Security Operations.

To export your Security Command Center Premium tier findings to Google Security Operations, set the Security Command Center Premium Findings toggle to enabled.

Exporting Sensitive Data Protection data to Google Security Operations

To ingest Sensitive Data Protection asset metadata (DLP_CONTEXT), perform the following:

1. Enable Google Cloud data ingestion by completing the previous section in this document.
2. Configure Sensitive Data Protection to profile data.
3. Configure the scan configuration to publish data profiles to Google Security Operations.

See Sensitive Data Protection documentation for detailed information about creating data profiles for BigQuery data.

Disabling Google Cloud data ingestion

Note: By completing these steps, you will need to obtain a new one-time access code from your Google Security Operations representative and complete the Enabling Google Cloud data ingestion procedure again to restart Google Cloud log ingestion.

1. Check the box labeled I want to disconnect Google Security Operations and stop sending Google Cloud Logs into Google Security Operations.

2. Click Disconnect Google Security Operations.

   

Troubleshooting

• If the relationships between resources and identities are missing from your Google Security Operations system, set the Export Cloud logs to Google Security Operations toggle to disabled and then to enabled again.
• The asset metadata are periodically ingested into Google Security Operations. Allow several hours for changes to be visible at the Google Security Operations UI and APIs.

What's next

• Open your Google Security Operations account using the customer specific URL provided by your Google Security Operations representative.
• Learn more about Google Security Operations.