Ingest Google Cloud data to Chronicle

This page shows how to enable and disable the ingestion of your Google Cloud data into Chronicle. Chronicle enables you to store, search and examine the aggregated security information for your enterprise going back for months or longer in accordance to your data retention period.

Overview

There are two options to send Google Cloud data to Chronicle. Choosing the right option depends on log type.

Option 1: Native Ingestion

A special Cloud Logging filter can be configured in Google Cloud to send specific log types to Chronicle in real-time. These logs are generated by Google Cloud services.

Chronicle receives logs even if they're excluded at the project level in Google Cloud, but included in both the log export filter and organization-level Google Cloud logging. To exclude logs from Chronicle, you must update the Chronicle log export filter in Google Cloud.

Available log types include:

To collect Compute Engine or Google Kubernetes Engine (GKE) application logs (such as Apache, Nginx, or IIS), use Option 2. Additionally, raise a Support ticket with Chronicle to provide feedback for future consideration to support as a native log type via Option 1.

For specific log filters and more ingestion details, see Exporting Google Cloud Logs to Chronicle.

Additional Google Cloud metadata to be used as context for enrichment purposes can also be sent to Chronicle. See Exporting Google Cloud Asset Metadata to Chronicle for more details.

Option 2: Google Cloud Storage

Cloud Logging can route logs to Cloud Storage to be fetched by Chronicle on a scheduled basis.

For details on how to configure Cloud Storage for Chronicle, see Feed Management: Cloud Storage.

Before you begin

Before you can ingest your Google Cloud data into your Chronicle instance, you must complete the following steps:

  1. Contact your Chronicle representative and obtain the one-time access code you need to ingest your Google Cloud data.

  2. Grant the following IAM roles required for you to access the Chronicle section:

    • Chronicle Service Admin (roles/chroniclesm.admin): IAM role for performing all activities.
    • Chronicle Service Viewer (roles/chroniclesm.viewer): IAM role to only view the state of ingestion.
    • Security Center Admin Editor (roles/securitycenter.adminEditor): Required to enable the ingestion of Cloud Asset Metadata.
  3. If you plan to enable Cloud Asset Metadata, you must also enable either the Security Command Center Standard tier or Security Command Center Premium tier Google Cloud service. See Activate Security Command Center for an organization for more information.

Granting IAM Roles

You can grant the required IAM roles using either the Google Cloud console or using the gcloud CLI.

To grant IAM roles using Google Cloud console, complete the following steps:

  1. Log on to the Google Cloud Organization you want to connect to and navigate to the IAM screen using Products > IAM & Admin > IAM.
  2. From the IAM screen, select the user and click Edit Member.

  3. In the Edit Permissions screen, click Add Another Role and search for Chronicle to find the IAM roles.

  4. Once you have assigned the roles, click Save.

To grant IAM roles using the Google Cloud CLI, complete the following steps:

  1. Ensure you are logged into the correct organization. Verify this by running the gcloud init command.

  2. To grant the Chronicle Service Admin IAM role using gcloud, run the following command:

    gcloud organizations add-iam-policy-binding ORGANIZATION_ID \
    --member "user:USER_EMAIL" \
    --role roles/chroniclesm.admin
    

    Replace the following:

    • ORGANIZATION_ID: the numeric organization ID.
    • USER_EMAIL: the admin user's email address.
  3. To grant the Chronicle Service Viewer IAM role using gcloud, run the following command:

    gcloud organizations add-iam-policy-binding ORGANIZATION_ID \
    --member "user:USER_EMAIL" \
    --role roles/chroniclesm.viewer
    
  4. To grant the Security Center Admin Editor role using gcloud, run the following command:

     gcloud organizations add-iam-policy-binding ORGANIZATION_ID \
     --member "user:USER_EMAIL" \
     --role roles/securitycenter.adminEditor`
    

Enabling Google Cloud data ingestion

Google Cloud data is ingested using a private internal API between Security Command Center and Chronicle. The ingestion never reaches the external network and never uses IP addresses. Google accesses the Google Cloud logs directly based on the authentication done using the one-time code provided by Chronicle for Security Command Center.

To enable data ingestion from your Google Cloud organization into your Chronicle account, complete the following steps:

  1. Navigate to the Chronicle page for the Google Cloud console.
    Go to the Chronicle page

  2. Enter your one-time access code in the 1-time Chronicle access code field.

  3. Check the box labeled I consent to the terms and conditions of Chronicle's usage of my Google Cloud data.

  4. Click Connect Chronicle.

    Connect Chronicle page.

Your Google Cloud data is now going to be sent to Chronicle. You can use Chronicle's analysis features to investigate any security related issues. The sections that follow describe ways to adjust the types of Google Cloud data that are going to be sent to Chronicle

Exporting Google Cloud Logs to Chronicle

You can export the following types of Google Cloud data to your Chronicle account:

  • GCP_CLOUDAUDIT:
    • log_id("cloudaudit.googleapis.com/activity") (exported by the default filter)
    • log_id("cloudaudit.googleapis.com/system_event") (exported by the default filter)
    • log_id("cloudaudit.googleapis.com/policy")
    • log_id("cloudaudit.googleapis.com/access_transparency")
  • GCP_CLOUD_NAT:
    • log_id("compute.googleapis.com/nat_flows")
  • GCP_DNS:
    • log_id("dns.googleapis.com/dns_queries") (exported by the default filter)
  • GCP_FIREWALL:
    • log_id("compute.googleapis.com/firewall")
  • GCP_IDS:
    • log_id("ids.googleapis.com/threat")
    • log_id("ids.googleapis.com/traffic")
  • GCP_LOADBALANCING:
    • log_id("requests")
    • log_id("loadbalancing.googleapis.com/external_regional_requests")
    • log_id("networksecurity.googleapis.com/network_dos_attack_mitigations")
    • log_id("networksecurity.googleapis.com/dos_attack")
    • This includes logs from Google Cloud Armor and Cloud Load Balancing
  • GCP_CLOUDSQL:
    • log_id("cloudsql.googleapis.com/mysql-general.log")
    • log_id("cloudsql.googleapis.com/mysql.err")
    • log_id("cloudsql.googleapis.com/postgres.log")
    • log_id("cloudsql.googleapis.com/sqlagent.out")
    • log_id("cloudsql.googleapis.com/sqlserver.err")
  • NIX_SYSTEM:
    • log_id("syslog")
    • log_id("authlog")
    • log_id("securelog")
  • LINUX_SYSMON:
    • log_id("sysmon.raw")
  • WINEVTLOG:
    • log_id("winevt.raw")
    • log_id("windows_event_log")
  • BRO_JSON:
    • log_id("zeek_json_streaming_conn")
    • log_id("zeek_json_streaming_dhcp")
    • log_id("zeek_json_streaming_dns")
    • log_id("zeek_json_streaming_http")
    • log_id("zeek_json_streaming_ssh")
    • log_id("zeek_json_streaming_ssl")
  • KUBERNETES_NODE:
    • log_id("events")
    • log_id("stdout")
    • log_id("stderr")
  • AUDITD:
    • log_id("audit_log")
  • GCP_APIGEE_X:
    • log_id("apigee-logs")
    • logName =~ "^projects/[\w\-]+/logs/apigee\.googleapis\.com[\w\-]*$"
    • Adjust the log filter's regular expression as needed
  • GCP_RECAPTCHA_ENTERPRISE:
    • log_id("recaptchaenterprise.googleapis.com/assessment")
    • log_id("recaptchaenterprise.googleapis.com/annotation")
  • GCP_RUN:
    • log_id("run.googleapis.com/stderr")
    • log_id("run.googleapis.com/stdout")
    • log_id("run.googleapis.com/requests")
    • log_id("run.googleapis.com/varlog/system")
  • GCP_NGFW_ENTERPRISE:
    • log_id("networksecurity.googleapis.com/firewall_threat")

To export your Google Cloud logs to Chronicle, set the Google Cloud logs toggle to enabled. All of the Google Cloud log types listed above will be exported to your Chronicle account.

Google Cloud Logs

For best practices on which log filters to use, see Security log analytics in Google Cloud.

Export Filter Settings

By default, your Cloud Audit logs (admin activity and system event) and Cloud DNS logs are sent to your Chronicle account. However, you can customize the export filter to include or exclude specific types of logs. The export filter is based on the Google logging query language.

To define a custom filter for your logs, complete the following steps:

  1. Define your filter by creating a custom filter for your logs using the logging query language. The following documentation describes how to define this type of filter: https://cloud.google.com/logging/docs/view/logging-query-language

  2. Navigate to the Logs Explorer using the link provided on the EXPORT FILTER SETTINGS tab, copy your new query into the Query field and click Run Query to test it.

    Verify that the matched logs displayed in the Logs Explorer are exactly what you intend to export to Chronicle.

    Export Filter Settings tab

    Export Filter Settings tab

Complete the following steps from the Export Filter Settings tab:

  1. When the filter is ready, click the edit icon and paste the filter to the Export filter field on the EXPORT FILTER SETTINGS tab.

  2. Click Save Custom Filter. Your new custom filter works against all new logs exported to your Chronicle account.

  3. You can reset the export filter to the default version by clicking Reset to Default. Be sure to save a copy of your custom filter first.

Tuning Cloud Audit log filters

Cloud Audit Data Access logs can produce a large volume of logs without much threat detection value. If you choose to send these logs to Chronicle, you should filter out logs that are generated by routine activities.

The following export filter captures data access logs and excludes high volume events such as Read and List operations of Cloud Storage and Cloud SQL:

( `log_id("cloudaudit.googleapis.com/data_access")`
  AND NOT protoPayload.methodName =~ "^storage\.(buckets|objects)\.(get|list)$"
  AND NOT protoPayload.request.cmd = "select" )
``` none

For more information on tuning Cloud Audit Data Access logs, see [here](/architecture/security-log-analytics#manage_the_volume_of_data_access_audit_logs).

### Export Filter Examples

The following export filter examples illustrate how you can include or exclude certain types of logs from being exported to your Chronicle account.

Note: However you configure your export filters, you can only export the Google Cloud logs listed [here](#export-logs).

#### Export Filter Example 1: Include additional log types

The following export filter exports access transparency logs in addition to the default logs:

log_id("dns.googleapis.com/dns_queries") OR log_id("cloudaudit.googleapis.com/activity") OR log_id("cloudaudit.googleapis.com/system_event") OR log_id("cloudaudit.googleapis.com/access_transparency") ```

Export Filter Example 2: Include additional logs from a specific project

The following export filter exports access transparency logs from a specific project, in addition to the default logs:

log_id("dns.googleapis.com/dns_queries") OR
log_id("cloudaudit.googleapis.com/activity") OR
log_id("cloudaudit.googleapis.com/system_event") OR
logName = "projects/my-project-id/logs/cloudaudit.googleapis.com%2Faccess_transparency"

Export Filter Example 3: Include additional logs from a specific folder

The following export filter exports access transparency logs from a specific folder, in addition to the default logs:

log_id("dns.googleapis.com/dns_queries") OR
log_id("cloudaudit.googleapis.com/activity") OR
log_id("cloudaudit.googleapis.com/system_event") OR
logName = "folders/my-folder-id/logs/cloudaudit.googleapis.com%2Faccess_transparency"

Export Filter Example 4: Exclude logs from a specific project

The following export filter exports the default logs from the entire Google Cloud organization with the exception of a specific project:

(log_id("dns.googleapis.com/dns_queries") OR
log_id("cloudaudit.googleapis.com/activity") OR
log_id("cloudaudit.googleapis.com/system_event")) AND
(NOT logName =~ "^projects/my-project-id/logs/.*$")

Exporting Google Cloud Asset Metadata to Chronicle

You can export your Google Cloud Asset metadata to Chronicle. This asset metadata is drawn from your Google Cloud Asset Inventory and consists of information about your assets, resources, and identities including the following:

  • Environment
  • Location
  • Zone
  • Hardware models
  • Access control relationships between resources and identities

The following are the specific types of Google Cloud Asset metadata that will be exported to your Chronicle account:

  • GCP_BIGQUERY_CONTEXT
  • GCP_COMPUTE_CONTEXT
  • GCP_IAM_CONTEXT
  • GCP_IAM_ANALYSIS
  • GCP_STORAGE_CONTEXT

The following are some examples of Google Cloud Asset metadata:

  • Application name—Google-iamSample/0.1
  • Project name—projects/my-project

To export your Google Cloud Asset metadata to Chronicle, set the Cloud Asset Metadata toggle to Enabled.

Enable Cloud Asset Metadata.

The Google Cloud asset metadata surfaces in Chronicle from the log sources GCP IAM Context and GCP IAM Analysis.

Exporting Security Command Center findings to Chronicle

You can export Security Command Center Premium Event Threat Detection (ETD) findings and all other findings to Chronicle.

For more information about ETD findings, see here.

To export your Security Command Center Premium findings to Chronicle, set the Security Command Center Premium Findings toggle to enabled.

Enable Security Command Center Premium Findings.

Exporting Sensitive Data Protection data to Chronicle

To ingest Sensitive Data Protection asset metadata (DLP_CONTEXT), perform the following:

  1. Enable Google Cloud data ingestion by completing the previous section in this document.
  2. Configure Sensitive Data Protection to profile data.
  3. Configure the scan configuration to publish data profiles to Chronicle.

See Sensitive Data Protection documentation for detailed information about creating data profiles for BigQuery data.

Disabling Google Cloud data ingestion

  1. Check the box labeled I want to disconnect Chronicle and stop sending Google Cloud Logs into Chronicle.

  2. Click Disconnect Chronicle.

    Disconnect Chronicle.

Troubleshooting

  • If the relationships between resources and identities are missing from your Chronicle system, set the Export Cloud logs to Chronicle toggle to disabled and then to enabled again.
  • The asset metadata are periodically ingested into Chronicle. Allow several hours for changes to be visible at the Chronicle UI and APIs.

What's next

  • Open your Chronicle account using the customer specific URL provided by your Chronicle representative.
  • Learn more about Chronicle.