Ingest Google Cloud data to Google Security Operations

Supported in:

This page shows how to enable and disable the ingestion of your Google Cloud data into Google Security Operations. Google Security Operations lets you to store, search, and examine the aggregated security information for your enterprise going back for months or longer in accordance with your data retention period.

Overview

There are two options to send Google Cloud data to Google Security Operations. Choosing the right option depends on log type.

Option 1: Direct ingestion

A special Cloud Logging filter can be configured in Google Cloud to send specific log types to Google Security Operations in real-time. These logs are generated by Google Cloud services.

Available log types include:

  • Cloud Audit Logs
  • Cloud NAT
  • Cloud DNS
  • Cloud Next Generation Firewall
  • Cloud Intrusion Detection System
  • Cloud Load Balancing
  • Cloud SQL
  • Windows Event logs
  • Linux syslog
  • Linux Sysmon
  • Zeek
  • Google Kubernetes Engine
  • Audit Daemon (auditd)
  • Apigee
  • reCAPTCHA Enterprise
  • Cloud Run logs (GCP_RUN)

To collect Compute Engine or Google Kubernetes Engine (GKE) application logs (such as Apache, Nginx, or IIS), use Option 2. Additionally, raise a Support ticket with Google Security Operations to provide feedback for future consideration to support using direct ingestion (Option 1).

For specific log filters and more ingestion details, see Export Google Cloud Logs to Google Security Operations.

You can also send Google Cloud asset metadata that is used for context enrichment. See Export Google Cloud asset metadata to Google Security Operations for more details.

Option 2: Google Cloud Storage

Cloud Logging can route logs to Cloud Storage to be fetched by Google Security Operations on a scheduled basis.

For details about how to configure Cloud Storage for Google Security Operations, see Feed Management: Cloud Storage.

Before you begin

Before you can ingest Google Cloud data into a Google Security Operations instance, you must complete the following steps:

  1. Grant the following IAM roles required for you to access the Google Security Operations section:

    • Chronicle Service Admin (roles/chroniclesm.admin): IAM role for performing all activities.
    • Chronicle Service Viewer (roles/chroniclesm.viewer): IAM role to only view the state of ingestion.
    • Security Center Admin Editor (roles/securitycenter.adminEditor): Required to enable the ingestion of Cloud Asset Metadata.

  2. If you plan to enable Cloud Asset Metadata, you must also enable either the Security Command Center Standard tier, Security Command Center Premium tier, or Security Command Center Enterprise tier. See Overview of organization-level activation for more information.

Granting IAM Roles

You can grant the required IAM roles using either the Google Cloud console or using the gcloud CLI.

To grant IAM roles using Google Cloud console, complete the following steps:

  1. Log on to the Google Cloud organization you want to connect to and navigate to the IAM screen using Products > IAM & Admin > IAM.

  2. From the IAM screen, select the user and click Edit Member.

  3. In the Edit Permissions screen, click Add Another Role and search for Google Security Operations to find the IAM roles.

  4. Once you have assigned the roles, click Save.

To grant IAM roles using the Google Cloud CLI, complete the following steps:

  1. Ensure you are logged into the correct organization. Verify this by running the gcloud init command.

  2. To grant the Chronicle Service Admin IAM role using gcloud, run the following command:

    gcloud organizations add-iam-policy-binding ORGANIZATION_ID \
--member "user:USER_EMAIL" \
--role roles/chroniclesm.admin

    Replace the following:

    • ORGANIZATION_ID: the numeric organization ID.
    • USER_EMAIL: the user's email address.

  3. To grant the Chronicle Service Viewer IAM role using gcloud, run the following command:

    gcloud organizations add-iam-policy-binding ORGANIZATION_ID \
--member "user:USER_EMAIL" \
--role roles/chroniclesm.viewer

  4. To grant the Security Center Admin Editor role using gcloud, run the following command:

     gcloud organizations add-iam-policy-binding ORGANIZATION_ID \
 --member "user:USER_EMAIL" \
 --role roles/securitycenter.adminEditor`

Enable direct ingestion from Google Cloud

The steps to enable direct ingestion from Google Cloud are different depending on the ownership of the project that your Google Security Operations instance is bound to.

After you configure direct ingestion, your Google Cloud data is sent to Google Security Operations. You can use Google Security Operations's analysis features to investigate security related issues.

Configure ingestion when project owned by customer

Do the following steps if you own the Google Cloud project.

You can configure direct ingestion from multiple organizations using the same project-level configuration page. Do the following steps to create a new configuration and edit an existing configuration.

When you migrate an existing Google Security Operations instance so that it binds to a project that you own, and if direct ingestion was configured before the migration, the direct ingestion configuration is migrated as well.

  1. Navigate to the Google SecOps > Ingestion Settings page in the Google Cloud console.
    Go to the Google SecOps page
  2. Select the project that is bound to your Google Security Operations instance.

  3. In the Organization menu, select the organization from which logs will be exported. The menu displays organizations that you have have permission to access. The list can includes organizations that are not linked to the Google SecOps instance. You cannot configure an organization that sends data to a different Google SecOps instance.

    Select organization

  4. Under the Google Cloud Ingestion setting section, click the Sending data to Google Security Operations toggle to enable logs to be sent to Google Security Operations.

  5. Select one or more of the following options to define the type of data sent to Google Security Operations:

  6. Under the Customer export filter settings section, define export filters that customize the Cloud Logging data exported to Google Security Operations. See Export Google Cloud logs for the types of log data you export.

  7. To ingest logs from an additional organization to the same Google Security Operations instance, select the organization from the Organization menu, and then repeat the steps to define the type of data to export and export filters. You will see multiple organizations listed in the Organization menu.

  8. To export Google Cloud Cloud Data Loss Prevention data to Google Security Operations, see Export Google Cloud Cloud Data Loss Prevention data to Google Security Operations.

Configure ingestion when a project is owned by Google Cloud

If Google Cloud own's the project, do the following to configure direct ingestion from your Google Cloud organization into your Google Security Operations instance:

  1. Navigate to the Google SecOps > Overview > Ingestion tab in the Google Cloud console. Go to the Google SecOps Ingestion tab
  2. Click the Manage organization ingestion settings button.
  3. If you see the a Page not viewable for projects. message, select an organization, then click Select.
  4. Enter your one-time access code in the 1-time Google Security Operations access code field.
  5. Check the box labeled I consent to the terms and conditions of Google Security Operations's usage of my Google Cloud data.
  6. Click Connect Google SecOps.
  7. Go to the Global Ingestion Settings tab for the organization.

  8. Select the type of data that will be sent by enabling one or more of the following options:

  9. Go to Export Filter Settings tab.

  10. Under the Customer export filter settings section, define export filters that customize the Cloud Logging data exported to Google Security Operations. See Export Google Cloud logs for a the types of log data you export.

  11. To export Google Cloud Cloud Data Loss Prevention data to Google Security Operations, see Export Google Cloud Cloud Data Loss Prevention data to Google Security Operations.

Export Google Cloud logs

After you enable Cloud Logging, you can export the following types of Google Cloud data to your Google Security Operations instance, listed by type of log and Google Security Operations ingestion label:

  • Cloud Audit Logs(GCP_CLOUDAUDIT): this includes Admin Activity, System Event, Access Transparency and Policy Denied logs.
    • log_id("cloudaudit.googleapis.com/activity") (exported by the default filter)
    • log_id("cloudaudit.googleapis.com/system_event") (exported by the default filter)
    • log_id("cloudaudit.googleapis.com/policy")
    • log_id("cloudaudit.googleapis.com/access_transparency")
  • Cloud NAT logs(GCP_CLOUD_NAT):
    • log_id("compute.googleapis.com/nat_flows")
  • Cloud DNS logs (GCP_DNS):
    • log_id("dns.googleapis.com/dns_queries") (exported by the default filter)
  • Cloud Next Generation Firewall logs(GCP_FIREWALL):
    • log_id("compute.googleapis.com/firewall")
  • GCP_IDS:
    • log_id("ids.googleapis.com/threat")
    • log_id("ids.googleapis.com/traffic")
  • GCP_LOADBALANCING:
    • log_id("requests") This includes logs from Google Cloud Armor and Cloud Load Balancing
  • GCP_CLOUDSQL:
    • log_id("cloudsql.googleapis.com/mysql-general.log")
    • log_id("cloudsql.googleapis.com/mysql.err")
    • log_id("cloudsql.googleapis.com/postgres.log")
    • log_id("cloudsql.googleapis.com/sqlagent.out")
    • log_id("cloudsql.googleapis.com/sqlserver.err")
  • NIX_SYSTEM:
    • log_id("syslog")
    • log_id("authlog")
    • log_id("securelog")
  • LINUX_SYSMON:
    • log_id("sysmon.raw")
  • WINEVTLOG:
    • log_id("winevt.raw")
    • log_id("windows_event_log")
  • BRO_JSON:
    • log_id("zeek_json_streaming_conn")
    • log_id("zeek_json_streaming_dhcp")
    • log_id("zeek_json_streaming_dns")
    • log_id("zeek_json_streaming_http")
    • log_id("zeek_json_streaming_ssh")
    • log_id("zeek_json_streaming_ssl")
  • KUBERNETES_NODE:
    • log_id("events")
    • log_id("stdout")
    • log_id("stderr")
  • AUDITD:
    • log_id("audit_log")
  • GCP_APIGEE_X:
    • logName =~ "^projects/[\w\-]+/logs/apigee\.googleapis\.com[\w\-]*$"
  • GCP_RECAPTCHA_ENTERPRISE:
    • log_id("recaptchaenterprise.googleapis.com/assessment")
    • log_id("recaptchaenterprise.googleapis.com/annotation")
  • GCP_RUN:
    • log_id("run.googleapis.com/stderr")
    • log_id("run.googleapis.com/stdout")
    • log_id("run.googleapis.com/requests")
    • log_id("run.googleapis.com/varlog/system")
  • GCP_NGFW_ENTERPRISE:
    • log_id("networksecurity.googleapis.com/firewall_threat")

To export Google Cloud logs to Google Security Operations, set the Enable Cloud logs toggle to Enabled. The supported Google Cloud log types can be exported to your Google Security Operations instance.

For best practices on which log filters to use, see Security log analytics in Google Cloud.

Export filter settings

The following sections provide information on the export filters.

Custom export filter settings

By default, your Cloud Audit Logs (Admin Activity and System Event) and Cloud DNS logs are sent to your Google Security Operations instance. However, you can customize the export filter to include or exclude specific types of logs. The export filter is based on the Google logging query language.

To define a custom filter for your logs, complete the following steps:

  1. Define your filter by creating a custom filter for your logs using the logging query language. See Logging query language for information about how to define this type of filter.

  2. Navigate to the Google SecOps page in the Google Cloud console and select a project.
    Go to the Google Security Operations page

  3. Launch the Logs Explorer using the link provided on the Export Filter Settings tab.

  4. Copy your new query into the Query field and click Run Query to test it.

  5. Copy your new query into the Logs Explorer > Query field, and then click Run Query to test it.

  6. Verify that the matched logs displayed in the Logs Explorer are exactly what you intend to export to Google Security Operations. When the filter is ready, copy it to the Custom export filter settings section for Google Security Operations.

  7. Go back to the Custom export filter settings section on the Google SecOps page.

    Custom export filter settings section

  8. Click the edit icon for the Export filter field and paste the filter to the field.

  9. Click Save button. Your new custom filter works against all new logs exported to your Google Security Operations instance.

  10. You can reset the export filter to the default version by clicking Reset to Default. Be sure to save a copy of your custom filter first.

Tune Cloud Audit Logs filters

Data Access logs written by Cloud Audit Logs can produce a large volume of data without much threat detection value. If you choose to send these logs to Google Security Operations, you should filter out logs that are generated by routine activities.

The following export filter captures data access logs and excludes high volume events such as Read and List operations of Cloud Storage and Cloud SQL:

( log_id("cloudaudit.googleapis.com/data_access")
  AND NOT protoPayload.methodName =~ "^storage\.(buckets|objects)\.(get|list)$"
  AND NOT protoPayload.request.cmd = "select" )

For more information about tuning Data Access logs generated by Cloud Audit Logs, see Manage the volume of Data Access audit logs.

Export Filter examples

The following export filter examples illustrate how you can include or exclude certain types of logs from being exported to your Google Security Operations instance.

Export Filter Example: Include additional log types

The following export filter exports access transparency logs in addition to the default logs:

log_id("dns.googleapis.com/dns_queries") OR
log_id("cloudaudit.googleapis.com/activity") OR
log_id("cloudaudit.googleapis.com/system_event") OR
log_id("cloudaudit.googleapis.com/access_transparency")

Export Filter Example: Include additional logs from a specific project

The following export filter exports access transparency logs from a specific project, in addition to the default logs:

log_id("dns.googleapis.com/dns_queries") OR
log_id("cloudaudit.googleapis.com/activity") OR
log_id("cloudaudit.googleapis.com/system_event") OR
logName = "projects/my-project-id/logs/cloudaudit.googleapis.com%2Faccess_transparency"

Export Filter Example: Include additional logs from a specific folder

The following export filter exports access transparency logs from a specific folder, in addition to the default logs:

log_id("dns.googleapis.com/dns_queries") OR
log_id("cloudaudit.googleapis.com/activity") OR
log_id("cloudaudit.googleapis.com/system_event") OR
logName = "folders/my-folder-id/logs/cloudaudit.googleapis.com%2Faccess_transparency"

Export Filter Example: Exclude logs from a specific project

The following export filter exports the default logs from the entire Google Cloud organization with the exception of a specific project:

(log_id("dns.googleapis.com/dns_queries") OR
log_id("cloudaudit.googleapis.com/activity") OR
log_id("cloudaudit.googleapis.com/system_event")) AND
(NOT logName =~ "^projects/my-project-id/logs/.*$")

Export Google Cloud asset metadata

You can export your Google Cloud asset metadata from Cloud Asset Inventory to Google Security Operations. This asset metadata is drawn from your Cloud Asset Inventory and consists of information about your assets, resources, and identities including the following:

  • Environment
  • Location
  • Zone
  • Hardware models
  • Access control relationships between resources and identities

The following types of Google Cloud asset metadata will be exported to your Google Security Operations instance:

  • GCP_BIGQUERY_CONTEXT
  • GCP_COMPUTE_CONTEXT
  • GCP_IAM_CONTEXT
  • GCP_IAM_ANALYSIS
  • GCP_STORAGE_CONTEXT
  • GCP_CLOUD_FUNCTIONS_CONTEXT
  • GCP_SQL_CONTEXT
  • GCP_NETWORK_CONNECTIVITY_CONTEXT
  • GCP_RESOURCE_MANAGER_CONTEXT

The following are examples of Google Cloud asset metadata:

  • Application name—Google-iamSample/0.1
  • Project name—projects/my-project

To export Google Cloud asset metadata to Google Security Operations, set the Cloud Asset Metadata toggle to Enabled.

Enable Cloud Asset Metadata.

For more about context parsers, see Google Security Operations context parsers.

Export Security Command Center findings

You can export Security Command Center Premium Event Threat Detection findings and all other findings to Google Security Operations.

For more information about ETD findings, see Overview of Event Threat Detection.

To export your Security Command Center Premium findings to Google Security Operations, set the Security Command Center Premium Findings toggle to Enabled.

Export Sensitive Data Protection data to Google Security Operations

To ingest Sensitive Data Protection asset metadata (DLP_CONTEXT), perform the following:

  1. Enable Google Cloud data ingestion by completing the previous section in this document.
  2. Configure Sensitive Data Protection to profile data.
  3. Configure the scan configuration to publish data profiles to Google Security Operations.

See Sensitive Data Protection documentation for detailed information about creating data profiles for BigQuery data.

Disable Google Cloud data ingestion

The steps to disable direct ingestion of data from Google Cloud are different depending on how Google Security Operations is configured. Choose one of the following:

  • If your Google Security Operations instance is bound to a project that you own and manage, perform the following steps:

    1. Select the project that is bound to your Google Security Operations instance.
    2. In Google Cloud console, navigate to the Ingestion tab under Google SecOps.
      Go to the Google SecOps page
    3. In the Organization menu, select the organization from which logs are exported.
    4. Set the Sending data to Google Security Operations toggle to Disabled.
    5. If you configured data export from multiple Organizations, and you want to disable these as well, do these steps for each organization.

  • If your Google Security Operations instance is bound to a project that Google Cloud owns and manages, perform the following steps:

    1. Navigate to the Google SecOps > Ingestion page in the Google Cloud console.
      Go to the Google SecOps page
    2. IN the resource meny, select the organization that is bound to your Google Security Operations instance and that you are ingesting data from.
    3. Check the box labeled I want to disconnect Google Security Operations and stop sending Google Cloud Logs into Google Security Operations.
    4. Click Disconnect Google Security Operations.

Troubleshooting

  • If the relationships between resources and identities are missing from your Google Security Operations instance, disable and then re-enable direct ingestion of log data to Google Security Operations.
  • The Google Cloud asset metadata are periodically ingested into Google Security Operations. Allow several hours for changes to be visible at the Google Security Operations UI and APIs.

What's next

  • Open your Google Security Operations instance using the customer specific URL provided by your Google Security Operations representative.
  • Learn more about Google Security Operations.