Ingest Google Cloud data to Google Security Operations

Supported in:

Google SecOps SIEM

This page describes how to enable and disable Google Cloud data ingestion into Google SecOps. This lets you store, search, and examine aggregated security information for your enterprise, going back for months or longer, in accordance with your data retention period.

Overview

There are two options to send Google Cloud data to Google SecOps. Choosing the right option depends on log type.

Option 1: Direct ingestion

A special Cloud Logging filter can be configured in Google Cloud to send specific log types to Google SecOps in real-time. These logs are generated by Google Cloud services.

Google Security Operations only ingests supported log types. Available log types include:

• Cloud Audit Logs
• Cloud NAT
• Cloud DNS
• Cloud Next Generation Firewall
• Cloud Intrusion Detection System
• Cloud Load Balancing
• Cloud SQL
• Windows Event logs
• Linux syslog
• Linux Sysmon
• Zeek
• Google Kubernetes Engine
• Audit Daemon (auditd)
• Apigee
• reCAPTCHA Enterprise
• Cloud Run logs (GCP_RUN)

For details about the specific log filters and more ingestion details, see Export Google Cloud logs to Google SecOps.

You can also send Google Cloud asset metadata used for context enrichment. For details, see Export Google Cloud asset metadata to Google SecOps.

Note: To collect Compute Engine or Google Kubernetes Engine (GKE) application logs (such as Apache, Nginx, or IIS), use Option 2. Additionally, raise a Support ticket with Google SecOps to provide your feedback in support of possibly collecting this type of log using direct ingestion (Option 1) in the future.

Note: At this time, the direct ingestion method does not support Namespace or Label assignment.

Option 2: Google Cloud Storage

Cloud Logging can route logs to Cloud Storage by Google SecOps on a scheduled basis.

Note: This option will generate additional costs in Google Cloud account for the use of Cloud Storage.

For details about how to configure Cloud Storage for Google SecOps, see Feed Management: Cloud Storage.

Before you begin

Before you can ingest Google Cloud data into a Google SecOps instance, you must complete the following steps:

1. Grant the following IAM roles required for you to access the Google SecOps section:

   • Chronicle Service Admin (roles/chroniclesm.admin): IAM role for performing all activities.
   • Chronicle Service Viewer (roles/chroniclesm.viewer): IAM role to only view the state of ingestion.
   • Security Center Admin Editor (roles/securitycenter.adminEditor): Required to enable the ingestion of Cloud Asset Metadata.

2. If you plan to enable Cloud Asset Metadata, you must onboard the organization to the Security Command Center. See Overview of organization-level activation for more information.

Granting IAM Roles

You can grant the required IAM roles using either the Google Cloud console or using the gcloud CLI.

To grant IAM roles using Google Cloud console, complete the following steps:

1. Sign in to the Google Cloud organization you want to connect to and go to the IAM screen using Products > IAM & Admin > IAM.

2. From the IAM screen, select the user and click Edit Member.

   Note: If you're not in the Organization view of IAM, the Edit Member button is disabled. To enable it, go to the organization's IAM screen.

3. In the Edit Permissions screen, click Add Another Role and search for Google SecOps to find the IAM roles.

4. Once you have assigned the roles, click Save.

To grant IAM roles using the Google Cloud CLI, complete the following steps:

1. Ensure you are logged into the correct organization. Verify this by running the gcloud init command.

2. To grant the Chronicle Service Admin IAM role using gcloud, run the following command:

   gcloud organizations add-iam-policy-binding ORGANIZATION_ID \
   --member "user:USER_EMAIL" \
   --role roles/chroniclesm.admin
   

   Replace the following:

   • ORGANIZATION_ID: the numeric organization ID.
   • USER_EMAIL: the user's email address.

3. To grant the Chronicle Service Viewer IAM role using gcloud, run the following command:

   gcloud organizations add-iam-policy-binding ORGANIZATION_ID \
   --member "user:USER_EMAIL" \
   --role roles/chroniclesm.viewer
   

4. To grant the Security Center Admin Editor role using gcloud, run the following command:

    gcloud organizations add-iam-policy-binding ORGANIZATION_ID \
    --member "user:USER_EMAIL" \
    --role roles/securitycenter.adminEditor`
   

Enable direct ingestion from Google Cloud

The steps to enable direct ingestion from Google Cloud are different depending on the ownership of the project that your Google SecOps instance is bound to.

• If it is bound to a project that you own and manage, follow the steps in Configure ingestion when project owned by customer. This approach lets you configure the data ingestion from more than one Google Cloud organization.

• If it is bound to a project that Google Cloud owns and manages, follow the steps in Configure ingestion when project owned by Google Cloud.

After you configure direct ingestion, your Google Cloud data is sent to Google SecOps. You can use Google SecOps's analysis features to investigate security related issues.

Configure ingestion when project owned by customer

Do the following steps if you own the Google Cloud project.

You can configure direct ingestion from multiple organizations using the same project-level configuration page. Do the following steps to create a new configuration and edit an existing configuration.

When you migrate an existing Google SecOps instance so that it binds to a project that you own, and if direct ingestion was configured before the migration, the direct ingestion configuration is migrated as well.

1. Go to the Google SecOps > Ingestion Settings page in the Google Cloud console.
   Go to the Google SecOps page
2. Select the project that is bound to your Google SecOps instance.

3. In the Organization menu, select the organization from which logs will be exported. The menu displays organizations you have permission to access. The list can include organizations that aren't linked to the Google SecOps instance. You cannot configure an organization that sends data to a different Google SecOps instance.

   

4. Under the Google Cloud Ingestion setting section, click the Sending data to Google Security Operations toggle to enable logs to be sent to Google SecOps.

5. Select one or more of the following options to define the type of data sent to Google SecOps:

   • Google Cloud Logging: For more information about this option, see Export Google Cloud logs.
   • Cloud Asset Metadata: For more information about this option, see Export Google Cloud asset metadata.
   • Security Center Premium findings: For more information about this option, see Export Security Center Premium findings.

6. Under the Customer export filter settings section, configure export filters to customize the Cloud Logging data sent to Google SecOps. See Google Cloud log types supported for export.

7. To ingest logs from an additional organization to the same Google SecOps instance, select the organization from the Organization menu, and then repeat the steps to define the type of data to export and export filters. You will see multiple organizations listed in the Organization menu.

8. To export Sensitive Data Protection data (previously called Google Cloud Data Loss Prevention data) to Google SecOps, see Export Sensitive Data Protection data.

Configure ingestion when a project is owned by Google Cloud

If Google Cloud owns the project, do the following to configure direct ingestion from your Google Cloud organization into your Google SecOps instance:

1. Go to the Google SecOps > Overview > Ingestion tab in the Google Cloud console. Go to the Google SecOps Ingestion tab
2. Click the Manage organization ingestion settings button.
3. If a Page not viewable for projects message appears, select an organization, and click Select.
4. Enter your one-time access code in the 1-time Google SecOps access code field.
5. Check the box labeled I consent to the terms and conditions of Google SecOps's usage of my Google Cloud data.
6. Click Connect Google SecOps.
7. Go to the Global Ingestion Settings tab for the organization.

8. Select the type of data that will be sent by enabling one or more of the following options:

   • Google Cloud Logging: For more information about this option, see Export Google Cloud Logs.
   • Cloud Asset Metadata For more information about this option, see Export Google Cloud asset metadata.
   • Security Center Premium findings: For more information about this option, see Export Security Center Premium findings.

9. Go to the Export Filter Settings tab.

10. Under the Customer export filter settings section, configure export filters to customize the Cloud Logging data sent to Google SecOps. See Google Cloud log types supported for export.

11. To export Sensitive Data Protection data (previously called Google Cloud Data Loss Prevention data) to Google SecOps, see Export Sensitive Data Protection data.

Export Google Cloud logs

After enabling Cloud Logging, you can export log data for the supported Google Cloud log types to your Google SecOps instance.

To export Google Cloud logs to Google SecOps, set the Enable Cloud logs toggle to Enabled.

Note: You can only export Google Cloud log types to Google SecOps that are supported for export.

Log types supported for export

You can customize the export filter of the logs to export to Google SecOps. Include or exclude log types by adding or removing supported export filters listed in the following list:

Note: Use the text of the individual export filters without modification.

You can export the following Google Cloud log types to your Google SecOps instance. The following list is organized by log type and corresponding Google SecOps ingestion label:

• Cloud Audit Logs (GCP_CLOUDAUDIT):

  This includes: Admin Activity, System Event, Access Transparency, and Policy Denied logs.

  • log_id("cloudaudit.googleapis.com/activity") (exported by the default filter)
  • log_id("cloudaudit.googleapis.com/system_event") (exported by the default filter)
  • log_id("cloudaudit.googleapis.com/policy")
  • log_id("cloudaudit.googleapis.com/access_transparency")

• Cloud NAT logs (GCP_CLOUD_NAT):

  • log_id("compute.googleapis.com/nat_flows")

• Cloud DNS logs (GCP_DNS):

  • log_id("dns.googleapis.com/dns_queries") (exported by the default filter)

• Cloud Next Generation Firewall logs (GCP_FIREWALL):

  • log_id("compute.googleapis.com/firewall")

• GCP_IDS:

  • log_id("ids.googleapis.com/threat")
  • log_id("ids.googleapis.com/traffic")

• GCP_LOADBALANCING:

  This includes logs from Google Cloud Armor and Cloud Load Balancing.

  • log_id("requests")

• GCP_CLOUDSQL:

  • log_id("cloudsql.googleapis.com/mysql-general.log")
  • log_id("cloudsql.googleapis.com/mysql.err")
  • log_id("cloudsql.googleapis.com/postgres.log")
  • log_id("cloudsql.googleapis.com/sqlagent.out")
  • log_id("cloudsql.googleapis.com/sqlserver.err")

• NIX_SYSTEM:

  • log_id("syslog")
  • log_id("authlog")
  • log_id("securelog")

• LINUX_SYSMON:

  • log_id("sysmon.raw")

• WINEVTLOG:

  • log_id("winevt.raw")
  • log_id("windows_event_log")

• BRO_JSON:

  • log_id("zeek_json_streaming_conn")
  • log_id("zeek_json_streaming_dhcp")
  • log_id("zeek_json_streaming_dns")
  • log_id("zeek_json_streaming_http")
  • log_id("zeek_json_streaming_ssh")
  • log_id("zeek_json_streaming_ssl")

• KUBERNETES_NODE:

  • log_id("events")
  • log_id("stdout")
  • log_id("stderr")

• AUDITD:

  • log_id("audit_log")

• GCP_APIGEE_X:

  • log_id("apigee.googleapis.com/ingress_instance")
  • log_id("apigee.googleapis.com")
  • log_id("apigee-logs")
  • log_id("apigee")
  • logName =~ "^projects/[\w\-]+/logs/apigee[\w\-\.]*$"

• GCP_RECAPTCHA_ENTERPRISE:

  • log_id("recaptchaenterprise.googleapis.com/assessment")
  • log_id("recaptchaenterprise.googleapis.com/annotation")

• GCP_RUN:

  • log_id("run.googleapis.com/stderr")
  • log_id("run.googleapis.com/stdout")
  • log_id("run.googleapis.com/requests")
  • log_id("run.googleapis.com/varlog/system")

• GCP_NGFW_ENTERPRISE:

  • log_id("networksecurity.googleapis.com/firewall_threat")

Customize export filter settings

By default, your Cloud Audit Logs (Admin Activity and System Event) and Cloud DNS logs are sent to your Google SecOps instance. However, you can customize the export filter to include or exclude specific types of logs.

Note: You can only export log types to Google SecOps that are listed in: Google Cloud log types supported for export, and that have the following log ID and log name format: log_id = xx, logName = xxx.

To define a custom filter for your logs, do the following:

1. Identify the logs for your custom filter using the log scoping tool.

2. In the Auto-generated log filter section that follows the log scoping tool, copy the generated custom log filter code.

3. Go to the Google SecOps page in the Google Cloud console and select a project.
   Go to the Google SecOps page
   

4. Launch the Logs Explorer using the link on the Export Filter Settings tab.

5. Copy your new query into the Query field and click Run Query to test it.

6. Copy your new query into the Logs Explorer > Query field, and then click Run Query to test it.

7. Verify that the matched logs displayed in the Logs Explorer are exactly what you intend to export to Google SecOps. When the filter is ready, copy it to the Custom export filter settings section for Google SecOps.

8. Go back to the Custom export filter settings section on the Google SecOps page.

9. Click the Edit icon in the Export filter field, and then paste the copied filter into the field.

10. Click Save.

    • If the following error message appears: "The provided filter can potentially allow unsupported log types", there may be an unsupported log type included in the export filter. Remove the unsupported log type from the export filter. Only include log types listed in: Google Cloud log types supported for export.

    • If the save is successful, your new custom filter works against all new logs exported to your Google SecOps instance.

    • Optional: To reset the export filter to the default version, save a copy of your custom filter, and then click Reset to Default.

Tune Cloud Audit Logs filters

Data Access logs written by Cloud Audit Logs can produce a large volume of data without much threat detection value. If you choose to send these logs to Google SecOps, you should filter out logs that are generated by routine activities.

The following export filter captures data access logs and excludes high volume events such as Read and List operations of Cloud Storage and Cloud SQL:

( log_id("cloudaudit.googleapis.com/data_access")
  AND NOT protoPayload.methodName =~ "^storage\.(buckets|objects)\.(get|list)$"
  AND NOT protoPayload.request.cmd = "select" )

For more information about tuning Data Access logs generated by Cloud Audit Logs, see Manage the volume of Data Access audit logs.

Export Filter examples

The following export filter examples illustrate how you can include or exclude certain types of logs from the export to your Google SecOps instance.

Note: You can only export Google Cloud logs listed in: Google Cloud log types supported for export.

Export Filter Example: Include additional log types

The following export filter exports access transparency logs in addition to the default logs:

log_id("dns.googleapis.com/dns_queries") OR
log_id("cloudaudit.googleapis.com/activity") OR
log_id("cloudaudit.googleapis.com/system_event") OR
log_id("cloudaudit.googleapis.com/access_transparency")

Export Filter Example: Include additional logs from a specific project

The following export filter exports access transparency logs from a specific project, in addition to the default logs:

log_id("dns.googleapis.com/dns_queries") OR
log_id("cloudaudit.googleapis.com/activity") OR
log_id("cloudaudit.googleapis.com/system_event") OR
logName = "projects/my-project-id/logs/cloudaudit.googleapis.com%2Faccess_transparency"

Export Filter Example: Include additional logs from a specific folder

The following export filter exports access transparency logs from a specific folder, in addition to the default logs:

log_id("dns.googleapis.com/dns_queries") OR
log_id("cloudaudit.googleapis.com/activity") OR
log_id("cloudaudit.googleapis.com/system_event") OR
logName = "folders/my-folder-id/logs/cloudaudit.googleapis.com%2Faccess_transparency"

Export Filter Example: Exclude logs from a specific project

The following export filter exports the default logs from the entire Google Cloud organization with the exception of a specific project:

(log_id("dns.googleapis.com/dns_queries") OR
log_id("cloudaudit.googleapis.com/activity") OR
log_id("cloudaudit.googleapis.com/system_event")) AND
(NOT logName =~ "^projects/my-project-id/logs/.*$")

Export Google Cloud asset metadata

You can export your Google Cloud asset metadata from Cloud Asset Inventory to Google SecOps. This asset metadata is drawn from your Cloud Asset Inventory and consists of information about your assets, resources, and identities including the following:

• Environment
• Location
• Zone
• Hardware models
• Access control relationships between resources and identities

The following types of Google Cloud asset metadata will be exported to your Google SecOps instance:

• GCP_BIGQUERY_CONTEXT
• GCP_COMPUTE_CONTEXT
• GCP_IAM_CONTEXT
• GCP_IAM_ANALYSIS
• GCP_STORAGE_CONTEXT
• GCP_CLOUD_FUNCTIONS_CONTEXT
• GCP_SQL_CONTEXT
• GCP_NETWORK_CONNECTIVITY_CONTEXT
• GCP_RESOURCE_MANAGER_CONTEXT

The following are examples of Google Cloud asset metadata:

• Application name—Google-iamSample/0.1
• Project name—projects/my-project

Note: You need to have either Security Command Center Standard or Security Command Center Premium enabled to export Google Cloud asset metadata to Google SecOps.

To export Google Cloud asset metadata to Google SecOps, set the Cloud Asset Metadata toggle to Enabled.

For more about context parsers, see Google SecOps context parsers.

Export Security Command Center findings

You can export Security Command Center Premium Event Threat Detection findings and all other findings to Google SecOps.

For more information about ETD findings, see Overview of Event Threat Detection.

Note: You must have Security Command Center Premium enabled at the organization level to export your Premium findings to Google SecOps.

To export your Security Command Center Premium findings to Google SecOps, set the Security Command Center Premium Findings toggle to Enabled.

Export Sensitive Data Protection data

Note: Google Cloud Data Loss Prevention (Cloud DLP) is now a part of Sensitive Data Protection.

You can export your Sensitive Data Protection data to Google SecOps.

To ingest Sensitive Data Protection asset metadata (DLP_CONTEXT), perform the following:

1. Enable Google Cloud data ingestion by completing the previous section in this document.
2. Configure Sensitive Data Protection to profile data.
3. Set the scan configuration to publish data profiles to Google SecOps.

See Sensitive Data Protection documentation for detailed information about creating data profiles for BigQuery data.

Disable Google Cloud data ingestion

The steps to disable direct ingestion of data from Google Cloud are different depending on how Google SecOps is configured. Choose one of the following:

• If your Google SecOps instance is bound to a project that you own and manage, perform the following steps:

  1. Select the project that is bound to your Google SecOps instance.
  2. In the Google Cloud console, go to the Ingestion tab under Google SecOps.
     Go to the Google SecOps page
  3. In the Organization menu, select the organization from which logs are exported.
  4. Set the Sending data to Google Security Operations toggle to Disabled.
  5. If you configured data export from multiple Organizations, and you want to disable these as well, do these steps for each organization.

• If your Google SecOps instance is bound to a project that Google Cloud owns and manages, perform the following steps:

  Note: After completing these steps, you will need to obtain a new one-time access code from your Google SecOps representative and complete the procedure to enable direct ingestion from Google Cloud to restart Google Cloud log ingestion.
  1. Go to the Google SecOps > Ingestion page in the Google Cloud console.
     Go to the Google SecOps page
  2. In the resource menu, select the organization that is bound to your Google SecOps instance and that you are ingesting data from.
  3. Check the box labeled I want to disconnect Google SecOps and stop sending Google Cloud Logs into Google SecOps.
  4. Click Disconnect Google SecOps.

Control the rate of ingestion

When the data ingestion rate for a tenant reaches a certain threshold, Google Security Operations restricts the rate of ingestion for new data feeds to prevent a source with a high ingestion rate from affecting the ingestion rate of another data source. In this case, there is a delay but no data is lost. The ingestion volume and tenant's usage history determine the threshold.

You can request a rate limit increase by contacting Cloud Customer Care.

Troubleshooting

• If the relationships between resources and identities are missing from your Google SecOps instance, disable and then re-enable direct ingestion of log data to Google SecOps.
• Google Cloud asset metadata is periodically ingested into Google SecOps. Allow several hours for changes to appear in the Google SecOps UI and APIs.

• When you add a log type to the export filter, you may see this message: "The provided filter can potentially allow unsupported log types".

  Workaround: Only include log types to the export filter, that appear in the following list: Google Cloud log types supported for export.

What's next

• Open your Google SecOps instance using the customer specific URL provided by your Google SecOps representative.
• Learn more about Google SecOps.

Need more help? Get answers from Community members and Google SecOps professionals.