Configure data export to BigQuery in a self-managed Google Cloud project

Google Security Operations lets you export Unified Data Model (UDM) data to a project that you own and manage. You can exclusively use a self-managed project linked to your Google SecOps instance and independently configure IAM permissions without relying on Google.

Google SecOps exports the following categories of data to your BigQuery project:

  • udm_events: Data about UDM events.
  • udm_events_aggregates: Aggregated data, summarized by each hour of normalized events.
  • entity_graph: Data about UDM entities.
  • rule_detections: Detections returned by rules run in Google SecOps.
  • ioc_matches: Indicators of compromise (IOC) matches found against UDM events.
  • ingestion_metrics: Data/Statistics about log ingestion.

Retention period

If you're an existing customer and you enable this feature, the BigQuery data that has been exported to your Google-managed project stays in the respective project for the specified retention period.

The retention period begins from the date of the earliest data export:

  • The retention period for BigQuery export is configurable per data source, and can be set to a maximum retention period equivalent to the default log retention period in Google SecOps.
  • If no retention period is specified, the default behavior is to keep exporting data without any cleanup or purging, to limit the retention period. In this case, you can directly create custom retention policies for the Cloud Storage bucket, where data is exported in the Bring your own project (BYOP) project for consumption as an external table in BigQuery.

Data migration for existing customers

If you're an existing customer, your data from the existing Google-managed project is not migrated to the self-managed project. Because data isn't migrated, your data is located in two separate projects. To query the data across a time range that includes the self-managed project activation date, you need to complete one of the following actions:

  • Use a single query that joins data across both projects.
  • Run two separate queries on the respective projects, one for data before the self-managed project activation date and one for data after. When the retention period for your Google-managed project expires, that data is deleted. You can only query data that is within your Google Cloud project after that point.

Permissions required to export data

To access your BigQuery data, run your queries within BigQuery itself. Assign the following IAM roles to any user that needs access:

Initiate BigQuery data export to your self-managed project

  1. Create a Google Cloud project where you want your data to be exported. For more information, see Configure a Google Cloud project for Google SecOps.

  2. Link your self-managed project to your Google SecOps instance to establish a connection between Google SecOps and your self-managed project. For more information, see Link Google Security Operations to Google Cloud services. After the Google SecOps representative enables the export for the data that you have selected, the data export process begins.

  3. To validate that the data is exported to your self-managed project, check the tables under the datalake dataset in BigQuery.

You can write ad-hoc queries against Google SecOps data stored in BigQuery tables. You can also create more advanced analytics using other third-party tools that integrate with BigQuery.

All the resources created in the your self-managed Google Cloud project to enable exports, including Cloud Storage bucket and BigQuery tables, are in the same region as Google SecOps.

If you get an error like Unrecognized name: <field_name> at [<some_number>:<some_number>] when querying BigQuery, it means the field you're trying to access is not in your dataset and because your schema is dynamically generated during the export process.

For more information about Google SecOps data in BigQuery, see Google Security Operations data in BigQuery.