Configure a data export to BigQuery in a self-managed Google Cloud project

Google Security Operations lets you export UDM data to a project that you own and manage. You no longer need a Google SecOps project that is provided by Google. You can only use the self-managed project that is linked to your Google SecOps instance. As this project is not owned by Google, you can set up IAM permissions on your own without depending on Google to do it for you. This change applies to both new and existing customers.

Note: This feature available only to the Legacy, Standard, and Enterprise customers.

Google SecOps exports the following categories of data to your BigQuery project:

• udm_events: data about UDM events.
• udm_events_aggregates: aggregated data that is summarized by each hour of normalized events.
• entity_graph: data about UDM entities.
• rule_detections: detections that are returned by rules run in Google SecOps.
• ioc_matches: IOC matches that are found against UDM events.

Retention period

If you are an existing customer and you enable this feature, the BigQuery data that has been exported to your Google-managed project stays in the respective project for the specified retention period.

The retention period begins from the date of the earliest data export:

• If old project data exists: The retention period starts from the earliest export date in the old project.
• If old project data is deleted: The retention period starts from the earliest export date in the new project.

By default, your data is automatically deleted after the default retention period as per the contract with Google SecOps is over. Google SecOps deletes the data between 0 and 12 hours after the retention period expires. If you want to keep your data for longer, contact your Google SecOps representative to set up infinite retention. Infinite retention means that Google does not clean up your data. Your data is stored in your project until you decide to delete it.

If you are an existing customer, your data from the existing Google-managed project is not migrated to the self-managed project. Because data isn't migrated, your data is located in two separate projects. To query the data across a time range that includes the self-managed project activation date, you need to complete one of the following actions:

• Use a single query that joins data across both projects.
• Run two separate queries on the respective projects, one for data before the self-managed project activation date and one for data after. When the retention period for your Google-managed project expires, that data is deleted. You can only query data that is within your Google Cloud project after that point.

Permissions required to export data

To access your BigQuery data, run your queries within BigQuery itself. Assign the following IAM roles to any user that needs access:

• BigQuery Data Viewer (roles/bigquery.dataViewer)
• BigQuery Job User (roles/bigquery.jobUser)
• Storage Object Viewer (roles/storage.objectViewer) You can also assign roles at dataset level. For more information, see BigQuery IAM roles and permissions.

Initiate BigQuery data export to your self-managed project

1. Create a Google Cloud project where you want your data to be exported. For more information, see Configure a Google Cloud project for Google SecOps.

   Note: The self-managed project has to be linked to the Google SecOps instance. Once this capability is enabled, you cannot revert to a Google-managed project.

2. Link your self-managed project to your Google SecOps instance to establish a connection between Google SecOps and your self-managed project. For more information, see Link Google Security Operations to Google Cloud services. After the Google SecOps representative enables the export for the data that you have selected, the data export process begins.

3. To validate that the data is exported to your self-managed project, check the tables under the datalake dataset in BigQuery.

You can write ad-hoc queries against Google SecOps data stored in BigQuery tables. You can also create more advanced analytics using other third-party tools that integrate with BigQuery.

All the resources created in the your self-managed Google Cloud project to enable exports including Cloud Storage bucket and BigQuery tables are in the same region as Google SecOps.

If you get an error like Unrecognized name: <field_name> at [<some_number>:<some_number>] when querying BigQuery, it means the field you are trying to access is not in your dataset. This error happens because your schema is dynamically generated during the export process.

For more information about Google SecOps data in BigQuery, see Google Security Operations data in BigQuery.