Configure a data export to BigQuery in a self-managed Google Cloud project

During onboarding, your Google SecOps representative helps you connect your Google SecOps instance to a Google-owned project. Google sets up permissions in the project to grant access to the Google SecOps APIs so that Google SecOps can read and write data to the BigQuery instance that is enabled in the Google-owned project.

You no longer need a Google SecOps project that is provided by Google. You can now use a project that is owned and managed by you so that you have complete control over the data that is stored in the project. As this project is not owned by Google, you can set up IAM permissions on your own without depending on Google to do it for you. This change applies to both new and existing customers.

Note: This feature available only to the Standard and Enterprise customers.

Google SecOps exports the following categories of data to your BigQuery project:

• udm_events: data about UDM events.
• udm_events_aggregates: aggregated data that is summarized by each hour of normalized events.
• entity_graph: data about UDM entities.
• rule_detections: detections that are returned by rules run in Google SecOps.
• ioc_matches: IOC matches that are found against UDM events.

Retention period

If you are an existing customer and you enable this feature, the BigQuery data that has been exported to your Google-managed project stays in the respective project for the specified retention period.

The retention period begins from the date of the earliest data export:

• If old project data exists: The retention period starts from the earliest export date in the old project.
• If old project data is deleted: The retention period starts from the earliest export date in the new project.

By default, your data is automatically deleted after the default retention period as per the contract with Google SecOps is over. Google SecOps deletes the data between 0 and 12 hours after the retention period expires. If you want to keep your data for longer, contact your Google SecOps representative to set up infinite retention. Infinite retention means that Google does not clean up your data. Your data is stored in your project until you decide to delete it.

If you are an existing customer, your data from the existing Google-managed project is not migrated to the self-managed project. Because data isn't migrated, your data is located in two separate projects. To query the data across a time range that includes the self-managed project activation date, you need to complete one of the following actions:

• Use a single query that joins data across both projects.
• Run two separate queries on the respective projects, one for data before the self-managed project activation date and one for data after. When the retention period for your Google-managed project expires, that data is deleted. You can only query data that is within your Google Cloud project after that point.

Permissions required to export data

To access your BigQuery data, run your queries within BigQuery itself. Assign the following IAM roles to any user that needs access:

• BigQuery Data Viewer (roles/bigquery.dataViewer)
• BigQuery Job User (roles/bigquery.jobUser)
• Storage Object Viewer (roles/storage.objectViewer) You can also assign roles at dataset level. For more information, see BigQuery IAM roles and permissions.

Initiate BigQuery data export to your self-managed project

1. Create a Google Cloud project where you want your data to be exported. For more information, see Configure a Google Cloud project for Google SecOps.

   Note: After you have selected to export to a self-managed project, you cannot revert to a Google-managed project.

2. Contact your Google SecOps representative to enable the export and help you select categories of data to export.

3. Link your self-managed project to your Google SecOps instance to establish a connection between Google SecOps and your self-managed project. For more information, see Link Google Security Operations to Google Cloud services. After the Google SecOps representative enables the export for the data that you have selected, the data export process begins.

4. To validate that the data is exported to your self-managed project, check the tables under the datalake dataset in BigQuery.

You can write ad-hoc queries against Google SecOps data stored in BigQuery tables. You can also create more advanced analytics using other third-party tools that integrate with BigQuery.

All the resources created in the your self-managed Google Cloud project to enable exports including Cloud Storage bucket and BigQuery tables are in the same region as Google SecOps.

If you get an error like Unrecognized name: <field_name> at [<some_number>:<some_number>] when querying BigQuery, it means the field you are trying to access is not in your dataset. This error happens because your schema is dynamically generated during the export process.

For more information about Google SecOps data in BigQuery, see Google Security Operations data in BigQuery.