Changes to BigQuery metadata stored in Dataplex Catalog

Between October 15, 2024, and November 15, 2024, some of the BigQuery metadata that is stored in Dataplex Catalog is changing.

This change brings the metadata that is stored in Dataplex Catalog into consistency with BigQuery metadata and metadata from other Google Cloud sources.

If you have workloads that depend on BigQuery metadata, you must adjust them to preserve continuity.

For more information about Dataplex Catalog metadata, see Dataplex Catalog overview.

Scope of changes

The changes that are described in this section apply to all BigQuery entries. BigQuery entries are entries that belong to an entry type under the BigQuery system.

The following API methods are affected by these changes:

The changes to data type values and table subtype values are also reflected in the Google Cloud console, on the Entry Details page.

Data type values

The data type values are changing for the aspect type SchemaAspect. The field dataType is affected.

Value before the change Value after the change
INT64 INTEGER
FLOAT64 FLOAT
STRUCT RECORD
BOOL BOOLEAN

Resource names

The resource name is changing for the top-level entry field entrySource.resource. The field is modified to contain the resource ID instead of the full resource name.

Value before the change Value after the change
//bigquery.googleapis.com/projects/PROJECT_ID/datasets/DATASET/tables/TABLE projects/PROJECT_ID/datasets/DATASET/tables/TABLE

Ancestor type values

The ancestor type value is changing for the top-level entry field entrySource.ancestors.type.

Value before the change Value after the change
655216118709.global.Dataset dataplex-types.global.bigquery-dataset

What you need to do

The rollout period starts on October 15, 2024, and ends on November 15, 2024. The changes to BigQuery metadata will be applied gradually throughout the rollout period.

Here's what you need to do:

  • Before October 15, 2024: update your workloads to interpret both the legacy version of metadata and the updated version of metadata.

  • During the rollout period: BigQuery entries might capture the legacy version of metadata or the updated version of metadata. Your workloads should be able to interpret both versions.

    To determine whether a BigQuery entry uses the updated metadata, in the entry's aspects object, locate the relevant aspect type. In the aspect type's aspectSource object, the data_version field indicates the metadata version. For example, for a bigquery-table entry, refer to the field Entry.aspects.dataplex-types.global.bigquery-table.aspectSource.data_version.

    Uses legacy metadata version Uses updated metadata version
    "" (empty string) "Ingestion/1.0.0"

  • After November 15, 2024: optionally, update your workloads to interpret only the new version of metadata.

Support

For questions, contact dataplex-msa-support@google.com.