Between October 15, 2024, and November 15, 2024, some of the BigQuery metadata that is stored in Dataplex Catalog is changing.
This change brings the metadata that is stored in Dataplex Catalog into consistency with BigQuery metadata and metadata from other Google Cloud sources.
If you have workloads that depend on BigQuery metadata, you must adjust them to preserve continuity.
For more information about Dataplex Catalog metadata, see Dataplex Catalog overview.
Scope of changes
The changes that are described in this section apply to all BigQuery entries. BigQuery entries are entries that belong to an entry type under the BigQuery system.
The following API methods are affected by these changes:
The changes to data type values and table subtype values are also reflected in the Google Cloud console, on the Entry Details page.
Data type values
The data type values are changing for the aspect type SchemaAspect
. The field
dataType
is affected.
Value before the change | Value after the change |
---|---|
INT64 |
INTEGER |
FLOAT64 |
FLOAT |
STRUCT |
RECORD |
BOOL |
BOOLEAN |
Resource names
The resource name is changing for the top-level entry field
entrySource.resource
. The field is modified to contain the resource ID
instead of the full resource name.
Value before the change | Value after the change |
---|---|
//bigquery.googleapis.com/projects/PROJECT_ID/datasets/DATASET/tables/TABLE |
projects/PROJECT_ID/datasets/DATASET/tables/TABLE |
Ancestor type values
The ancestor type value is changing for the top-level entry field
entrySource.ancestors.type
.
Value before the change | Value after the change |
---|---|
655216118709.global.Dataset |
dataplex-types.global.bigquery-dataset |
What you need to do
The rollout period starts on October 15, 2024, and ends on November 15, 2024. The changes to BigQuery metadata will be applied gradually throughout the rollout period.
Here's what you need to do:
- Before October 15, 2024: update your workloads to interpret both the legacy version of metadata and the updated version of metadata.
During the rollout period: BigQuery entries might capture the legacy version of metadata or the updated version of metadata. Your workloads should be able to interpret both versions.
To determine whether a BigQuery entry uses the updated metadata, in the entry's
aspects
object, locate the relevant aspect type. In the aspect type'saspectSource
object, thedata_version
field indicates the metadata version. For example, for abigquery-table
entry, refer to the fieldEntry.aspects.dataplex-types.global.bigquery-table.aspectSource.data_version
.Uses legacy metadata version Uses updated metadata version ""
(empty string)"Ingestion/1.0.0"
After November 15, 2024: optionally, update your workloads to interpret only the new version of metadata.
Support
For questions, contact dataplex-msa-support@google.com.