This page documents production updates to Dataplex. Check this page for announcements about new or updated features, bug fixes, known issues, and deprecated functionality.
You can see the latest product updates for all of Google Cloud on the Google Cloud page, browse and filter all release notes in the Google Cloud console, or programmatically access release notes in BigQuery.
To get the latest product updates delivered to you, add the URL of this page to your feed reader, or add the feed URL directly.
November 05, 2024
Dataplex automatic discovery is available in public preview. Automatic discovery is a feature in BigQuery that lets you scan data in Cloud Storage buckets to extract and catalog metadata. Automatic discovery creates BigLake or external tables and object tables you can use for analytics and AI, and catalogs that data in Dataplex Catalog. For more information, see Discover and catalog Cloud storage data.
November 04, 2024
Project-based semantic search offered by Dataplex Search is available in Preview. Semantic search, powered by Gemini, simplifies the search process without the need for complex search syntax. It supports natural language queries. For more information, see Discover data using semantic search.
October 15, 2024
Some of the BigQuery metadata that is stored in Dataplex Catalog is changing. If you have workloads that depend on BigQuery metadata, you must adjust them to preserve continuity. For more information about the scope of this change and what you need to do, see Changes to BigQuery metadata stored in Dataplex Catalog.
October 10, 2024
In the data lineage list view, you can filter lineage information based on the time that lineage occurred. For more information, see About data lineage.
September 30, 2024
Managed connectivity pipelines are generally available (GA). Use a managed connectivity pipeline to extract metadata from third-party sources and import it into Dataplex Catalog. You develop your own connector that extracts metadata, and use Workflows for orchestration and scheduling.
For more information, see Managed connectivity overview, Import metadata from a custom source using Workflows, and Develop a custom connector for metadata import.
Also, the metadata import API methods are GA. For more information, see Import metadata using a custom pipeline.
August 28, 2024
Data insights is generally available (GA). Data insights offers an automated way to explore and understand your data. It uses Gemini to generate queries based on the metadata of a table, and helps you uncover patterns, assess data quality, and perform statistical analysis.
You generate data insights in BigQuery. You can view data insights in Dataplex and in BigQuery.
August 12, 2024
Data lineage list view is available in preview. The lineage list view displays full lineage information in a single table. For more information, see Data lineage list view.
July 29, 2024
Metadata import for Dataplex Catalog entries and their aspects is available in preview. For more information, see Import metadata.
July 24, 2024
Column-level data lineage for BigQuery is available in Preview for allowlisted users. The existing data lineage feature tracks how BigQuery data moves through your systems at the table level. Column-level lineage extends this feature to let you track BigQuery data movement at the column level.
To sign up for access, fill out the Column-level lineage sign-up form.
July 22, 2024
Dataplex Explore is deprecated. Please follow the instructions for how to migrate Dataplex Explore to BigQuery Studio.
July 08, 2024
Dataplex Catalog is generally available (GA). Dataplex Catalog provides a platform for storing, managing, and accessing your metadata.
For more information, see Dataplex Catalog overview, Search for data assets, Manage aspects and enrich metadata, and Manage entries and ingest custom sources.
July 03, 2024
Data Lineage now supports location organization policy. For more information, see Resource locations supported services.
June 30, 2024
Dataplex is available in the following regions:
- Berlin (
europe-west10) - Dallas (
us-south1) - Doha (
me-central1) - Johannesburg (
africa-south1) - Osaka (
asia-northeast2) - Tel Aviv (
me-west1) - Turin (
europe-west12)
May 28, 2024
Dataplex automatic data quality supports the following capabilities:
- Email notifications to alert people about the status and results of a data quality job
- Data quality scores that indicate the percentage of rules that passed
- API support for rule recommendations based on data profiling scans
For more information, see Use auto data quality and Auto data quality overview.
April 25, 2024
Dataplex automatic data quality supports the following capabilities:
- The SQL assertion rule type for custom SQL rules lets you check for an invalid state of a dataset.
- You can use the data reference parameter in a custom SQL rule to refer to a data source table and all of its precondition filters, instead of explicitly mentioning the table and its filters.
March 27, 2024
Data insights in Dataplex is available in Preview. Data insights offers an automated and intuitive way to explore and understand your data. It uses Gemini large language models to generate queries based on the metadata of a table, and lets you uncover patterns, assess data quality, and perform statistical analysis.
March 25, 2024
Automated cataloging of Vertex AI feature store is available in Preview. With this integration, you can discover Vertex AI feature groups and features across projects and regions using the Console or Dataplex API. Dataplex fully automates the process of ingesting and indexing metadata, while performing source IAM permission checks, providing a governed single-pane-of-glass experience for data and AI artifacts across Cloud services.
December 17, 2023
Automated cataloging of Spanner is generally available (GA) in Dataplex. With this integration, can discover Spanner instances, databases, and tables across projects and regions using the Console or the Dataplex API. The metadata ingestion and indexing operations are fully automated, with IAM permissions set at the data source, providing a critical foundation for data management and governance.
December 01, 2023
Automated cataloging of Vertex AI models and datasets is generally available (GA) in Dataplex. With this integration, you are able to discover Vertex AI models and datasets across projects and regions using the Dataplex Console and API. Dataplex fully automates the process of ingesting and indexing metadata, while performing source IAM permission checks, providing a governed single-pane-of-glass experience for data and AI artifacts across Cloud services.
October 06, 2023
Automated cataloging of Bigtable is generally available (GA) in Dataplex. With this integration, you can discover Bigtable tables and instances across projects and regions using the Console or theDataplex API. The metadata ingestion and indexing operations are fully automated, with IAM permissions set at the data source, providing a critical foundation for data management and governance.
October 03, 2023
Dataplex BigLake integration is generally available (GA). Dataplex BigLake integration lets you upgrade a Cloud Storage bucket to managed, creating BigLake tables and Object tables instead of external tables. This allows the application of column-level, row-level, and table-level policies, enabling fine-grained security and dynamic data masking.
September 29, 2023
Dataplex is available in the following regions:
- Delhi (
asia-south2) - Melbourne (
australia-southeast2) - Toronto (
northamerica-northeast2)
August 21, 2023
Dataplex automatic data quality and data profiling are generally available.
- Data profiling
- Jump start your data analytics with statistical insights, such as average values, unique values, data bounds, and top-N.
- Understand drifts and build anomaly models with the generated metadata.
- Publish data quality and data profiling information in the BigQuery console. Learn more.
- Profile data in BigQuery tables, views, BigLake, and external tables.
- Ease deployment through a managed, serverless, and zero-copy execution.
- Take advantage of advanced features like filtering, sampling, and saving results to a central BigQuery table.
- Automatic data quality
- Deliver trusted data by building an end-to-end data quality monitoring pipeline.
- View rule recommendations, enhance with business rules, monitor on a routine or in a pipeline, generate reports, get alerted on failures, and troubleshoot the issues.
- View quality information in the BigQuery UI for every table user to see. Learn more.
- Improve data quality in BigQuery tables, views, BigLake, and external tables.
- Ease deployment through managed, serverless, and zero-copy execution.
- Take advantage of advanced features like filtering, sampling, and saving results to a central BigQuery table.
August 14, 2023
Data lineage at entry level for spark jobs executed in Dataproc is GA.
August 01, 2023
Dataplex is available in the following regions:
- Los Angeles (
us-west2) - Salt Lake City (
us-west3) - Las Vegas (
us-west4) - Columbus (
us-east5) - Santiago (
southamerica-west1) - Finland (
europe-north1) - Warsaw (
europe-central2) - Madrid (
europe-southwest1) - Milan (
europe-west8) - Paris (
europe-west9) - Jakarta (
asia-southeast2)
May 18, 2023
- Dataplex auto data quality (AutoDQ) and data profiling can be used on any BigQuery tables, including tables that aren't part of a Dataplex lake. You don't need to create a Dataplex lake to run Dataplex AutoDQ and data profiling.
- Dataplex AutoDQ and data profiling support BigQuery views, BigLake tables, and BigQuery external tables.
- Dataplex AutoDQ and data profiling support sampling your data to reduce time and cost.
March 13, 2023
Dataplex data lineage is generally available (GA). Data lineage lets you track how data moves through your systems: where it comes from, where it is passed to, and what transformations are applied to it.
January 30, 2023
Dataplex business glossary is now available in Preview. Dataplex business glossary lets you manage business related terminologies and definitions across the organization, and use them for describing and discovering data entries.
Dataplex Attribute Store is now available in Preview. Dataplex Attribute Store lets you associate attributes (with behavior specifications, such as resource access and column access) with tables and columns.
December 22, 2022
Dataplex data lineage is now available in Preview. Data lineage lets you track how data moves through your systems: where it comes from, where it is passed to, and what transformations are applied to it.
December 16, 2022
Dataplex BigLake integration is now available in Preview. Dataplex BigLake integration allows upgrading a Cloud Storage bucket to managed, creating BigLake tables instead of external tables. This allows the manual application of column-level, row-level, and table-level policies.
December 12, 2022
Dataplex auto data quality (AutoDQ) is now available in Preview. Dataplex auto data quality helps data users build trust in their data with a turnkey and automated product that encapsulates the entire process of data quality.
Dataplex data profiling is now available in Preview. Dataplex data profiling helps data users build deeper understanding about their data by identifying common data characteristics. Dataplex utilizes this information to recommend the data quality rules as well.
December 01, 2022
Dataplex Source and Sink plugins are generally available (GA) in Cloud Data Fusion for ingesting and processing data.
October 20, 2022
Data exploration workbench (Explore) is generally available (GA). Explore provides a fully-managed, serverless data exploration experience powered by fully-governed collaboration, one-click scheduling, and interactive querying using Spark SQL scripts and Jupyter notebooks.
July 20, 2022
Dataplex is now unified with Data Catalog to provide a complete data management and governance experience with built-in data intelligence and automation capabilities. See Dataplex product overview.
May 23, 2022
The Dataplex Source and Sink plugins are available in Public Preview for ingesting and processing data in Cloud Data Fusion versions 6.6.0 and later.
May 02, 2022
Added support for scheduling Google-provided and custom Dataflow templates from the Dataplex page in the Cloud Console. Monitor these templates from the Dataplex page in the Cloud Console.
April 15, 2022
Dataplex Data Quality tasks support running data quality validations on BigQuery tables that may not be part of a Dataplex lake, and on GCS data that's available as a BigQuery external table.
March 25, 2022
February 17, 2022
Dataplex Explore is available in Preview. Explore provides a fully-managed, serverless data exploration experience that enables you to query your data using Apache SparkSQL queries and Jupyter notebooks.