See Dataproc Metastore version policy for a list of current and past supported software versions.
You can see the latest product updates for all of Google Cloud on the Google Cloud page, browse and filter all release notes in the Google Cloud console, or programmatically access release notes in BigQuery.
To get the latest product updates delivered to you, add the URL of this page to your
feed
reader, or add the feed URL directly: https://cloud.google.com/feeds/metastore-release-notes.xml
April 01, 2024
Dataproc Metastore now supports managed migrations.
Managed migration is an automated feature that helps you migrate data from a self-managed Hive Metastore to a Dataproc Metastore service, without any sizable down time.
Dataproc Metastore now supports autoscaling. Autoscaling automatically increases or decreases the scaling factor required to run your workloads.
March 08, 2024
Dataproc Metastore now supports scheduled backups. Backups can be scheduled to run at user-specified cron job intervals, including running daily, weekly, or monthly.
November 06, 2023
The Data Catalog Sync feature is generally available (GA). With this launch, Data Catalog also now supports syncing metadata from Dataproc Metastore services using the Spanner Database.
October 31, 2023
Dataproc Metastore is available in the following multi-regional configurations, nam11 and eur5.
October 25, 2023
Dataproc Metastore is now available in the me-central2 (Dammam) region. For more information, see Dataproc Metastore locations.
October 16, 2023
Dataproc Metastore now supports multi-regional configurations.
May 31, 2023
Dataproc Metastore gRPC endpoints are generally available (GA).
Metadata federation support for BigQuery and BigLake is generally available (GA).
May 08, 2023
Administrator interface is generally available (GA). The GA release adds support for altering Hive table properties.
March 27, 2023
Metadata federation now supports Dataplex lakes as a metadata source (in preview)
March 09, 2023
Dataproc Metastore 2 is now Generally Available (GA). Dataproc Metastore 2 provides horizontal scalability through fine grained scaling options. For more information, see Datproc Metastore versions.
The Spanner database type is generally available (GA).
Auxiliary versions is generally available (GA).
January 19, 2023
Dataproc Metastore is available in the following regions: asia-east2 (Hong Kong), europe-central2 (Warsaw), europe-north1 (Finland), and us-west4 (Las Vagas). For more information, see Dataproc Metastore locations.
December 05, 2022
Dataproc Metastore administrator interface is available in preview.
The administrator interface provides you with a centralized tool to inspect and manage the metadata stored in your Dataproc Metastore service.
September 23, 2022
Metadata federation now supports BigQuery datasets as a metadata source (in preview).
July 18, 2022
Dataproc Metastore is available in the following regions: us-west2 (Los Angeles), us-west3 (Salt Lake City), europe-west4 (Netherlands), europe-west6 (Zürich), and asia-east1 (Taiwan). For more information, see Dataproc Metastore locations.
Note that these services are immediately available through the gcloud CLI and the REST API. Cloud console availability will vary by region over the next few weeks.
June 30, 2022
Metadata federation is generally available (GA).
Metadata federation lets you access metadata that is stored in multiple Dataproc Metastore instances.
To set up a federation, you create a federation service and then configure multiple Dataproc Metastore instances as your backend metastores. The federation service then exposes a single gRPC endpoint, which you can use to access metadata across all of your metastore instances.
Private Service Connect for Dataproc Metastore is generally available (GA).
June 06, 2022
Updated Dataproc Metastore auxiliary versions to support the Spanner database type.
March 01, 2022
gRPC endpoint protocol is available in Preview.
February 28, 2022
Fixed the issue causing metadata batch sync from Dataproc Metastore to Data Catalog to not work.
February 19, 2022
Fixed the issue causing Dataproc Metastore service creation to fail with the error NO_MATCHING_ACCESS_LEVEL due to a known issue where dns.googleapis.com is in the service perimeter but not in the allowlist.
February 11, 2022
Performing import, export, backup, or restore on Spanner-backed services now returns a 4XX error since these operations aren't supported.
Added additional mutual exclusion validation for Data Catalog and Spanner-backed services.
Fixed the issue causing request_count metric spikes due to a bug in the logic of our metrics reporting pipeline.
February 04, 2022
Creating a Dataproc Metastore service results in the error NO_MATCHING_ACCESS_LEVEL due to dns.googleapis.com in the service perimeter but not in the allowlist. To work around this issue, remove dns.googleapis.com from the perimeter during API calls.
Data Catalog sync users must request roles/metastore.metadataViewer to view synced Dataproc Metastore entries in Data Catalog. The roles/metastore.Admin and roles/metastore.Editor no longer support metastore databases and tables permissions.
The request_count metric spikes due to a bug in the logic of our metrics reporting pipeline.
January 26, 2022
Dataproc Metastore Auxiliary versions are available in Preview.
Spanner database type is available in Preview.
January 07, 2022
Dataproc Metastore is available in the following regions:
asia-northeast3 (Seoul)southamerica-east1 (São Paulo)
For more information, see Locations.
December 14, 2021
An Apache Log4j 2 vulnerability that impacted Dataproc clusters has been addressed (see Recreate and update a cluster, which provides guidance to Dataproc users). Dataproc Metastore users do not need to take any action; the fix applied by Dataproc Metastore is sufficient to address the issue.
November 29, 2021
Fixed the issue causing Dataproc Metastore service creations in a VPC-SC perimeter to fail due to a known issue that requires Google-managed service accounts to have access to Dataproc Metastore and Cloud Storage APIs.
November 24, 2021
Dataproc Metastore service creations in a VPC-SC perimeter may fail due to a known issue that requires Google-managed service accounts to have access to Dataproc Metastore and Cloud Storage APIs.
To work around this issue, or if you encounter a VPC-SC issue of type NO_MATCHING_ACCESS_LEVEL while creating your Dataproc Metastore service, create an access level for your service so it can access the APIs inside the service perimeter.
November 15, 2021
Private Service Connect with Dataproc Metastore is available in Preview.
October 07, 2021
Fixed the issue causing Dataproc Metastore service creations with CMEK enabled to fail if a service without CMEK enabled has never been created before in the project.
September 30, 2021
CMEK integration with Dataproc Metastore is generally available (GA).
September 16, 2021
For new projects, Dataproc Metastore service creations with CMEK enabled fail if a service without CMEK enabled has never been created before.
To work around this issue, create a service without CMEK enabled first.
September 06, 2021
The default Dataproc Metastore service creation version is changed to Hive 3.1.2.
August 30, 2021
Hive version 3.1.2 will become the default Dataproc Metastore service creation version in 1 week on September 6, 2021.
August 24, 2021
Fixed the issue causing metadata changes introduced through imports and backups to not be reflected in Data Catalog due to broken batch sync.
August 20, 2021
CMEK integration with Dataproc Metastore is available in Preview.
August 05, 2021
Hive version 3.1.2 will become the default Dataproc Metastore service creation version in 5 weeks on September 6, 2021.
July 30, 2021
Dataproc Metastore is available in the following regions:
asia-southeast1 (Singapore)europe-west1 (Belgium)northamerica-northeast1 (Montréal)
For more information, see Locations.
July 29, 2021
There is an issue with Dataproc Metastore to Data Catalog batch sync. Metadata changes introduced through imports and backups will not be reflected in Data Catalog until batch sync is restored.
July 22, 2021
July 09, 2021
Backing up and restoring service metadata are generally available (GA).
June 25, 2021
Dataproc Metastore performs a Hive metadata schema validation when importing metadata into a service.
- For SQL dump, it verifies the tables in the SQL dump file.
- For Avro import, it verifies the Avro file names.
- Both approaches ensure that all tables exist in the import source.
If the verification fails, the operation fails with INVALID_ARGUMENT code and an error message describing which table is missing.
The metadata import history is limited to 25. The oldest import is automatically deleted when the 26th import is created.
April 15, 2021
The asynchronous workflows logs have labels that appear in Cloud logging.
You no longer need to manually override metastore.expression.proxy to use PartitionProxyForMetastore in Hive 3.1.2.
March 31, 2021
Dataproc Metastore is generally available (GA).
Dataproc Metastore imports and exports support Avro storage format. This feature is available in Preview.
Dataproc Metastore supports backing up and restoring service metadata and configuration. This feature is available in Preview.
You can enable sync from a Dataproc Metastore service to Data Catalog. This feature is available in Preview.
Dataproc Metastore supports asynchronous background tasks through the Canary release channel.
March 03, 2021
Fixed an issue where specifying a Cloud Storage URI without an object would return an internal error.
Fixed an issue where the metastore.googleapis.com/service/health metric didn't show up for some services.
February 19, 2021
Hive configuration overrides are rejected if either the key or value contains a newline or "<" character.
Fixed an issue where services would fail to create in projects with project IDs that contain the colon "(:)" character.
Logs query builder doesn't work when selecting location and service ID.
The MetadataImport.DatabaseDump.source_database field is deprecated. It will be removed from the v1beta API channel no earlier than August 18, 2021.
February 16, 2021
You must have storage.objects.get permission on the Cloud Storage object in order to import metadata from the Cloud Storage file.
February 03, 2021
You can create Dataproc Metastore services in cross-product networks (shared VPC).
Dataproc Metastore suppports the use of non-RFC 1918 private IP address ranges in metastore services.
New Cloud Monitoring service metric is available:
metastore.googleapis.com/service/request_count
You can update the description of metadata imports under a Dataproc Metastore service.
Fixed an issue in which a service could get stuck in the UPDATING state.
Fixed an issue where Cloud Storage buckets with single character directories would fail request validation.
January 19, 2021
Fixed the Dataproc Metastore Cloud Logging and Monitoring issue.
January 08, 2021
Dataproc Metastore Cloud Logging and Monitoring is unavailable. The issue will be fixed shortly.
December 14, 2020
Dataproc Metastore is available in Preview.
Legacy Dataproc Metastore services created during private Preview (prior to December 14, 2020 at 12:00 PM Pacific Standard Time) will be automatically deleted on January 29, 2021.
The Thrift endpoints of legacy services will continue to function normally, but certain pre-existing functionality such as metadata imports will cease to work. Furthermore, new features (including those announced on December 14, 2020) and bugfixes will not be available to legacy services.
To ensure you receive the newest features, patches, and stability, we strongly recommend you recreate legacy Dataproc Metastore services. Since the new metadata export feature is not available for legacy services, if you need help migrating metadata from a legacy service, the Dataproc Metastore team will be happy to assist you with a manual migration.
Please contact dataproc-metastore-support@google.com with any questions or to request help migrating metadata.