Release notes

These release notes apply to the core Dataproc Metastore product. You can periodically check this page for announcements about new or updated features, bug fixes, known issues, and deprecated functionality.

See Dataproc Metastore version policy for a list of current and past supported software versions.

You can see the latest product updates for all of Google Cloud on the Google Cloud page, browse and filter all release notes in the Google Cloud console, or you can programmatically access release notes in BigQuery.

To get the latest product updates delivered to you, add the URL of this page to your feed reader, or add the feed URL directly: https://cloud.google.com/feeds/metastore-release-notes.xml

March 01, 2022

v1beta1

gRPC endpoint protocol is available in Preview.

February 28, 2022

v1beta1

Fixed the issue causing metadata batch sync from Dataproc Metastore to Data Catalog to not work.

February 19, 2022

v1

Fixed the issue causing Dataproc Metastore service creation to fail with the error NO_MATCHING_ACCESS_LEVEL due to a known issue where dns.googleapis.com is in the service perimeter but not in the allowlist.

February 11, 2022

v1beta1

Performing import, export, backup, or restore on Spanner-backed services now returns a 4XX error since these operations aren't supported.

Added additional mutual exclusion validation for Data Catalog and Spanner-backed services.

v1

Fixed the issue causing request_count metric spikes due to a bug in the logic of our metrics reporting pipeline.

February 04, 2022

v1beta1 & v1

Creating a Dataproc Metastore service results in the error NO_MATCHING_ACCESS_LEVEL due to dns.googleapis.com in the service perimeter but not in the allowlist. To work around this issue, remove dns.googleapis.com from the perimeter during API calls.

Data Catalog sync users must request roles/metastore.metadataViewer to view synced Dataproc Metastore entries in Data Catalog. The roles/metastore.Admin and roles/metastore.Editor no longer support metastore databases and tables permissions.

The request_count metric spikes due to a bug in the logic of our metrics reporting pipeline.

January 26, 2022

v1beta1

Dataproc Metastore Auxiliary versions are available in Preview.

Spanner database type is available in Preview.

January 07, 2022

v1

Dataproc Metastore is available in the following regions:

  • asia-northeast3 (Seoul)
  • southamerica-east1 (São Paulo)

For more information, see Locations.

December 14, 2021

v1beta1 & v1

An Apache Log4j 2 vulnerability that impacted Dataproc clusters has been addressed (see Recreate and update a cluster, which provides guidance to Dataproc users). Dataproc Metastore users do not need to take any action; the fix applied by Dataproc Metastore is sufficient to address the issue.

November 29, 2021

v1

Fixed the issue causing Dataproc Metastore service creations in a VPC-SC perimeter to fail due to a known issue that requires Google-managed service accounts to have access to Dataproc Metastore and Cloud Storage APIs.

November 24, 2021

v1

Dataproc Metastore service creations in a VPC-SC perimeter may fail due to a known issue that requires Google-managed service accounts to have access to Dataproc Metastore and Cloud Storage APIs.

To work around this issue, or if you encounter a VPC-SC issue of type NO_MATCHING_ACCESS_LEVEL while creating your Dataproc Metastore service, create an access level for your service so it can access the APIs inside the service perimeter.

November 15, 2021

v1beta1

October 07, 2021

v1beta1 & v1

Fixed the issue causing Dataproc Metastore service creations with CMEK enabled to fail if a service without CMEK enabled has never been created before in the project.

September 30, 2021

v1

CMEK integration with Dataproc Metastore is generally available (GA).

September 16, 2021

v1beta1

For new projects, Dataproc Metastore service creations with CMEK enabled fail if a service without CMEK enabled has never been created before.

To work around this issue, create a service without CMEK enabled first.

September 06, 2021

v1

The default Dataproc Metastore service creation version is changed to Hive 3.1.2.

August 30, 2021

v1

Hive version 3.1.2 will become the default Dataproc Metastore service creation version in 1 week on September 6, 2021.

August 24, 2021

v1beta1

Fixed the issue causing metadata changes introduced through imports and backups to not be reflected in Data Catalog due to broken batch sync.

August 20, 2021

v1beta1

August 05, 2021

v1

Hive version 3.1.2 will become the default Dataproc Metastore service creation version in 5 weeks on September 6, 2021.

July 30, 2021

v1

Dataproc Metastore is available in the following regions:

  • asia-southeast1 (Singapore)
  • europe-west1 (Belgium)
  • northamerica-northeast1 (Montréal)

For more information, see Locations.

Dataproc Metastore backups and restores support cross-service restoration of metadata.

July 29, 2021

v1beta1

There is an issue with Dataproc Metastore to Data Catalog batch sync. Metadata changes introduced through imports and backups will not be reflected in Data Catalog until batch sync is restored.

July 22, 2021

v1

Avro based imports and exports are generally available (GA).

July 09, 2021

v1

Backing up and restoring service metadata are generally available (GA).

June 25, 2021

v1

Dataproc Metastore performs a Hive metadata schema validation when importing metadata into a service.

  • For SQL dump, it verifies the tables in the SQL dump file.
  • For Avro import, it verifies the Avro file names.
  • Both approaches ensure that all tables exist in the import source.

If the verification fails, the operation fails with INVALID_ARGUMENT code and an error message describing which table is missing.

The metadata import history is limited to 25. The oldest import is automatically deleted when the 26th import is created.

April 15, 2021

v1

The asynchronous workflows logs have labels that appear in Cloud logging.

You no longer need to manually override metastore.expression.proxy to use PartitionProxyForMetastore in Hive 3.1.2.

March 31, 2021

v1

Dataproc Metastore is generally available (GA).

Dataproc Metastore imports and exports support Avro storage format. This feature is available in Preview.

Dataproc Metastore supports backing up and restoring service metadata and configuration. This feature is available in Preview.

Dataproc Metastore supports asynchronous background tasks through the Canary release channel.

March 03, 2021

v1beta1

Fixed an issue where specifying a Cloud Storage URI without an object would return an internal error.

Fixed an issue where the metastore.googleapis.com/service/health metric didn't show up for some services.

February 19, 2021

v1beta1

Hive configuration overrides are rejected if either the key or value contains a newline or "<" character.

Fixed an issue where services would fail to create in projects with project IDs that contain the colon "(:)" character.

Logs query builder doesn't work when selecting location and service ID.

The MetadataImport.DatabaseDump.source_database field is deprecated. It will be removed from the v1beta API channel no earlier than August 18, 2021.

February 16, 2021

v1beta1

You must have storage.objects.get permission on the Cloud Storage object in order to import metadata from the Cloud Storage file.

February 03, 2021

v1beta1

You can create Dataproc Metastore services in cross-product networks (shared VPC).

Dataproc Metastore suppports the use of non-RFC 1918 private IP address ranges in metastore services.

New Cloud Monitoring service metric is available:

  • metastore.googleapis.com/service/request_count

You can update the description of metadata imports under a Dataproc Metastore service.

Fixed an issue in which a service could get stuck in the UPDATING state.

Fixed an issue where Cloud Storage buckets with single character directories would fail request validation.

January 19, 2021

v1beta1

Fixed the Dataproc Metastore Cloud Logging and Monitoring issue.

January 08, 2021

v1beta1

Dataproc Metastore Cloud Logging and Monitoring is unavailable. The issue will be fixed shortly.

December 14, 2020

v1beta1

Dataproc Metastore is available in Preview.

Legacy Dataproc Metastore services created during private Preview (prior to December 14, 2020 at 12:00 PM Pacific Standard Time) will be automatically deleted on January 29, 2021.

The Thrift endpoints of legacy services will continue to function normally, but certain pre-existing functionality such as metadata imports will cease to work. Furthermore, new features (including those announced on December 14, 2020) and bugfixes will not be available to legacy services.

To ensure you receive the newest features, patches, and stability, we strongly recommend you recreate legacy Dataproc Metastore services. Since the new metadata export feature is not available for legacy services, if you need help migrating metadata from a legacy service, the Dataproc Metastore team will be happy to assist you with a manual migration.

Please contact dataproc-metastore-support@google.com with any questions or to request help migrating metadata.