Release notes

These release notes apply to the core Dataproc Metastore product. You can periodically check this page for announcements about new or updated features, bug fixes, known issues, and deprecated functionality.

See Dataproc Metastore version policy for a list of current and past supported software versions.

You can see the latest product updates for all of Google Cloud on the Google Cloud page, browse and filter all release notes in the Google Cloud Console, or you can programmatically access release notes in BigQuery.

To get the latest product updates delivered to you, add the URL of this page to your feed reader, or add the feed URL directly: https://cloud.google.com/feeds/metastore-release-notes.xml

September 16, 2021

v1beta1

For new projects, Dataproc Metastore service creations with CMEK enabled fail if a service without CMEK enabled has never been created before.

To work around this issue, create a service without CMEK enabled first.

September 06, 2021

v1

The default Dataproc Metastore service creation version is changed to Hive 3.1.2.

August 30, 2021

v1

Hive version 3.1.2 will become the default Dataproc Metastore service creation version in 1 week on September 6, 2021.

August 24, 2021

v1beta1

Fixed the issue causing metadata changes introduced through imports and backups to not be reflected in Data Catalog due to broken batch sync.

August 20, 2021

v1beta1

August 05, 2021

v1

Hive version 3.1.2 will become the default Dataproc Metastore service creation version in 5 weeks on September 6, 2021.

July 30, 2021

v1

Dataproc Metastore is available in the following regions:

  • asia-southeast1 (Singapore)
  • europe-west1 (Belgium)
  • northamerica-northeast1 (Montréal)

For more information, see Locations.

Dataproc Metastore backups and restores support cross-service restoration of metadata.

July 29, 2021

v1beta1

There is an issue with Dataproc Metastore to Data Catalog batch sync. Metadata changes introduced through imports and backups will not be reflected in Data Catalog until batch sync is restored.

July 22, 2021

v1

Avro based imports and exports are generally available (GA).

July 09, 2021

v1

Backing up and restoring service metadata are generally available (GA).

June 25, 2021

v1

Dataproc Metastore performs a Hive metadata schema validation when importing metadata into a service.

  • For SQL dump, it verifies the tables in the SQL dump file.
  • For Avro import, it verifies the Avro file names.
  • Both approaches ensure that all tables exist in the import source.

If the verification fails, the operation fails with INVALID_ARGUMENT code and an error message describing which table is missing.

The metadata import history is limited to 25. The oldest import is automatically deleted when the 26th import is created.

April 15, 2021

v1

The asynchronous workflows logs have labels that appear in Cloud logging.

You no longer need to manually override metastore.expression.proxy to use PartitionProxyForMetastore in Hive 3.1.2.

March 31, 2021

v1

Dataproc Metastore is generally available (GA).

Dataproc Metastore imports and exports support Avro storage format. This feature is available in Preview.

Dataproc Metastore supports backing up and restoring service metadata and configuration. This feature is available in Preview.

Dataproc Metastore supports asynchronous background tasks through the Canary release channel.

March 03, 2021

v1beta1

Fixed an issue where specifying a Cloud Storage URI without an object would return an internal error.

Fixed an issue where the metastore.googleapis.com/service/health metric didn't show up for some services.

February 19, 2021

v1beta1

Hive configuration overrides are rejected if either the key or value contains a newline or "<" character.

Fixed an issue where services would fail to create in projects with project IDs that contain the colon "(:)" character.

Logs query builder doesn't work when selecting location and service ID.

The MetadataImport.DatabaseDump.source_database field is deprecated. It will be removed from the v1beta API channel no earlier than August 18, 2021.

February 16, 2021

v1beta1

You must have storage.objects.get permission on the Cloud Storage object in order to import metadata from the Cloud Storage file.

February 03, 2021

v1beta1

You can create Dataproc Metastore services in cross-product networks (shared VPC).

Dataproc Metastore suppports the use of non-RFC 1918 private IP address ranges in metastore services.

New Cloud Monitoring service metric is available:

  • metastore.googleapis.com/service/request_count

You can update the description of metadata imports under a Dataproc Metastore service.

Fixed an issue in which a service could get stuck in the UPDATING state.

Fixed an issue where Cloud Storage buckets with single character directories would fail request validation.

January 19, 2021

v1beta1

Fixed the Dataproc Metastore Cloud Logging and Monitoring issue.

January 08, 2021

v1beta1

Dataproc Metastore Cloud Logging and Monitoring is unavailable. The issue will be fixed shortly.

December 14, 2020

v1beta1

Dataproc Metastore is available in Preview.

Legacy Dataproc Metastore services created during private Preview (prior to December 14, 2020 at 12:00 PM Pacific Standard Time) will be automatically deleted on January 29, 2021.

The Thrift endpoints of legacy services will continue to function normally, but certain pre-existing functionality such as metadata imports will cease to work. Furthermore, new features (including those announced on December 14, 2020) and bugfixes will not be available to legacy services.

To ensure you receive the newest features, patches, and stability, we strongly recommend you recreate legacy Dataproc Metastore services. Since the new metadata export feature is not available for legacy services, if you need help migrating metadata from a legacy service, the Dataproc Metastore team will be happy to assist you with a manual migration.

Please contact dataproc-metastore-support@google.com with any questions or to request help migrating metadata.