Back up metadata from Dataproc Metastore

Stay organized with collections Save and categorize content based on your preferences.

A backup takes a snapshot of both the metadata stored in the Dataproc Metastore service and the service configuration. You can recover it in a service by restoring.

This page explains how to create a backup of an existing Dataproc Metastore service.

Access control

  • To backup metadata, you must request an IAM role containing the metastore.backups.create IAM permission. The Dataproc Metastore specific roles roles/metastore.admin, roles/metastore.editor, and roles/metastore.metadataOperator include backup permission.

  • You can give backup permission to users or groups by using the roles/owner and roles/editor legacy roles.

For more information, see Dataproc Metastore IAM and access control.

Back up metadata from a service

To backup metadata from a service, start a backup on the Service detail page opened in a local browser, use the gcloud CLI, or issue a Dataproc Metastore API method backups.create.

You can create and store up to 7 backups at a time for each Dataproc Metastore service. The backups are fully managed by Dataproc Metastore.

While a backup is running, no updates can be made to the service. You can still use the service while it's undergoing a backup.

To backup metastore from a Dataproc Metastore service, complete the following steps:

Console

  1. In the Google Cloud console, open the Dataproc Metastore page:

    Open Dataproc Metastore in the Google Cloud console

  2. On the Dataproc Metastore page, click the name of the service you'd like to backup metadata from. The Service detail page opens.

    Service detail page

  3. At the top of the page, click Backup. The Backup page opens.

  4. Enter the Backup name.

  5. Optional: Enter a Description of the backup.

  6. Click Backup to start the backup.

  7. Verify that you have returned to the Service detail page, and that your backup appears under Backups on the Backup/Restore tab.

gcloud

  1. Run the following gcloud metastore services backups create command to backup metadata from a service:

    gcloud metastore services backups create BACKUP  \
    --location=LOCATION \
    --service=SERVICE \
    --description=DESCRIPTION \

    Replace the following:

    • BACKUP: The ID of the backup or fully qualified identifier for the backup.
    • LOCATION: The location of the service.
    • SERVICE: The name of the service.
    • DESCRIPTION: The description of the backup.

  2. Verify that the backup creation was successful.

REST

Follow the API instructions to backup metadata from a service by using the API Explorer.

When the backup completes, the service automatically enters active state regardless of whether or not it succeeded.

To view a service's backups, refer to the Backup/Restore tab on the Service detail page in the Google Cloud console.

Backup caveats

  • You can see a list of backups on the UI on the Backup page from the Service details page. Deleting the service itself deletes all backups under that service.

  • If there are already 7 backups in a service, then you must first manually delete a backup before creating a new one. You can delete existing backups from the Backup/Restore tab.

Common failures

  • The user creating the backup doesn't have metastore.backups.create permission.

What's next