Manage backups

This page describes Bigtable backup operations and explains how to back up a table and restore from it to a new table. Before you read this page, you should be familiar with Backups.

You can work with Bigtable backups by using the following:

  • The Google Cloud console.
  • The Google Cloud CLI.
  • The Cloud Bigtable client libraries.

You can also access the API directly, but we strongly recommend that you do so only if you need to use a language that is not supported by the Cloud Bigtable client libraries.

Before you begin

Make sure that you or the user account that you are using have the required permissions for the action you need to perform.

If you plan to use the gcloud CLI, install the gcloud CLI for Bigtable.

Use automated backup

You can use the gcloud CLI to enable automated backup when you create or update a table. The steps to delete, update, copy, and restore a backup are the same whether the backup is created manually or as a result of automated backup.

During the preview, consider the following if you plan to enable automated backup:

  • If a cluster is added to an instance that contains a table with automated backup enabled, daily backups aren't created on the added cluster. To create daily backups on all clusters, including the newly added cluster, disable automated backup for the table, and then re-enable automated backup.
  • If a table with automated backup enabled is deleted and subsequently undeleted, you need to re-enable automated backup on the recovered table.
  • Restoring an automated backup of a table doesn't automatically enable automated backup on the restored table. You need to manually enable automated backup.

Enable automated backup

To enable automated backup for a table, run the gcloud bigtable instances tables update command. For more information about enabling automated backup when creating a table, see Create a table.

After the automated backup policy is enabled for a table, the updated configuration can take up to 1 hour to take effect.

gcloud bigtable instances tables update TABLE_ID \
--instance=INSTANCE_ID \
--enable-automated-backup

Replace the following:

  • TABLE_ID: ID of the table to be backed up.
  • INSTANCE_ID: The permanent identifier for the instance.
  1. If you don't know the instance ID, use the bigtable instances list command to view a list of your project's instances:

      gcloud bigtable instances list
    
  2. View the list of tables to confirm the ID of the table you want to back up.

        gcloud bigtable instances tables list --instances=INSTANCE_ID
    

Backups that are created as part of automated backup are assigned IDs that begin with auto. For example, a table with an ID of my-table would have an automated backup ID like auto.my-table.c7x3.20230220-1455, where my-table is the truncated table ID, c7x3 is the unique hash ID, and 20230220-1455 is the date and time in UTC format.

Disable automated backup

To disable automated backup for an existing table, run the gcloud bigtable instances tables update command.

After the automated backup policy is disabled for a table, the updated configuration can take up to 1 hour to take effect. It might take up to 24 hours for your first backup to be available.

gcloud bigtable instances tables update TABLE_ID \
--instance=INSTANCE_ID \
--disable-automated-backup

Replace the following:

  • TABLE_ID: ID of the table.
  • INSTANCE_ID: The permanent identifier for the instance.

View automated backup policy

To view the automated backup policy for a table, run the gcloud bigtable instances tables describe command.

gcloud bigtable instances tables describe TABLE_ID \
--instance=INSTANCE_ID

Replace the following:

  • TABLE_ID: ID of the table.
  • INSTANCE_ID: The permanent identifier for the instance.

The output looks similar to the following:

automatedBackupPolicy:
  retentionPeriod: 3d
  frequency: 24h
columnFamilies:
  my-family: {}
createTime: '2023-02-07T20:10:55.613546Z'
granularity: MILLIS
name: projects/my-project/instances/my-instance/tables/my-table
updateTime: '2023-02-07T20:10:55.613546Z'

Note the following:

  • The automatedBackupPolicy field is absent when automated backup is not enabled for a table.
  • The retentionPeriod indicates the retention period of automatically created backups. You can modify the retention period for a backup to up to 90 days from backup creation time. To update the backup retention period, see Modify a backup.

Create a backup

Console

  1. Go to the Bigtable Instances page in the Google Cloud console.

    Open the instance list

  2. Click the instance that contains the table you want to back up.

  3. In the left navigation pane, click Tables.

  4. Click Create backup for the table you want to back up.

  5. If you are using replication, use the drop-down menu to choose the Cluster ID for the cluster that should store the backup. (If you clicked Create backup next to a cluster ID rather than an instance ID on the Tables page, the cluster is pre-selected.)

  6. Enter a unique ID for the backup.

  7. Set an expiration date.

  8. Click Create.

    The console displays the Backups page filtered to show the backup and its details.

  9. Click Activity to view the status of the backup.

    The status column displays Backup complete when the backup has been completed.

gcloud

For all commands, substitute actual values for the following:

  • INSTANCE_ID: The permanent identifier for the instance.
  • CLUSTER_ID: The permanent identifier for the cluster.
  • TABLE_ID: The ID of the table to be backed up.
  • BACKUP_ID: The ID you assign to the backup.
  • EXPIRATION_DATE: A date that is 90 days or less in the future, formatted as a "Zulu" UTC timestamp, accurate to nanoseconds.

    Example: 2022-10-02T15:01:23.045123456Z

  • RETENTION_PERIOD: A period of up to 90 days, expressed as a number with a unit of m, h, or d (minutes, hours, or days).

    Examples: 36h or 89d

  1. If you don't know the instance ID, use the bigtable instances list command to view a list of your project's instances:

    gcloud bigtable instances list
    
  2. View the list of backups for the instance that contains the table you want to back up, so that you can choose a backup ID that is not already in use on the intended cluster.

    gcloud bigtable backups list --instance=INSTANCE_ID \
      --cluster=CLUSTER_ID
    
  3. View the list of tables to confirm the ID of the table you want to back up.

    gcloud bigtable instances tables list --instances=INSTANCE_ID
    
  4. Run the gcloud bigtable instances backups create command to create a backup from the table, providing either an expiration date or a retention period of up to 90 days.

    gcloud bigtable backups create BACKUP_ID --instance=INSTANCE_ID \
      --cluster=CLUSTER_ID \
      --table=TABLE_ID \
      --async /
      --expiration-date=EXPIRATION_DATE \
      --retention-period=RETENTION_PERIOD
    

    The --async flag is optional. Use it if you want the terminal to return the operation ID before the operation is complete. This is helpful if you want to check the status of the operation.

  5. View the list of backups to see if your backup has been created.

    gcloud bigtable backups list --instance=INSTANCE_ID \
      --cluster=CLUSTER_ID
    
  6. If the backup is not listed, check the status of the operation.

    A status of Ready indicates the backup is complete.

Restore from a backup

Console

You can't restore to a different project using the Google Cloud console. If you need to do so, use the gcloud CLI instead.

  1. Go to the Bigtable Instances page in the Google Cloud console.

    Open the instance list

  2. Click the instance that contains the backup you want to restore.

  3. In the left navigation pane, click Backups.

  4. Click Restore for the backup that you want to restore.

  5. Select the instance that you want to restore to.

    Instances that don't have enough storage for the new table are unavailable. If you don't have permission to create a table in an instance, the instance is unavailable. Mouse over the icon for more information.

    If you restore from a backup that is protected by CMEK, the destination instance must be CMEK-protected as well.

    You are not able to restore to a different project using the Google Cloud console. If you need to, use the gcloud CLI instead.

  6. Enter a unique ID for the table that will be created from the backup. You are not able to change this ID later.

  7. Click Restore.

    The console displays the Tables page filtered to show the new table.

    The console displays the restore status for each cluster. When the status column for all clusters shows Ready the table has been restored and replicated to all clusters in the instance.

gcloud

For all commands, substitute actual values for the following:

  • PROJECT_ID_SOURCE: The ID of the project containing the backup to be restored from.
  • INSTANCE_ID_SOURCE: The permanent ID for the source instance.
  • PROJECT_ID_DESTINATION: The ID of the project where you want to restore to. This flag is optional. If you don't specify this option, the backup is restored to a new table in the same instance where it was created.
  • INSTANCE_ID_DESTINATION: The permanent ID for the destination instance.
  • CLUSTER_ID: The permanent identifier for the cluster.
  • BACKUP_ID: The ID of the backup you want to restore from.
  • TABLE_ID_NEW: The ID for the new table.
  1. If you don't know the instance ID, use the bigtable instances list command to view a list of your project's instances:

    gcloud bigtable instances list
    
  2. View the list of backups for the instance that contains the backup, to verify the backup exists.

    gcloud bigtable backups list --instance=INSTANCE_ID_SOURCE \
      --cluster=CLUSTER_ID_SOURCE
    
  3. View the list of tables in the destination instance, so you can choose an ID for the new table that is not already in use.

    gcloud bigtable instances tables list --instances=INSTANCE_ID_DESTINATION
    
  4. Run the gcloud bigtable instances tables restore command to restore from the backup to a new table.

    gcloud bigtable instances tables restore \
    --source=projects/PROJECT_ID_SOURCE/instances/INSTANCE_ID_SOURCE \
    clusters/CLUSTER_ID/backups/BACKUP_ID \
    --async \
    --destination=TABLE_ID_NEW \
    --destination-instance=INSTANCE_ID_DESTINATION \
    --project=PROJECT_ID_DESTINATION
    

    The --async flag is optional. Use it if you want the terminal to return the operation ID before the operation is complete. This is helpful if you want to check the status of the operation.

  5. View the list of tables to see if your table has been created.

    gcloud bigtable instances tables list --instances=INSTANCE_ID_DESTINATION
    
  6. If the backup is not listed, check the status of the operation.

Create a copy of a backup

Console

You are not able to create a copy of a backup in a different project using the console. If you need to do so, use the gcloud CLI instead.

  1. Go to the Bigtable Instances page in the Google Cloud console.

    Open the instance list

  2. Click the instance that contains the backup you want to copy.

  3. In the left navigation pane, click Backups.

  4. For the backup that you want to copy, expand the More menu next to the word Restore, and then click Copy.

  5. Click Copy for the backup that you want to make a copy of.

    If the backup is a copy of another backup, then Copy is not available.

  6. Select the destination instance.

    You are not able to create a copy of a backup in a different project using the console. If you need to do so, use the gcloud CLI instead.

    Instances that don't have enough storage for the backup copy are unavailable. If you don't have permission to create a backup in an instance, the instance is unavailable. Hold the pointer over the icon for more information.

    If you copy a backup that is protected by CMEK, the destination instance must be CMEK-protected as well.

  7. Select the destination cluster.

    Like any backup, a backup copy is stored on only one cluster in an instance, even if that instance uses replication.

  8. Enter a unique ID for the copy that will be created from the backup. You are not able to change this ID later.

  9. Set an expiration time for the backup copy, providing a value for Time and selecting a Unit from the drop-down menu.

  10. Click Copy.

  11. To view your copy, click View copy.

    The console displays the Backups page for the destination instance, filtered for the backup copy just created. When the status column shows Ready the copy is complete and ready for further operations.

gcloud

For all commands, substitute actual values for the following:

  • PROJECT_ID_SOURCE: The project containing the backup to be copied.
  • INSTANCE_ID_SOURCE: The permanent ID for the source instance.
  • CLUSTER_ID_SOURCE: The ID of cluster where the source backup is stored.
  • BACKUP_ID_SOURCE: The ID of the backup to copy.
  • PROJECT_ID_DESTINATION: The project where you want to put the copy.
  • INSTANCE_ID_DESTINATION: The permanent ID for the destination instance.
  • CLUSTER_ID_DESTINATION: The ID of the cluster in the destination instance that you want to store the copy.
  • BACKUP_ID_DESTINATION: The ID you assign to the copy of the backup.
  • RETENTION_PERIOD: The time to live for the backup copy, expressed as a number and a unit (d or h). Examples include 5d for five days or 15h for 15 hours. Use either this or an expiration date.
  • EXPIRATION_DATE: The date and time to let the backup copy expire, formatted like 2022-09-01T10:00:00.0Z.
  1. If you don't know the instance ID, use the bigtable instances list command to view a list of your project's instances:

    gcloud bigtable instances list
    
  2. View the list of backups for the source instance that contains the backup, to verify that the backup that you want to copy exists.

    gcloud bigtable backups list --instance=INSTANCE_ID_SOURCE \
    --cluster=CLUSTER_ID_SOURCE
    
  3. View the list of backups in the destination instance, so you can choose an ID for the copy that is not already in use.

    gcloud bigtable backups list --instances=DESTINATION_INSTANCE
    
  4. Run the gcloud bigtable instances backups copy command to create the backup copy.

    gcloud bigtable backups copy \
      --source-project=PROJECT_ID_SOURCE \
      --source-instance=INSTANCE_ID_SOURCE \
      --source-cluster=CLUSTER_ID_SOURCE \
      --source-backup=BACKUP_ID_SOURCE \
      --destination-project=PROJECT_ID_DESTINATION \
      --destination-instance=INSTANCE_ID_DESTINATION \
      --destination-cluster=CLUSTER_ID_DESTINATION \
      --destination-backup=BACKUP_ID_DESTINATION \
    

    Append the command with one of the following:

    • --retention-period`=RETENTION_PERIOD: Retention period for the backup copy. Must be at least 6 hours and at most 30 days from the time the source backup was created.
    • --expiration-date`=EXPIRATION_DATE: Expiration for the backup copy. Must be at least 6 hours and at most 30 days from the time the source backup was created.

    You can optionally add --async to the command. Use it if you want the terminal to return the operation ID before the operation is complete. This is helpful if you want to check the status of the operation.

  5. View the list of backups to see if your copy has been created.

    gcloud bigtable backups list --instances=INSTANCE_ID_DESTINATION
    
  6. If the backup is not listed, check the status of the operation.

In the rare event that a backup copy operation fails because of capacity limitations, try again with a different destination cluster. If that operation also fails, contact Support.

Check the status of an operation

Console

  1. Go to the Bigtable Instances page in the Google Cloud console.

    Open the instance list

  2. Click the instance that contains the backup.

  3. In the left navigation pane, click Backups.

  4. On the Backups page, click Activity. The status column displays the current status for each operation.

Operation statuses when restoring a table include the following:

  • Waiting to copy
  • Restoring backup
  • Initializing
  • Ready-optimizing
  • Ready

Operation statuses when creating a backup include the following:

  • Creating backup
  • Backup complete

Operation statuses when copying a backup include the following:

  • Copying backup
  • Copied backup

gcloud

To check the status of a backup or restore operation, you need the unique ID of the operation. The following is an example of an operation ID:

`instances/instance-name/tables/table-name/locations/us-east1-b/operations/917168358249360635`

After you run a command to create a backup or restore to a new table using the option --async, the terminal displays the operation ID.

  1. Copy the operation ID.

  2. Run the gcloud bigtable operations describe command, substituting the value you copied for OPERATION_ID.

    gcloud bigtable operations describe OPERATION_ID
    

    Example:

    gcloud bigtable operations describe \
    instances/instance-name/tables/table-name/locations/us-east1-b/operations/917168358249360635
    

    You can execute this command every few moments until the value for DONE is TRUE.

If you don't know the operation ID, either because you didn't just execute the command or you are in a different terminal window, you can view a list of operations for the instance and determine which operation you want to check.

  1. View a list of recent operation activity for the instance that contains the table you are backing up or restoring to, substituting the ID of the instance for INSTANCE_ID:

    gcloud bigtable operations list --instance=INSTANCE_ID
    

    The terminal displays a list of operations in columns labeled NAME, START_TIME, END_TIME, and DONE. The value in the NAME column is the operation ID.

  2. Beginning with the first item on the list, copy the operation ID.

  3. Use gcloud bigtable operations describe to view metadata for the operation.

    gcloud bigtable operations describe OPERATION_ID
    

    Example:

    gcloud bigtable operations describe \
    instances/instance-name/tables/table-name/locations/us-east1-b/operations/917168358249360635
    
  4. If the metadata for an operation shows a value for backupInfo, you have found the correct operation, and you can repeat this command every so often until the value for DONE is TRUE.

    If the metadata does not show a value for backupInfo, check the next operation from the list of operations displayed in Step 1.

Modify a backup or backup copy

Console

The Google Cloud console does not let you modify a backup, including copies.

gcloud

The gcloud CLI lets you modify the expiration date or period for a backup or backup copy.

For all commands, substitute actual values for the following:

  • INSTANCE_ID: The permanent identifier for the instance.
  • CLUSTER_ID: The permanent identifier for the cluster.
  • BACKUP_ID: The unique ID for a backup.
  • EXPIRATION_DATE: A date that is 90 days or less in the future, formatted as a "Zulu" UTC timestamp, accurate to nanoseconds.

    Example: 2019-10-02T15:01:23.045123456Z

  • RETENTION_PERIOD: A period of up to 90 days, expressed as a number with a unit of m, h, or d (minutes, hours, or days).

    _Examples_: `36h` or `89d`
    
  1. View a list of backups:

        gcloud bigtable backups list --instance=INSTANCE_ID
    

    The terminal displays a list of backups.

  2. Note the backup and cluster IDs for the backup you need to update.

  3. Run the gcloud bigtable instances backups update command to either add a new expiration date or a new retention period.

      gcloud bigtable backups update BACKUP_ID \
      --instance=INSTANCE_ID \
      --cluster=CLUSTER_ID \
      --expiration-date=EXPIRATION_DATE \
      --retention-period=RETENTION_PERIOD
    
  4. Run the gcloud bigtable instances backups describe command to verify that you have successfully updated the expiration.

        gcloud bigtable backups describe BACKUP_ID \
        --instance=INSTANCE_ID \
        --cluster=CLUSTER_ID
    ```
    

Delete a backup

You can delete a backup or backup copy before its scheduled expiration.

Console

  1. Go to the Bigtable Instances page in the Google Cloud console.

    Open the instance list

  2. Click the instance that contains the backup.

  3. In the left navigation pane, click Backups.

  4. For the backup you need to delete, expand the More menu next to the word Restore, then click Delete.

  5. Enter the backup ID in the Confirm deletion field, then click Delete.

gcloud

  1. View a list of backups, substituting the ID of the instance for INSTANCE_ID:

    gcloud bigtable backups list --instance=INSTANCE_ID
    

    The terminal displays a list of backups.

  2. Note the backup ID and the ID of the cluster it is stored on.

  3. Run the gcloud bigtable instances backups delete command, substituting the backup ID you noted for BACKUP_ID and the cluster ID for CLUSTER_ID.

    gcloud bigtable backups delete BACKUP_ID \
      --instance=INSTANCE_ID \
      --cluster=CLUSTER_ID
    

What's next