Work with point-in-time recovery (PITR)

Work with point-in-time recovery (PITR)

This page describes how to use point-in-time recovery (PITR) to retain and recover data in Firestore.

To understand PITR concepts, see Point-in-time recovery.

Permissions

To get the permissions that you need to manage PITR settings, ask your administrator to grant you the Cloud Datastore Owner (roles/datastore.owner) IAM role on the project whose PITR settings you want to enable. For more information about granting roles, see Manage access to projects, folders, and organizations.

This predefined role contains the permissions required to manage PITR settings. To see the exact permissions that are required, expand the Required permissions section:

Required permissions

The following permissions are required to manage PITR settings:

• To enable PITR when creating a database: datastore.databases.create
• To update PITR settings on existing database: datastore.databases.update,datastore.databases.list
• To perform reads from PITR data: datastore.databases.get,datastore.entities.get,datastore.entities.list
• To export PITR data: datastore.databases.export
• To import PITR data: datastore.databases.import

You might also be able to get these permissions with custom roles or other predefined roles.

Before you begin

Note the following points before you start using PITR:

• You can't start reading from seven days in the past immediately after you enable PITR.
• If you want to enable PITR when you create a database, you must use the gcloud firestore databases create command. Enabling PITR while creating a database using the Google Cloud console is not supported.
• Firestore starts retaining versions from the point forward after enabling PITR.
• You cannot read PITR data in the PITR window after you disable PITR.
• If you re-enable PITR immediately after disabling it, the past PITR data is no longer available. Any PITR data created before disabling PITR will be deleted after the PITR expiration date.
• If you accidentally deleted data in the last hour and PITR is disabled, you can restore your data by enabling PITR within one hour of deletion.
• Any read performed on expired PITR data fails.

Enable PITR

Before using PITR, enable billing for your Google Cloud project. Only Google Cloud projects with billing enabled can use the PITR functionality.

To enable PITR for your database:

Console

1. In the Google Cloud console, go to the Databases page.

   Go to Databases

2. Select the required database from the list of databases.

3. In the navigation menu, click Disaster Recovery.

4. Click Edit to edit the settings.

5. Select the Enable point-in-time recovery check box, and then click Save.

Enabling PITR would incur storage costs. See Pricing for more information.

To disable PITR, clear the Enable point-in-time recovery check box from the Disaster Recovery page in the Google Cloud console.

gcloud

Enable PITR during database creation with the gcloud firestore databases create command as follows:

gcloud firestore databases create\
  --location=LOCATION\
  [--database=DATABASE_ID; default="(default)"]\
  [--type=TYPE; default="firestore-native"]\
  --enable-pitr

Replace the values as follows:

• LOCATION - location where you want to create your database.
• DATABASE_ID - set to the database ID or (default).
• TYPE - set to firestore-native.

You can disable PITR using the gcloud firestore databases update command as follows:

gcloud firestore databases update\
  [--database=DATABASE_ID; default="(default)"]\
  --no-enable-pitr

Replace the values as follows:

• DATABASE_ID - set to the database ID or (default).

Get the retention period and earliest version time

Console

1. In the Google Cloud console, go to the Databases page.

   Go to Databases

2. Select the required database from the list of databases.

3. In the navigation menu, click Disaster Recovery.

4. In the Settings section, note the Retention period and Earliest version time.

   • Retention period: the period in which Firestore retains all versions of data for the database. The value is one hour when PITR is disabled and seven days when PITR is enabled.
   • Earliest version time: the earliest timestamp at which older versions of the data can be read in the PITR window. This value is continuously updated by Firestore and becomes stale the moment it is queried. If you are using this value to recover data, make sure to account for the time from the moment w̦hen the value is queried to the moment when you initiate the recovery.
   • Point-in-time recovery: shows Enabled, if PITR is enabled. If PITR is disabled, you will see Disabled.

gcloud

Run the gcloud firestore databases describe command as follows:

gcloud firestore databases describe --database=DATABASE_ID

Replace DATABASE_ID with the database ID or '(default)'.

Here's the output:

    appEngineIntegrationMode: ENABLED
    concurrencyMode: PESSIMISTIC
    createTime: '2021-03-24T17:02:35.234Z'
    deleteProtectionState: DELETE_PROTECTION_DISABLED
    earliestVersionTime: '2023-06-12T16:17:25.222474Z'
    etag: IIDayqOevv8CMNTvyNK4uv8C
    keyPrefix: s
    locationId: nam5
    name: projects/PROJECT_ID/databases/DATABASE_ID
    pointInTimeRecoveryEnablement: POINT_IN_TIME_RECOVERY_DISABLED
    type: FIRESTORE_NATIVE
    uid: 5230c382-dcd2-468f-8cb3-2a1acfde2b32
    updateTime: '2021-11-17T17:48:22.171180Z'
    versionRetentionPeriod: 3600s

where,

• earliestVersionTime: timestamp of the earliest PITR data stored.
• pointInTimeRecoveryEnablement: shows POINT_IN_TIME_RECOVERY_ENABLED, if PITR is enabled. If PITR is disabled, you will either see POINT_IN_TIME_RECOVERY_DISABLED or the pointInTimeRecoveryEnablement field might not be displayed.
• versionRetentionPeriod: time period for which PITR data is retained in milliseconds. The value can be one hour when PITR is disabled or seven days if PITR is enabled.

Read PITR data

You can read PITR data using the client libraries, REST API methods, or FirestoreIO Apache Beam connector.

Client libraries

Note: You can read from PITR data in the server client libraries, namely Go, Java, Node.js, and PHP. The mobile and web SDKs, namely, Android, IOS, and Web are not supported.

Java

You must use the ReadOnly transaction to read PITR data. You cannot directly specify readTime in reads. See Transactions and batched writes for more information.

  Firestore firestore = …

  TransactionOptions options =
          TransactionOptions.createReadOnlyOptionsBuilder()
              .setReadTime(
                  com.google.protobuf.Timestamp.newBuilder()
                      .setSeconds(1684098540L)
                      .setNanos(0))
              .build();

  ApiFuture<Void> futureTransaction = firestore.runTransaction(
              transaction -> {
                // Does a snapshot read document lookup
                final DocumentSnapshot documentResult =
                    transaction.get(documentReference).get();

                // Executes a snapshot read query
                final QuerySnapshot queryResult =
                  transaction.get(query).get();
              },
              options);

  // Blocks on transaction to complete
  futureTransaction.get();

Node

You must use a ReadOnly transaction to read PITR data. You cannot directly specify readTime in reads. See Transactions and batched writes for more information.

  const documentSnapshot = await firestore.runTransaction(
    updateFunction => updateFunction.get(documentRef),
    {readOnly: true, readTime: new Firestore.Timestamp(1684098540, 0)}
);

  const querySnapshot = await firestore.runTransaction(
    updateFunction => updateFunction.get(query),
    {readOnly: true, readTime: new Firestore.Timestamp(1684098540, 0)}
  )

REST API

PITR reads are supported in all Firestore read methods, which are get, list, batchGet, listCollectionIds, listDocuments, runQuery, runAggregationQuery, and partitionQuery.

To perform a read using the REST methods, try one of the following options:

1. In the your read method request, pass the readTime value as as a supported PITR timestamp in the readOptions method. A PITR timestamp can be either microsecond precision timestamp within the past hour or a whole minute timestamp beyond the past hour, but not earlier than the earliestVersionTime.

2. Use the readTime parameter together with the BeginTransaction method as part of a ReadOnly transaction for multiple PITR reads.

Apache Beam

Use the FirestoreIO Apache Beam connector to read or write documents in a Firestore database at a large scale with Dataflow.

PITR reads are supported in the following read method of the FirestoreIO connector. These read methods support the withReadTime(@Nullable Instant readTime) method that you use can use for PITR reads:

• FirestoreV1.BatchGetDocuments
• FirestoreV1.ListCollectionIds
• FirestoreV1.ListDocuments
• FirestoreV1.PartitionQuery

• FirestoreV1.RunQuery

Java

The following code can be used with the example Dataflow pipeline code for bulk read or write operations. The example uses the withReadTime(@Nullable Instant readTime) method for PITR reads.

  Instant readTime = Instant.ofEpochSecond(1684098540L);

  PCollection<Document> documents =
      pipeline
          .apply(Create.of(collectionId))
          .apply(
              new FilterDocumentsQuery(
                  firestoreOptions.getProjectId(), firestoreOptions.getDatabaseId()))
          .apply(FirestoreIO.v1().read().runQuery().withReadTime(readTime).withRpcQosOptions(rpcQosOptions).build())
  ...

For a complete list of readTime examples in the Dataflow pipeline, see the Github repository.

Export and import from PITR data

You can export your database to Cloud Storage from PITR data using the gcloud firestore export command. You can export PITR data where the timestamp is a whole minute timestamp within the past seven days, but not earlier than the earliestVersionTime. If data no longer exists at the specified timestamp, the export operation fails.

The PITR export operation supports all filters, including export of all documents and export of specific collections.

1. Export the database, specifying the snapshot-time parameter to the desired recovery timestamp.

   gcloud

   Run the following command to export the database to your bucket.

   gcloud firestore export gs://BUCKET_NAME_PATH \
       --snapshot-time=PITR_TIMESTAMP \
       --collection-ids=COLLECTION_IDS \
       --namespace-ids=NAMESPACE_IDS
   

   Where,

   • BUCKET_NAME_PATH - a valid Cloud Storage bucket with an optional path prefix where export files are stored.
   • PITR_TIMESTAMP - a PITR timestamp at the minute granularity, for example, 2023-05-26T10:20:00.00Z or 2023-10-19T10:30:00.00-07:00.
   • COLLECTION_IDS - a list of collection IDs or collection group IDs, for example-'specific-collection-group1','specific-collection-group2'.
   • NAMESPACE_IDS - a list of namespace IDs, for example-'customer','orders'.

   Note the following points before exporting PITR data:

   • Specify the timestamp in RFC 3339 format. For example, 2023-05-26T10:20:00.00Z or 2023-10-19T10:30:00.00-07:00.
   • Make sure that the timestamp you specify is a whole minute timestamp within the past seven days, but not earlier than the earliestVersionTime. If data no longer exists at the specified timestamp, an error is generated. The timestamp must be a whole minute, even if the specified time is within the past hour.
   • You are not charged for a failed PITR export.

2. Import to a database.

   Use the steps in Import all documents to import your exported database. If any document already exists in your database, it will be overwritten.