Automatic Upgrade to Firestore

This page describes the upgrade path from Cloud Datastore to Firestore in Datastore mode.

Firestore can operate in Datastore mode, making it backwards-compatible with Cloud Datastore. With Firestore in Datastore mode, you can access Firestore's improved storage layer while keeping Datastore system behavior. Firestore in Datastore mode removes the following Cloud Datastore limitations:

  • Queries are no longer eventually consistent; instead, they are all strongly consistent.
  • Queries in transactions are no longer required to be ancestor queries.
  • Transactions are no longer limited to 25 entity groups.
  • Writes to an entity group are no longer limited to 1 per second.
  • Transactions now use pessimistic locks instead of optimistic concurrency

For more about Datastore mode, see Firestore in Datastore mode.

Starting in 2021, Google Cloud will gradually upgrade existing Cloud Datastore databases to Firestore in Datastore mode.

Automatic upgrade to Firestore in Datastore mode

If you manage an application that uses Cloud Datastore, you will not need to update your application code. We will notify you about the schedule of your application's upgrade to Firestore in Datastore mode. The upgrade does not require downtime.

For additional questions about the automatic upgrade process, contact one of our support channels.

Upgrade stages

At a high level, we follow this process to upgrade your Cloud Datastore database to Firestore in Datastore mode. This process does not require application downtime:

  1. Add a new Firestore in Datastore mode data replica to your existing Cloud Datastore database. Asynchronously duplicate entity write operations to Firestore in Datastore mode.

  2. Copy existing data and index entries from Cloud Datastore to Firestore in Datastore mode. After copying, verify the data.

  3. Redirect entity reads directly to Firestore in Datastore mode. First redirect eventually consistent reads, and then redirect strongly consistent reads.

  4. Redirect entity writes and transactional reads directly to Firestore in Datastore mode.

This process uses the following stages.

1. Copy and verify

This stage introduces a Firestore in Datastore mode replica and performs the following steps:

  1. Journal

    Entity write operations to Cloud Datastore also begin to flow through a side-channel to the Firestore in Datastore mode replica. This happens as part of Cloud Datastore's existing replication system. These write operations do not affect write latency. The Firestore in Datastore mode replica buffers these write operations to apply them after the copy step.

  2. Copy

    In the Firestore in Datastore mode replica, create an offline copy of your existing data and index entries. The copy step does not impact Cloud Datastore operations. This step may last several days.

  3. Drain Journal

    Apply the writes from the journal step on top of the data from the offline copy.

  4. Verify Data

    Re-verify the data in Firestore in Datastore mode by comparing against the data in Cloud Datastore.

2. Redirect eventually consistent reads

Serve eventually consistent reads (queries with no ancestor filter) from Firestore in Datastore mode. Cloud Datastore semantics for reads still apply at this point:

  • Ancestor queries are strongly consistent.
  • Non-ancestor queries are eventually consistent.
  • Lookups are strongly consistent (except those explicitly configured for eventual consistency).

Firestore in Datastore mode continues to act as a replica of your Cloud Datastore data.

3. Redirect strongly consistent reads

Serve strongly consistent reads (non-transactional) from Firestore in Datastore mode. Note that Cloud Datastore semantics for reads still apply. Even though reads now come directly from Firestore, Firestore still relies on Cloud Datastore to guarantee that it's up to date for strongly consistent reads.

4. Redirect writes

Redirect entity writes and transactional reads to Firestore in Datastore mode. Concurrent modifications to the same entity continue to cause transaction aborts. Concurrent modifications to different entities within the same entity group no longer result in transaction aborts.

At the start of this stage, Firestore in Datastore mode still relies on Cloud Datastore to guarantee it's up to date before each write. After a final pass that ensures all previous writes are applied, Firestore in Datastore mode stops consulting Cloud Datastore.

5. Migration complete

Firestore in Datastore mode semantics for reads now apply: all queries are strongly consistent.

Pricing remains the same, but your billing now lists Firestore SKUs. The App Engine Quotas page starts to show Firestore usage instead of Cloud Datastore usage.

Logging and progress notifications

The upgrade process uses Cloud Logging to publish progress updates. To view the logs use the Logs Explorer, the Cloud Logging API, or the Cloud SDK.

Updates are published to two logs under the datastore.googleapis.com logging service name:

Log name Monitored resource Payload
migration_state datastore_database type.googleapis.com/google.datastore.admin.v1.MigrationStateEvent
migration_progress datastore_database type.googleapis.com/google.datastore.admin.v1.MigrationProgressEvent

The migration_state log is updated when the overall state of the upgrade changes (RUNNING and COMPLETE).

The migration_progress log is updated each time the upgrade moves into a new stage (START, COPY_AND_VERIFY, REDIRECT_EVENTUALLY_CONSISTENT_READS, REDIRECT_STRONGLY_CONSISTENT_READS, and REDIRECT_WRITES).

To receive notifications as your upgrade proceeds, you can create logs-based metrics based on the two logs and create alerts based on those metrics.

Testing an existing application

For an existing app that uses Cloud Datastore, you can test app behavior with Firestore in Datastore mode by doing the following:

  1. Create a new project. In this project, create a Firestore in Datastore mode database.
  2. Using the managed export service, export some of your application's data to Cloud Storage.
  3. Using the managed import service, import your application's data to your new project.
  4. Copy app logic you want to test to the new project or simulate app behaviour against the new project.

Cloud Monitoring metrics

The Cloud Monitoring metrics available for your Datastore database remain the same throughout the upgrade process, see available Datastore metrics.