Automatic Upgrade to Firestore

This page describes the upgrade path from Cloud Datastore to Firestore in Datastore mode.

Firestore can operate in Datastore mode, making it backwards-compatible with Cloud Datastore. With Firestore in Datastore mode, you can access Firestore's improved storage layer while keeping Datastore system behavior. Firestore in Datastore mode removes the following Cloud Datastore limitations:

  • Queries are no longer eventually consistent. Instead, they are strongly consistent unless you explicitly request eventual consistency.
  • Queries in transactions are no longer required to be ancestor queries.
  • Transactions are no longer limited to 25 entity groups.
  • Writes to an entity group are no longer limited to 1 per second.

For more about Datastore mode, see Firestore in Datastore mode.

As of June 2021, migrations from Cloud Datastore to Firestore in Datastore mode have started. The migrations are starting from very low traffic databases and will expand to higher traffic databases over the coming months.

Automatic upgrade to Firestore in Datastore mode

If you manage an application that uses Cloud Datastore, you will not need to update your application code. We will notify you about the schedule of your application's upgrade to Firestore in Datastore mode. The upgrade does not require downtime.

For additional questions about the automatic upgrade process, contact one of our support channels.

Upgrade stages

At a high level, we follow this process to upgrade your Cloud Datastore database to Firestore in Datastore mode. This process does not require application downtime:

  1. Add a new Firestore in Datastore mode data replica to your existing Cloud Datastore database. Asynchronously duplicate entity write operations to Firestore in Datastore mode.

  2. Copy existing data and index entries from Cloud Datastore to Firestore in Datastore mode. After copying, verify the data.

  3. Redirect entity reads directly to Firestore in Datastore mode. First redirect eventually consistent reads, and then redirect strongly consistent reads.

  4. Redirect entity writes and transactional reads directly to Firestore in Datastore mode.

This process uses the following stages.

1. Apply writes synchronously

During this stage, writes are applied synchronously to Cloud Datastore: writes will not report success until all changes to entities and indexes have been applied to at least one replica. This simulates the behavior of Firestore in Datastore mode which also applies writes synchronously (and differs from the default behavior of Cloud Datastore in which writes are applied asynchronously after being committed).

This stage is intended to surface any latency impact of synchronous applies in Firestore in Datastore mode prior to the upgrade. Synchronous application of writes will continue during and after migration.

Databases with very little activity will skip this stage. To determine if this stage has been included in the upgrade of your database, inspect the logs for the APPLY_WRITES_SYNCHRONOUSLY stage.

2. Copy and verify

This stage represents the start of migration. It introduces a Firestore in Datastore mode replica and performs the following steps:

  1. Journal

    Entity write operations to Cloud Datastore also begin to flow through a side-channel to the Firestore in Datastore mode replica. This happens as part of Cloud Datastore's existing replication system. These write operations do not affect write latency. The Firestore in Datastore mode replica buffers these write operations to apply them after the copy step.

  2. Copy

    In the Firestore in Datastore mode replica, create an offline copy of your existing data and index entries. The copy step does not impact Cloud Datastore operations. This step may last several days.

  3. Drain Journal

    Apply the writes from the journal step on top of the data from the offline copy.

  4. Verify Data

    Re-verify the data in Firestore in Datastore mode by comparing against the data in Cloud Datastore.

3. Redirect eventually consistent reads

Serve eventually consistent reads (queries with no ancestor filter) from Firestore in Datastore mode. Cloud Datastore semantics for reads still apply at this point:

  • Ancestor queries are strongly consistent.
  • Non-ancestor queries are eventually consistent.
  • Lookups are strongly consistent (except those explicitly configured for eventual consistency).

Firestore in Datastore mode continues to act as a replica of your Cloud Datastore data.

4. Redirect strongly consistent reads

Serve strongly consistent reads (non-transactional) from Firestore in Datastore mode. Note that Cloud Datastore semantics for reads still apply. Even though reads now come directly from Firestore, Firestore still relies on Cloud Datastore to guarantee that it's up to date for strongly consistent reads.

5. Redirect writes

Redirect entity writes and transactional reads to Firestore in Datastore mode. Concurrent modifications to the same entity continue to cause transaction aborts. Concurrent modifications to different entities within the same entity group no longer result in transaction aborts.

At the start of this stage, Firestore in Datastore mode still relies on Cloud Datastore to guarantee it's up to date before each write. After a final pass that ensures all previous writes are applied, Firestore in Datastore mode stops consulting Cloud Datastore.

6. Migration complete

Firestore in Datastore mode semantics for reads now apply: all queries are strongly consistent.

Pricing remains the same, but your billing now lists Firestore SKUs. The App Engine Quotas page starts to show Firestore usage instead of Cloud Datastore usage.

Transactions

Cloud Datastore databases will use optimistic concurrency for transactions in Firestore in Datastore mode. Optimistic concurrency preserves the existing behaviors of transactions in Cloud Datastore.

Some previously migrated databases with very little activity were migrated with pessimistic locks for transactions in Firestore in Datastore mode instead of optimistic concurrency.

To verify the concurrency mode of your database during or after migration, inspect the logs for the PREPARE stage.

Logging and progress notifications

The upgrade process uses Cloud Logging to publish progress updates. To view the logs use the Logs Explorer, the Cloud Logging API, or the Cloud SDK.

Updates are published to two logs under the datastore.googleapis.com logging service name:

Log name Monitored resource Payload
migration_state datastore_database type.googleapis.com/google.datastore.admin.v1.MigrationStateEvent
migration_progress datastore_database type.googleapis.com/google.datastore.admin.v1.MigrationProgressEvent

The migration_state log is updated when the overall state of the upgrade changes (RUNNING and COMPLETE).

The migration_progress log is updated each time the upgrade moves into a new stage (PREPARE, START, APPLY_WRITES_SYNCHRONOUSLY, COPY_AND_VERIFY, REDIRECT_EVENTUALLY_CONSISTENT_READS, REDIRECT_STRONGLY_CONSISTENT_READS, and REDIRECT_WRITES).

To receive notifications as your upgrade proceeds, you can create logs-based metrics based on the two logs and create alerts based on those metrics.

Cloud Monitoring metrics

The Cloud Monitoring metrics available for your Datastore database remain the same throughout the upgrade process, see available Datastore metrics.