This page describes the upgrade path from Cloud Datastore to Firestore in Datastore mode.
Firestore can operate in Datastore mode, making it backwards-compatible with Cloud Datastore. With Firestore in Datastore mode, you can access Firestore's improved storage layer while keeping Datastore system behavior. Firestore in Datastore mode removes the following Cloud Datastore limitations:
- Queries are no longer eventually consistent. Instead, they are strongly consistent unless you explicitly request eventual consistency.
- Queries in transactions are no longer required to be ancestor queries.
- Transactions are no longer limited to 25 entity groups.
- Writes to an entity group are no longer limited to 1 per second.
For more about Datastore mode, see Firestore in Datastore mode.
As of June 2021, migrations from Cloud Datastore to Firestore in Datastore mode have started. The migrations are starting from very low traffic databases and will expand to higher traffic databases over the coming months.
Automatic upgrade to Firestore in Datastore mode
If you manage an application that uses Cloud Datastore, you will not need to update your application code. We will notify you about the schedule of your application's upgrade to Firestore in Datastore mode. The upgrade does not require downtime.
For additional questions about the automatic upgrade process, contact one of our support channels.
At a high level, we follow this process to upgrade your Cloud Datastore database to Firestore in Datastore mode. This process does not require application downtime:
Add a new Firestore in Datastore mode data replica to your existing Cloud Datastore database. Asynchronously duplicate entity write operations to Firestore in Datastore mode.
Copy existing data and index entries from Cloud Datastore to Firestore in Datastore mode. After copying, verify the data.
Redirect entity reads directly to Firestore in Datastore mode. First redirect eventually consistent reads, and then redirect strongly consistent reads.
Redirect entity writes and transactional reads directly to Firestore in Datastore mode.
This process uses the following stages.
1. Apply writes synchronously
During this stage, writes are applied synchronously to Cloud Datastore: writes will not report success until all changes to entities and indexes have been applied to at least one replica. This simulates the behavior of Firestore in Datastore mode which also applies writes synchronously (and differs from the default behavior of Cloud Datastore in which writes are applied asynchronously after being committed).
This stage is intended to surface any latency impact of synchronous applies in Firestore in Datastore mode prior to the upgrade. Synchronous application of writes will continue during and after migration.
Databases with very little activity will skip this stage. To determine if
this stage has been included in the upgrade of your database, inspect the logs
2. Copy and verify
This stage represents the start of migration. It introduces a Firestore in Datastore mode replica and performs the following steps:
Entity write operations to Cloud Datastore also begin to flow through a side-channel to the Firestore in Datastore mode replica. This happens as part of Cloud Datastore's existing replication system. These write operations do not affect write latency. The Firestore in Datastore mode replica buffers these write operations to apply them after the copy step.
In the Firestore in Datastore mode replica, create an offline copy of your existing data and index entries. The copy step does not impact Cloud Datastore operations. This step may last several days.
Apply the writes from the journal step on top of the data from the offline copy.
Re-verify the data in Firestore in Datastore mode by comparing against the data in Cloud Datastore.
3. Redirect eventually consistent reads
Serve eventually consistent reads (queries with no ancestor filter) from Firestore in Datastore mode. Cloud Datastore semantics for reads still apply at this point:
- Ancestor queries are strongly consistent.
- Non-ancestor queries are eventually consistent.
- Lookups are strongly consistent (except those explicitly configured for eventual consistency).
Firestore in Datastore mode continues to act as a replica of your Cloud Datastore data.
4. Redirect strongly consistent reads
Serve strongly consistent reads (non-transactional) from Firestore in Datastore mode. Note that Cloud Datastore semantics for reads still apply. Even though reads now come directly from Firestore, Firestore still relies on Cloud Datastore to guarantee that it's up to date for strongly consistent reads.
5. Redirect writes
Redirect entity writes and transactional reads to Firestore in Datastore mode. Concurrent modifications to the same entity continue to cause transaction aborts. Concurrent modifications to different entities within the same entity group no longer result in transaction aborts.
At the start of this stage, Firestore in Datastore mode still relies on Cloud Datastore to guarantee it's up to date before each write. After a final pass that ensures all previous writes are applied, Firestore in Datastore mode stops consulting Cloud Datastore.
6. Migration complete
Firestore in Datastore mode semantics for reads now apply: all queries are strongly consistent.
Pricing remains the same, but your billing now lists Firestore SKUs. The App Engine Quotas page starts to show Firestore usage instead of Cloud Datastore usage.
Behavior of transactions after the upgrade depends on database activity level.
Cloud Datastore databases with activity will use optimistic concurrency for transactions in Firestore in Datastore mode. Optimistic concurrency preserves the existing behaviors of transactions in Cloud Datastore.
Databases with very little activity will use pessimistic locks
for transactions after the upgrade instead of optimistic concurrency. To verify the concurrency
mode of your database during or after migration, inspect the logs for the
Logging and progress notifications
Updates are published to two logs under the
datastore.googleapis.com logging service name:
|Log name||Monitored resource||Payload|
The migration_state log is updated when the overall state of the upgrade changes (
The migration_progress log is updated each time the upgrade moves into a new stage (
Testing an existing application
For an existing app that uses Cloud Datastore, you can test app behavior with Firestore in Datastore mode by doing the following:
- Create a new project. In this project, create a Firestore in Datastore mode database.
- Using the managed export service, export some of your application's data to Cloud Storage.
- Using the managed import service, import your application's data to your new project.
- Copy app logic you want to test to the new project or simulate app behavior against the new project.