Automatic Upgrade to Firestore

This page describes the upgrade path from legacy Cloud Datastore to Firestore in Datastore mode.

Firestore can operate in Datastore mode, making it backwards-compatible with legacy Cloud Datastore. With Firestore in Datastore mode, you can access Firestore's improved storage layer while keeping Datastore system behavior. Firestore in Datastore mode removes the following legacy Cloud Datastore limitations:

• Queries are no longer eventually consistent. Instead, they are strongly consistent unless you explicitly request eventual consistency.
• Queries in transactions are no longer required to be ancestor queries¹.
• Transactions are no longer limited to 25 entity groups¹.
• Writes to an entity group are no longer limited to 1 per second¹.

For more about Datastore mode, see Firestore in Datastore mode.

As of June 2021, migrations from legacy Cloud Datastore to Firestore in Datastore mode have started. The migrations are starting from very low traffic databases and will expand to higher traffic databases over the coming months.

¹ Databases that migrate to the Optimistic With Entity Groups concurrency mode are still subject to the 25 entity group transaction limit as well as the 1 write per second limit in Firestore in Datastore mode. Queries in transactions are required to be ancestor queries. See the Optimistic With Entity Groups Concurrency mode section for more information.

Automatic upgrade to Firestore in Datastore mode

If you manage an application that uses legacy Cloud Datastore, you will not need to update your application code. We will notify you about the schedule of your application's upgrade to Firestore in Datastore mode. The upgrade does not require downtime.

For additional questions about the automatic upgrade process, contact one of our support channels.

View your database type

You can use the gcloud alpha firestore databases describe command to view your database type. Look for the presence of the type field in the output:

• type: DATASTORE_MODE

  The database type is Firestore in Datastore mode. It does not require an upgrade or already completed the upgrade.

• type not present in output

  The database type is legacy Cloud Datastore. The database will be upgraded to Firestore in Datastore mode.

• type: FIRESTORE_NATIVE

  The database type is Firestore in Native mode.

Upgrade stages

At a high level, we follow this process to upgrade your legacy Cloud Datastore database to Firestore in Datastore mode. This process does not require application downtime:

1. Add a new Firestore in Datastore mode data replica to your existing legacy Cloud Datastore database. Asynchronously duplicate entity write operations to Firestore in Datastore mode.

2. Copy existing data and index entries from legacy Cloud Datastore to Firestore in Datastore mode. After copying, verify the data.

3. Redirect entity reads directly to Firestore in Datastore mode. First redirect eventually consistent reads, and then redirect strongly consistent reads.

4. Redirect entity writes and transactional reads directly to Firestore in Datastore mode.

This process uses the following stages.

1. Apply writes synchronously

During this stage, writes are applied synchronously to legacy Cloud Datastore: writes will not report success until all changes to entities and indexes have been applied to at least one replica. This simulates the behavior of Firestore in Datastore mode which also applies writes synchronously (and differs from the default behavior of legacy Cloud Datastore in which writes are applied asynchronously after being committed).

This stage is intended to surface any latency impact of synchronous applies in Firestore in Datastore mode prior to the upgrade. Synchronous application of writes will continue during and after migration.

Note: legacy Cloud Datastore semantics for reads still apply during this stage: non-ancestor queries are still eventually consistent because preceding writes may not have been applied on the replica serving the read.

Databases with very little activity will skip this stage. To determine if this stage has been included in the upgrade of your database, inspect the [logs] for the APPLY_WRITES_SYNCHRONOUSLY stage.

2. Copy and verify

This stage represents the start of migration. It introduces a Firestore in Datastore mode replica and performs the following steps:

1. Journal

   Entity write operations to legacy Cloud Datastore also begin to flow through a side-channel to the Firestore in Datastore mode replica. This happens as part of legacy Cloud Datastore's existing replication system. These write operations do not affect write latency. The Firestore in Datastore mode replica buffers these write operations to apply them after the copy step.

2. Copy

   In the Firestore in Datastore mode replica, create an offline copy of your existing data and index entries. The copy step does not impact legacy Cloud Datastore operations. This step may last several days.

3. Drain Journal

   Apply the writes from the journal step on top of the data from the offline copy.

4. Verify Data

   Re-verify the data in Firestore in Datastore mode by comparing against the data in legacy Cloud Datastore.

3. Redirect eventually consistent reads

Serve eventually consistent reads (queries with no ancestor filter) from Firestore in Datastore mode. legacy Cloud Datastore semantics for reads still apply at this point:

• Ancestor queries are strongly consistent.
• Non-ancestor queries are eventually consistent.
• Lookups are strongly consistent (except those explicitly configured for eventual consistency).

Firestore in Datastore mode continues to act as a replica of your legacy Cloud Datastore data.

4. Redirect strongly consistent reads

Serve strongly consistent reads (non-transactional) from Firestore in Datastore mode. Note that legacy Cloud Datastore semantics for reads still apply. Even though reads now come directly from Firestore, Firestore still relies on legacy Cloud Datastore to guarantee that it's up to date for strongly consistent reads.

5. Redirect writes

Redirect entity writes and transactional reads to Firestore in Datastore mode. Concurrent modifications to the same entity continue to cause transaction aborts. Concurrent modifications to different entities within the same entity group no longer result in transaction aborts.

At the start of this stage, Firestore in Datastore mode still relies on legacy Cloud Datastore to guarantee it's up to date before each write. After a final pass that ensures all previous writes are applied, Firestore in Datastore mode stops consulting legacy Cloud Datastore.

6. Migration complete

Firestore in Datastore mode semantics for reads now apply: all queries are strongly consistent.

Pricing remains the same, but your billing now lists Firestore SKUs. The App Engine Quotas page starts to show Firestore usage instead of legacy Cloud Datastore usage.

Transactions

Firestore in Datastore mode supports three concurrency modes:

• Optimistic

  Most legacy Cloud Datastore databases will use optimistic concurrency for transactions in Firestore in Datastore mode. Optimistic concurrency preserves the existing behaviors of transactions in legacy Cloud Datastore.

• Optimistic With Entity Groups

  Databases that depend on entity group transactional semantics will migrate to this concurrency mode. See the Optimistic With Entity Groups Concurrency mode section for more information.

• Pessimistic

  Some previously migrated databases with very little activity were migrated with pessimistic locks for transactions in Firestore in Datastore mode.

The concurrency mode is accessible through the Firestore projects.databases REST resource:

curl -X GET -H "Authorization: Bearer "$(gcloud auth print-access-token) \
"https://firestore.googleapis.com/v1/projects/PROJECT_ID/databases"

The concurrency mode can also be found by inspecting the logs for the PREPARE stage.

Optimistic With Entity Groups concurrency mode

To remove Optimistic With Entity Groups's query, transaction and write throughput limitations, change your project's concurrency mode to Optimistic. To ensure this change is compatible with your project:

1. Create a test project in Firestore in Datastore mode.

2. Change the test project's concurrency mode to OPTIMISTIC. Issue an HTTP PATCH request, as demonstrated below.

3. Run tests on the test project to ensure that you workload performs as expected without Entity Groups.

4. Change your main project's concurrency mode from OPTIMISTIC_WITH_ENTITY_GROUPS to OPTIMISTIC.

Note: The Remote API library requires Entity Groups and will not work in the OPTIMISTIC concurrency mode. If you use the Remote API, either migrate off this API, or to continue using this library, keep your project in the OPTIMISTIC_WITH_ENTITY_GROUPS mode.

HTTP PATCH request to change the database concurrency mode:

curl --request PATCH \
--header "Authorization: Bearer "$(gcloud auth print-access-token) \
--header 'Accept: application/json' \
--header 'Content-Type: application/json' \
--data '{"concurrencyMode":"OPTIMISTIC"}' \
"https://firestore.googleapis.com/v1/projects/PROJECT_ID/databases/(default)?updateMask=concurrencyMode"

Logging and progress notifications

The upgrade process uses Cloud Logging to publish progress updates. To view the logs, use the Logs Explorer, the Cloud Logging API, or the Google Cloud CLI.

Note: Your Google Cloud project must have billing enabled in order to receive the progress logs. Learn how to confirm that billing is enabled for your project.

Updates are published to two logs under the datastore.googleapis.com logging service name:

Log name	Monitored resource	Payload
migration_state	datastore_database	type.googleapis.com/google.datastore.admin.v1.MigrationStateEvent
migration_progress	datastore_database	type.googleapis.com/google.datastore.admin.v1.MigrationProgressEvent

The migration_state log is updated when the overall state of the upgrade changes (RUNNING and COMPLETE).

The migration_progress log is updated each time the upgrade moves into a new stage (PREPARE, START, APPLY_WRITES_SYNCHRONOUSLY, COPY_AND_VERIFY, REDIRECT_EVENTUALLY_CONSISTENT_READS, REDIRECT_STRONGLY_CONSISTENT_READS, and REDIRECT_WRITES).

To receive notifications as your upgrade proceeds, you can create logs-based metrics based on the two logs and create alerts based on those metrics.

Note: Logs-based metrics based on these logs must used the global resource type.

Migration banner in the Google Cloud console

While your legacy Cloud Datastore database is in the migration process, an information banner will appear in the Datastore Studio page of the Google Cloud console. This banner includes a link to open Cloud Logging and filter for migration updates.

1. In the Google Cloud console, go to the Databases page.

   Go to Databases

2. Select the required database from the list of databases.

3. In the navigation menu, click Datastore Studio.

Viewing the current state on a CLI

To quickly view the current state of a migration, use the following gcloud command:

gcloud datastore operations describe datastore-firestore-migration

Pausing migration

Large database migrations are eligible to be paused and resumed. Pausing a migration prevents it from progressing to the next stage until it has been resumed. Pausing a migration may help you determine if an observed change in behavior or performance is the result of the migration process or from an unrelated factor.

After you receive the email notification about your database's migration, you can check whether it is eligible to be paused and resumed by running the pause command below. If the migration is not eligible, it will return an error indicating that the functionality is not available.

If your database's migration is eligible to be paused and resumed, the commands below will begin working once the migration reaches the START stage.

To pause a migration:

curl --request POST \
--header "Authorization: Bearer "$(gcloud auth print-access-token) \
--header 'Accept: application/json' \
--header 'Content-Type: application/json' \
--data '{}' \
"https://datastore.googleapis.com/v1/projects/PROJECT_ID:pauseMigration"

To resume a migration:

curl --request POST \
--header "Authorization: Bearer "$(gcloud auth print-access-token) \
--header 'Accept: application/json' \
--header 'Content-Type: application/json' \
--data '{}' \
"https://datastore.googleapis.com/v1/projects/PROJECT_ID:resumeMigration"

These commands will not work after migration is complete.

If you need to keep your migration paused for more than one week, reach out through a support channel. After two weeks, your migration may be automatically resumed.

Cloud Monitoring metrics

The Cloud Monitoring metrics available for your Datastore database remain the same throughout the upgrade process, see available Datastore metrics.