Migrating to the Cloud Datastore Emulator

Cloud Datastore Emulator provides local emulation of the production Datastore environment. Cloud Datastore Emulator is progressively being rolled out as the default Datastore implementation for dev_appserver.

Benefits of migrating to Datastore Emulator

The Datastore Emulator can be shared between App Engine standard environment client libraries such as DB and NDB, and the Cloud Datastore Client Libraries.

You can use the Datastore Emulator to share local data between services running on App Engine standard environment and App Engine flexible environment.

Before you start

  1. Download and install the Java runtime (JRE) version 11 or above.

  2. Run dev_appserver from the google-cloud-sdk/bin directory.

Choosing your emulation environment

The Cloud Datastore Emulator is the default emulator for a portion of dev_appserver users. If you are using the Cloud Datastore Emulator, dev_appserver will display:

... Using Cloud Datastore Emulator.

You can force dev_appserver to use the Cloud Datastore emulator by setting the --support_datastore_emulator flag to true when starting dev_appserver.py:

python2 DEVAPPSERVER_ROOT/google_appengine/dev_appserver.py --support_datastore_emulator=[true|false] app.yaml

Replace DEVAPPSERVER_ROOT with the path to the folder where you extract the archived version of devapp_server.py. For more information about downloading and using the archived version of dev_appserver.py, see Using the local development server.

Using dev_appserver with the Datastore Emulator

Dev_appserver decides whether to launch the emulator based on the environment variable DATASTORE_EMULATOR_HOST.

  • When this environment variable exists, dev_appserver communicates to an existing emulator process running on DATASTORE_EMULATOR_HOST.

  • When this environment variable does not exist, dev_appserver launches a new emulator process. You can specify the port for the Datastore Emulator by setting the --datastore_emulator_port when starting dev_appserver.py.

Behavior changes

Local data format conversion

Currently, the local Datastore emulator stores data in sqlite3 while the Cloud Datastore Emulator stores data as Java objects.

When dev_appserver is launched with legacy sqlite3 data, the data will be converted to Java objects. The original data is backed up with the filename {original-data-filename}.sqlitestub.

Local data file location

The local data file can be specified with the flag --datastore_path.

  • When dev_appserver launches the emulator, the location remain unchanged.

  • When dev_appserver uses existing Cloud Datastore Emulator, this flag will be ignored in favor of the emulator's setting.

Local index.yaml

  • An empty index.yaml will be considered as invalid. The syntax of index.yaml is the YAML format. The index.yaml file has a single list element called indexes.

  • When dev_appsever launches the emulator, index.yaml will stay in the application project directory.

  • When dev_appserver uses existing Cloud Datastore Emulator, index.yaml will be managed by that emulator.

Other features for the local datastore server remain unchanged.

Troubleshooting

Getting emulator output

By default dev_appserver runs Datastore Emulator in silent mode. To see the output of Datastore Emulator, run dev_appserver with --dev_appserver_log_level=debug.

Failing gRPC imports

The dev_appserver uses gRPC to communicate to the Cloud Datastore Emulator. gcloud CLI packages gRPC, however the packaged version is incompatible with some operating systems, causing import failures.

To fix import failures, you can install gRPC separately using pip with gcloud CLI 219.0.0 or higher:

pip install grpcio