Migrate from App Engine Datastore to Firestore in Datastore mode

This guide demonstrates how to migrate from App Engine Datastore to Firestore in Datastore mode (also referred to as Datastore).

Firestore in Datastore mode is similar to App Engine Datastore in that both refer to the same underlying Datastore service. While App Engine Datastore is accessible only through the App Engine legacy bundled services, Firestore in Datastore mode is a standalone Google Cloud product that is accessed through the Cloud Client Libraries.

Firestore in Datastore mode also offers a free tier and enables you to manage a highly scalable NoSQL document database and gives you the flexibility in the future to migrate to Cloud Run or another Google Cloud app hosting platform.

Before you begin

Review the different Firestore database modes to ensure you understand the best use case for your app. Note that this guide covers how to migrate to Datastore mode.
Review and understand Firestore in Datastore mode pricing and quotas.

Firestore in Datastore mode offers free usage with daily limits, and unlimited storage, read, and write operations for paid accounts. While App Engine apps are disabled, they won't get any traffic to incur charges, however Datastore usage may be billable if it exceeds the free quota limits.
Enable the following APIs in the project containing your app:
- Artifact Registry API to store and manage your build artifacts
- Cloud Build API to continuously build, test, and deploy your application.
- Cloud Datastore API to migrate from App Engine bundled Datastore to Firestore in Datastore mode.
  
  Enable APIs
Have an existing App Engine app running Java 8 or 11 that is connected to the App Engine Datastore service.

Process overview

At a high level, the process to migrate to Firestore in Datastore mode from App Engine Datastore consists of the following steps:

Update configuration files
Update your Java app
Commit your transaction
Query results

Update configuration files

Update your configuration files to use the Datastore mode client libraries.

Update the pom.xml file of your baseline Java app:

Remove the App Engine SDK appengine-api-1.0-sdk imports, like the following:

<dependency>
  <groupId>com.google.appengine</groupId>
  <artifactId>appengine-api-1.0-sdk</artifactId>
  <version>2.0.4</version>
</dependency>

Add the Datastore client, like the following:

<dependency>
  <groupId>com.google.cloud</groupId>
  <artifactId>google-cloud-datastore</artifactId>
  <!-- Use latest version -->
  <version>2.2.9</version>
</dependency>

Update your Java app

Update import statements

Modify your application files by updating the import and initialization lines:

Remove the following App Engine import statements for App Engine Datastore com.google.appengine.api.datastore.*:

  import com.google.appengine.api.datastore.DatastoreService;
  import com.google.appengine.api.datastore.DatastoreServiceFactory;
  import com.google.appengine.api.datastore.Entity;
  import com.google.appengine.api.datastore.FetchOptions;
  import com.google.appengine.api.datastore.Query;

Add the following Firestore in Datastore mode com.google.cloud.datastore.* imports:

  import com.google.cloud.Timestamp;
  import com.google.cloud.datastore.Datastore;
  import com.google.cloud.datastore.DatastoreOptions;
  import com.google.cloud.datastore.Entity;
  import com.google.cloud.datastore.Key;
  import com.google.cloud.datastore.FullEntity;
  import com.google.cloud.datastore.KeyFactory;
  import com.google.cloud.datastore.Query;
  import com.google.cloud.datastore.QueryResults;
  import com.google.cloud.datastore.StructuredQuery;

Modify how your app accesses the Datastore service

Firestore in Datastore mode uses the Datastore class instead of DatastoreService. To modify how your app access the Datastore service:

Find the lines that use the DatastoreServiceFactory.getDatastoreService() method, like the following:

// Initialize a client
DatastoreService datastore = DatastoreServiceFactory.getDatastoreService();

Replace the DatastoreServiceFactory.getDatastoreService() with DatastoreOptions.getDefaultInstance().getService() method, like the following:
```
// Initialize a client
Datastore datastore = DatastoreOptions.getDefaultInstance().getService();
```

Obtain a Datastore generated key

After you initialize a client, get your key by creating a new KeyFactory of the appropriate Kind, then have Datastore generate one for you. To obtain a Datastore generated key:

Create a newKeyFactory.
Call the setKind() method to determine the kind of entity used for query categorizations.

Append the newKey() method to generate a Datastore key:

//Prepare a new entity
String kind = "visit";
Key key = datastore.allocateId(datastore.newKeyFactory().setKind(kind).newKey());

Modify entity creation

After obtaining a Datastore key, create entities using the following methods:

Use Entity.newBuilder, and pass the key generated by the Datastore.

Find the lines that use the Entity constructor call, like the following:
```
Entity visit = new Entity(kind);
```
Replace the Entity constructor call with the Entity.newBuilder constructor call, like the following:
```
Entity visit = Entity.newBuilder(key);
```
Use the set method to set properties on entities.

The first parameter is the intended property, and the second is the value. In the case of the timestamp property, the value is a Timestamp instead of an Instant.toString().

Find the lines that use the setProperty method, like the following:
```
visit.setProperty("user_ip", userIp);
visit.setProperty("timestamp", Instant.now().toString());
```
Replace the setProperty method with the set method, like the following:
```
Entity visit = Entity.newBuilder(key).set("user_ip", userIp).set("timestamp", Timestamp.now()).build();
```

Commit your transaction

The Firestore in Datastore mode client library uses the add() method to commit a transaction. To commit your transaction:

Find lines that use the put() method, like the following:
```
// Save the entity
datastore.put(visit);
```
Replace the put() method with the add() method, like the following:
```
// Save the entity
datastore.add(visit);
```

Query Results

Queries retrieve entities that meet a specified set of conditions. You can use the following methods to display results:

The OrderBy method displays the results in ascending or descending order.
The Limit method limits the maximum number of results fetched using in the same builder.

Querying uses a builder pattern method with the kind variable. The kind variable is set to Visit from the Obtain a Datastore generated key step.

To retrieve the first 10 results:

Find lines that use the addSort() method, like the following:

  // Retrieve the last 10 visits from the datastore, ordered by timestamp.
  Query query = new Query(kind).addSort("timestamp", Query.SortDirection.DESCENDING);
  List<Entity> results = datastore.prepare(query).asList(FetchOptions.Builder.withLimit(10));

Replace the addSort() method with the setOrderBy() method and append the setLimit() method, like the following:

// Retrieve the last 10 visits from the datastore, ordered by timestamp.
Query<Entity> query = Query.newEntityQueryBuilder()
                        .setKind(kind)
                        .setOrderBy(StructuredQuery.OrderBy.desc("timestamp"))
                        .setLimit(10)
                        .build();

Once the query is ready, execute the code using datastore.run(), and collect the results in a QueryResultsEntity collection.

The resulting QueryResults object is an iterator with a hasNext() function.

Check if the result set has a next object for processing, instead of looping through the results list. For example:

QueryResults<Entity> results = datastore.run(query);

      resp.setContentType("text/plain");
      PrintWriter out = resp.getWriter();
      out.print("Last 10 visits:\n");
      while (results.hasNext()) {
          Entity entity = results.next();
          out.format(
                  "Time: %s Addr: %s\n", entity.getTimestamp("timestamp"), entity.getString("user_ip"));
      }

Examples

To see an example of how to migrate a Java 8 app to Firestore in Datastore mode, compare the App Engine Datastore for Java 8 code sample and the Firestore in Datastore mode code sample in GitHub.

What’s next

Learn how to Use Firestore in Datastore mode.
See Firestore in Datastore mode documentation for more details.
Learn how to Migrate off of legacy bundled services for Java.