Configuring Datastore Indexes

Google Cloud Datastore accessed from App Engine uses indexes for every query your application makes. These indexes are updated whenever an entity changes, so the results can be returned quickly when the app makes a query. To do this, Cloud Datastore needs to know in advance which queries the application will make. You specify which indexes your app needs in a configuration file. The development server can generate the Cloud Datastore index configuration automatically as you test your app.

Creating Cloud Datastore indexes

You specify configuration for Cloud Datastore indexes in WEB-INF/datastore-indexes.xml, in your app's war/ directory.

As described on the Datastore Indexes page, an index is a table of values for a set of given properties for entities of a given kind. Each column of property values is sorted either in ascending or descending order. Configuration for an index specifies the kind of the entities, and the names of the properties and their sort orders.

The following is an example of the datastore-indexes.xml that specifies two indexes:

<?xml version="1.0" encoding="utf-8"?>
    <datastore-index kind="Employee" ancestor="false">
        <property name="lastName" direction="asc" />
        <property name="hireDate" direction="desc" />
    <datastore-index kind="Project" ancestor="false">
        <property name="dueDate" direction="asc" />
        <property name="cost" direction="desc" />

Creating indexes using the development server

Determining the indexes required by your application's queries manually can be tedious and error-prone. Thankfully, the development server can determine the index configuration for you. To use automatic index configuration, add the attribute autoGenerate="true" to your WEB-INF/datastore-indexes.xml file's <datastore-indexes> element. Automatic index configuration is also used if your app does not have a datastore-indexes.xml file.

With automatic index configuration enabled, the development server maintains a file named WEB-INF/appengine-generated/datastore-indexes-auto.xml in your app's war/ directory. When your app, running in the development server, attempts a Cloud Datastore query for which there is no correspondin index in either datastore-indexes.xml or datastore-indexes-auto.xml, the server adds the appropriate configuration to datastore-indexes-auto.xml.

If automatic index configuration is enabled when you upload your application, AppCfg uses both datastore-indexes.xml and datastore-indexes-auto.xml to determine which indexes need to be built for your app in production.

If autoGenerate="false" is in your datastore-indexes.xml, the development server and AppCfg ignore the contents of datastore-indexes-auto.xml. If the app running locally performs a query whose index is not specified in datastore-indexes.xml, the development server throws an exception, just as the production Cloud Datastore would.

It's a good idea to occasionally move index configuration from datastore-indexes-auto.xml to datastore-indexes.xml, then disable automatic index configuration and test your app in the development server. This makes it easy to maintain indexes without having to manage two files, and ensures that your testing will reproduce errors caused by missing index configuration.

Creating indexes manually

You can add indexes to your datastore-indexes.xml file manually or you might shift indexes that the development server had automatically created in the datastore-indexes-auto.xml file into your datastore-indexes.xml file.

See the datastore-indexes.xml reference for syntax information.

Updating indexes

When you upload an application using the update action, the update includes the app's index configuration (the datastore-indexes.xml and generated/datastore-indexes-auto.xml files). If the index configuration defines an index that doesn't exist yet, App Engine creates the new index. Depending on how much data is already in Cloud Datastore that needs to be mentioned in the new index, the process of creating the index can take a while. If the app performs a query that requires an index that hasn't finished building yet, the query will raise an exception.

To prevent this, you must ensure that the new version of the app that requires a new index is not the live version of the application until the indexes finish building. One way to do this is to give the app a new version number in appengine-web.xml whenever you add or change an index in the configuration. The app is uploaded as a new version, and does not become the default version automatically. When your indexes have finished building, you can change the default version in the Cloud Platform Console.

Another way to ensure that new indexes are built before the new app goes live is to upload the index configuration separately before uploading the app. To upload only the index configuration for an app, use the update_indexes action:

./appengine-java-sdk/bin/ update_indexes myapp/war

The datastore-indexes.xml file should be in the default module's WEB-INF/ directory.

In the Cloud Platform Console, you can check the status of the app's indexes.

Deleting unused indexes

When you change or remove an index from the index configuration, the original index is not deleted from App Engine automatically. This gives you the opportunity to leave an older version of the app running while new indexes are being built, or to revert to the older version immediately if a problem is discovered with a newer version.

When you are sure that old indexes are no longer needed, you can delete them from App Engine using the vacuum_indexes action:

./appengine-java-sdk/bin/ vacuum_indexes myapp/war

This command deletes all indexes for the app that are not mentioned in the local versions of datastore-indexes.xml and WEB-INF/appengine-generated/datastore-indexes-auto.xml.