Cloud Datastore accessed from App Engine uses indexes for every query your application makes. These indexes are updated whenever an entity changes, so the results can be returned quickly when the app makes a query. To do this, Cloud Datastore needs to know in advance which queries the application will make. You specify which indexes your app needs in a configuration file. The development server can generate the Cloud Datastore index configuration automatically as you test your app.
Creating Cloud Datastore indexes
You specify configuration for Cloud Datastore indexes in
WEB-INF/datastore-indexes.xml, in your
app's war/ directory.
As described on the Datastore Indexes page, an index is a table of values for a set of given properties for entities of a given kind. Each column of property values is sorted either in ascending or descending order. Configuration for an index specifies the kind of the entities, and the names of the properties and their sort orders.
The following is an example of the datastore-indexes.xml that specifies two indexes:
<?xml version="1.0" encoding="utf-8"?>
<datastore-indexes
autoGenerate="true">
<datastore-index kind="Employee" ancestor="false">
<property name="lastName" direction="asc" />
<property name="hireDate" direction="desc" />
</datastore-index>
<datastore-index kind="Project" ancestor="false">
<property name="dueDate" direction="asc" />
<property name="cost" direction="desc" />
</datastore-index>
</datastore-indexes>
Creating indexes using the development server
Determining the indexes required by your application's queries manually can be
tedious and error-prone. Thankfully, the development server can determine the
index configuration for you. To use automatic index configuration, add the
attribute autoGenerate="true" to your WEB-INF/datastore-indexes.xml file's
<datastore-indexes> element. Automatic index configuration is also used if
your app does not have a datastore-indexes.xml file.
With automatic index configuration enabled, the development server maintains a
file named WEB-INF/appengine-generated/datastore-indexes-auto.xml in
your app's war/ directory. When your app, running in the development server,
attempts a Cloud Datastore query for which there is no corresponding
index in either datastore-indexes.xml or datastore-indexes-auto.xml, the server adds
the appropriate configuration to datastore-indexes-auto.xml.
If automatic index configuration is enabled when you upload your application,
AppCfg uses both datastore-indexes.xml and datastore-indexes-auto.xml to
determine which indexes need to be built for your app in production.
If autoGenerate="false" is in your datastore-indexes.xml, the development
server and AppCfg ignore the contents of datastore-indexes-auto.xml. If the
app running locally performs a query whose index is not specified in
datastore-indexes.xml, the development server throws an exception, just as the
production Cloud Datastore would.
It's a good idea to occasionally move index configuration from
datastore-indexes-auto.xml to datastore-indexes.xml, then disable automatic
index configuration and test your app in the development server. This makes it
easy to maintain indexes without having to manage two files, and ensures that
your testing will reproduce errors caused by missing index configuration.
Creating indexes manually
You can add indexes to your datastore-indexes.xml file manually or you might shift
indexes that the development server had automatically created in the
datastore-indexes-auto.xml file into your datastore-indexes.xml file.
See the datastore-indexes.xml reference for
syntax information.
Updating indexes
When you upload an application using the update action, the update
includes the app's index configuration (the datastore-indexes.xml
and generated/datastore-indexes-auto.xml files). If the index
configuration defines an index that doesn't exist yet, App Engine
creates the new index. Depending on how much data is already in
Cloud Datastore that needs to be mentioned in the new index, the
process of creating the index can take a while. If the app performs a query that
requires an index that hasn't finished building yet, the query will raise an
exception.
To prevent this, you must ensure that the new version of the app that
requires a new index is not the live version of the application until the
indexes finish building. One way to do this is to give the app a new version
number in appengine-web.xml whenever you add or change an index in
the configuration. The app is uploaded as a new version, and does not become the
default version automatically. When your indexes have finished building, you can
change the default
version in the
GCP Console.
Another way to ensure that new indexes are built before the new app goes live is
to upload the index configuration separately before uploading the app. To upload
only the index configuration for an app, use the update_indexes
action:
./appengine-java-sdk/bin/appcfg.sh update_indexes myapp/war
The datastore-indexes.xml file should be in the default module's WEB-INF/
directory.
In the GCP Console, you can check the status of the app's indexes.
Deleting unused indexes
When you change or remove an index from the index configuration, the original index is not deleted from App Engine automatically. This gives you the opportunity to leave an older version of the app running while new indexes are being built, or to revert to the older version immediately if a problem is discovered with a newer version.
When you are sure that old indexes are no longer needed, you can delete them
from App Engine using the vacuum_indexes action:
./appengine-java-sdk/bin/appcfg.sh vacuum_indexes myapp/war
This command deletes all indexes for the app that are not mentioned in the local
versions of datastore-indexes.xml and
WEB-INF/appengine-generated/datastore-indexes-auto.xml.
