Index Configuration

Google Cloud Datastore uses indexes for every query your application makes. These indexes are updated whenever an entity changes, so the results can be returned quickly when the application makes a query. Cloud Datastore provides built-in indexes automatically, but needs to know in advance which composite indexes the application will require. You specify which composite indexes your application needs in a configuration file. The Cloud Datastore emulator can generate the Cloud Datastore index configuration automatically as you test your application. The gcloud command-line tool provides commands to update the indexes that are available to your production Cloud Datastore instance.

System requirements

To use the gcloud tool, you must have installed the Google Cloud SDK.

About index.yaml

Every Cloud Datastore query made by an application needs a corresponding index. Indexes for simple queries, such as queries over a single property, are created automatically. Indexes for complex queries must be defined in a configuration file named index.yaml. This file is uploaded with the application to create indexes in Cloud Datastore.

The Cloud Datastore emulator automatically adds items to this file when the application tries to execute a query that needs an index that does not have an appropriate entry in the configuration file. You can adjust indexes or create new ones manually by editing the file. The index.yaml is located in the <project-directory>/WEB-INF/ folder. By default, the data directory that contains WEB-INF/appengine-generated/index.yaml is ~/.config/gcloud/emulators/datastore/. See Cloud Datastore emulator project directories for additional details.

Note: If during testing the application exercises every query it will make using the emulator, then the generated entries in index.yaml will be complete. You only need to edit the file manually to delete indexes that are no longer used, or to define indexes not created by the emulator.

The following is an example of an index.yaml file:

indexes:

- kind: Task
  ancestor: no
  properties:
  - name: done
  - name: priority
    direction: desc

- kind: Task
  properties:
  - name: collaborators
    direction: asc
  - name: created
    direction: desc

- kind: TaskList
  ancestor: yes
  properties:
  - name: percent_complete
    direction: asc
  - name: type
    direction: asc

The syntax of index.yaml is the YAML format. For more information about this syntax, see the YAML website.

Note: The YAML format supports comments. A line that begins with a pound (#) character is ignored:
# This is a comment.

Index definitions

index.yaml has a single list element called indexes. Each element in the list represents an index for the application.

An index element can have the following elements:

kind

The kind of the entity for the query. This element is required.

properties

A list of properties to include as columns of the index, in the order to be sorted: properties used in equality filters first, followed by the property used in inequality filters, then the sort orders and their directions.

Each element in this list has the following elements:

name

The Cloud Datastore name of the property.

direction

The direction to sort, either asc for ascending or desc for descending. This is only required for properties used in sort orders of the query, and must match the direction used by the query. The default is asc.

ancestor

yes if the query has an ancestor clause. The default is no.

Automatic and manual indexes

When the Cloud Datastore emulator adds a generated index definition to index.yaml, it does so below the following line, inserting it if necessary:

# AUTOGENERATED

The emulator considers all index definitions below this line to be automatic, and it may update existing definitions below this line as the application makes queries.

All index definitions above this line are considered to be under manual control, and are not updated by the emulator. The emulator will only make changes below the line, and will only do so if the complete index.yaml file does not describe an index that accounts for a query executed by the application. To take control of an automatic index definition, move it above this line.

Updating indexes

The datastore indexes create command looks at your local Datastore index configuration (the index.yaml file), and if the index configuration defines an index that doesn't exist yet in your production Cloud Datastore instance, Cloud Datastore creates the new index. See the development workflow using the gcloud tool for an example of how to use indexes create.

Depending on how much data is already in Cloud Datastore that belongs in the new index, the process of creating the index may take a while. If the application performs a query that requires an index that hasn't finished building yet, the query will raise an exception. To prevent this, you must be careful about deploying a new version of your application that requires a new index before the new index finishes building.

You can check the status of the Cloud Datastore instance's indexes from the Indexes page in the GCP Console.

Deleting unused indexes

When you change or remove an index from the index configuration, the original index is not deleted from Cloud Datastore automatically. This gives you the opportunity to leave an older version of the application running while new indexes are being built, or to revert to the older version immediately if a problem is discovered with a newer version.

When you are sure that old indexes are no longer needed, you can delete them from Cloud Datastore using the datastore indexes cleanup command. This command deletes all indexes for the production Cloud Datastore instance that are not mentioned in the local version of index.yaml. See the development workflow using the gcloud tool for an example of how to use indexes cleanup.

Command-line arguments

For details on command-line arguments for creating and cleaning indexes, see datastore indexes create and datastore indexes cleanup, respectively. For details on command-line arguments for the gcloud tool, see the gcloud tool reference.