Datastore Queries

A Datastore query retrieves entities from the Datastore that meet a specified set of conditions. The query operates on entities of a given kind; it can specify filters on the entities' property values, keys, and ancestors, and can return zero or more entities as results. A query can also specify sort orders to sequence the results by their property values. The results include all entities that have at least one value for every property named in the filters and sort orders, and whose property values meet all the specified filter criteria. The query can return entire entities, projected entities, or just entity keys.

A typical query includes the following:

• An entity kind to which the query applies
• Zero or more filters based on the entities' property values, keys, and ancestors
• Zero or more sort orders to sequence the results

Note: To conserve memory and improve performance, a query should, whenever possible, specify a limit on the number of results returned.

Every Datastore query computes its results using one or more indexes, tables containing entities in a sequence specified by the index's properties and, optionally, the entity's ancestors. The indexes are updated incrementally to reflect any changes the application makes to its entities, so that the correct results of all queries are immediately available with no further computation needed.

Note: The index-based query mechanism supports a wide range of queries and is suitable for most applications. However, it does not support some kinds of query common in other database technologies: in particular, joins and aggregate queries aren't supported within the Datastore query engine. See Restrictions on Queries, below, for limitations on Datastore queries.

Contents

1. Datastore query interface
2. Query structure
   1. Filters
      1. Property filters
      2. Key filters
      3. Ancestor filters
   2. Sort orders
   3. Special query types
      1. Kindless queries
      2. Ancestor queries
      3. Kindless ancestor queries
      4. Keys-only queries
      5. Projection queries
3. Restrictions on queries
4. Retrieving results
5. Query cursors
   1. Limitations of cursors
   2. Cursors and data updates
6. Data consistency

Datastore query interface

Here's a basic example of issuing a query against the Datastore. It retrieves all people whose height falls inside a given range:

var heightRangeQuery = {
  kinds: [{ name: 'Person' }],
  filter: {
    // Combine multiple filters by using a compositeFilter.
    compositeFilter: {
      operator: 'AND',
      filters: [{
        propertyFilter: {
          property: { name: 'height' },
          operator: 'GREATER_THAN_OR_EQUAL',
          value: { integerValue: minHeight }
        }
      }, {
        propertyFilter: {
          property: { name: 'height' },
          operator: 'LESS_THAN_OR_EQUAL',
          value: { integerValue: maxHeight }
        }
      }]
    },
  }
};

datastore.runQuery({
  query: heightRangeQuery
}).execute(function(err, result) {
  // Iterate over the results.
  if (!err && result.batch.entityResults) {
    result.batch.entityResults.forEach(function(er) {
      var entity = er.entity;
      console.log(entity.properties.firstName.stringValue, ' ',
                  entity.properties.lastName.stringValue, ', ',
                  entity.properties.height.integerValue, ' inches tall');
    });
    if (result.batch.moreResults == 'NOT_FINISHED') {
      var endCursor = result.batch.endCursor;
      heightRangeQuery.startCursor = endCursor;
      // Reissue the query to get more results...
    }
  }
  callback(err);
});

req = datastore.RunQueryRequest()
query = req.query

query.kind.add().name = 'Person'

# Use composite_filter to combine multiple filters
composite_filter = query.filter.composite_filter
composite_filter.operator = datastore.CompositeFilter.AND

height_min_filter = composite_filter.filter.add().property_filter
height_min_filter.property.name = 'height'
height_min_filter.operator = datastore.PropertyFilter.GREATER_THAN_OR_EQUAL
height_min_filter.value.integer_value = min_height

height_max_filter = composite_filter.filter.add().property_filter
height_max_filter.property.name = 'height'
height_max_filter.operator = datastore.PropertyFilter.LESS_THAN_OR_EQUAL
height_max_filter.value.integer_value = max_height

resp = self.datastore.run_query(req)

# Iterate over the results
for entity_result in resp.batch.entity_result:
  entity = entity_result.entity
  for prop in entity.property:
    if prop.name == 'first_name':
      first_name = prop.value.string_value
    elif prop.name == 'last_name':
      last_name = prop.value.string_value
    elif prop.name == 'height':
      height = prop.value.integer_value

  print '%s %s, %d inches tall' % (first_name, last_name, height)

if resp.batch.more_results == datastore.QueryResultBatch.NOT_FINISHED:
  end_cursor = resp.batch.end_cursor
  query.start_cursor.CopyFrom(end_cursor)
  # reissue the query to get more results...

// import static com.google.api.services.datastore.client.DatastoreHelper.*;
Filter heightMinFilter = makeFilter(
    "height", PropertyFilter.Operator.GREATER_THAN_OR_EQUAL, makeValue(minHeight)).build();

Filter heightMaxFilter = makeFilter(
    "height", PropertyFilter.Operator.LESS_THAN_OR_EQUAL, makeValue(maxHeight)).build();

// Use makeCompositeFilter() to combine multiple filters
Filter heightRangeFilter = makeFilter(heightMinFilter, heightMaxFilter).build();

// Use Query.Builder to assemble a query
Query.Builder q = Query.newBuilder();
q.addKindBuilder().setName("Person");
q.setFilter(heightRangeFilter).build();

// Assemble a RunQueryRequest
RunQueryRequest request = RunQueryRequest.newBuilder().setQuery(q).build();
RunQueryResponse response = datastore.runQuery(request);

// Iterate over the results
for (EntityResult result : response.getBatch().getEntityResultList()) {
  Map<String, Value> props = getPropertyMap(result.getEntity());
  String firstName = getString(props.get("firstName"));
  String lastName = getString(props.get("lastName"));
  Long height = getLong(props.get("height"));

  System.out.println(String.format("%s %s, %d inches tall", firstName, lastName, height));
}

if (response.getBatch().getMoreResults() == QueryResultBatch.MoreResultsType.NOT_FINISHED) {
  ByteString endCursor = response.getBatch().getEndCursor();
  q.setStartCursor(endCursor);
  // reissue the query to get more results...
}

Notice how we combine multiple filters into a single composite filter. If you're setting only one filter on a query, you can just put that one filter directly on the query:

var heightMinQuery = {
  kinds: [{ name: 'Person' }],
  filter: {
    propertyFilter: {
      property: { name: 'height' },
      operator: 'GREATER_THAN_OR_EQUAL',
      value: { integerValue: minHeight }
    }
  }
};

query = datastore.Query()
query.kind.add().name = 'Person'

height_min_filter = query.filter.property_filter
height_min_filter.property.name = 'height'
height_min_filter.operator = datastore.PropertyFilter.GREATER_THAN_OR_EQUAL
height_min_filter.value.integer_value = min_height

// import static com.google.api.services.datastore.client.DatastoreHelper.*;
Filter heightMinFilter = makeFilter(
    "height", PropertyFilter.Operator.GREATER_THAN_OR_EQUAL, makeValue(minHeight)).build();

Query.Builder q = Query.newBuilder();
q.addKindBuilder().setName("Person");
q.setFilter(heightMinFilter).build();

The Datastore currently only supports combining filters with the AND operator. However it's relatively straightforward to create your own OR query by issuing multiple queries and combining the results:

var async = require('async');

// Define the query for people who are too short.
var heightQuery1 = {
  kinds: [{ name: 'Person' }],
  filter: {
    propertyFilter: {
      property: { name: 'height' },
      operator: 'LESS_THAN',
      value: { integerValue: minHeight }
    }
  }
};
// Define the query for people who are too tall.
var heightQuery2 = {
  kinds: [{ name: 'Person' }],
  filter: {
    propertyFilter: {
      property: { name: 'height' },
      operator: 'GREATER_THAN',
      value: { integerValue: maxHeight }
    }
  }
};

async.waterfall([
  // Issue both queries in parallel.
  function(callback) {
    async.parallel([
      function(callback) {
        datastore.runQuery({ query: heightQuery1 }).execute(callback);
      },
      function(callback) {
        datastore.runQuery({ query: heightQuery2 }).execute(callback);
      }], callback);
  },
  // And assemble the results.
  // Both filters are by the same property and that property is not
  // multi-value, so we don't need to worry about de-duping results.
  function(results, callback) {
    var entities = [];
    if (results[0][0].batch.entityResults) {
      entities.push.apply(entities, results[0][0].batch.entityResults);
    }
    if (results[1][0].batch.entityResults) {
      entities.push.apply(entities, results[1][0].batch.entityResults);
    }
    callback(null, entities);
  }], callback);

# Get the people who are too short.
req = datastore.RunQueryRequest()
query = req.query
query.kind.add().name = 'Person'

too_short_filter = query.filter.property_filter
too_short_filter.property.name = 'height'
too_short_filter.operator = datastore.PropertyFilter.LESS_THAN
too_short_filter.value.integer_value = min_height

entity_results = []
resp = self.datastore.run_query(req)
entity_results.extend(resp.batch.entity_result)

# Now get the people who are too tall and add them to the result list.
req = datastore.RunQueryRequest()
query = req.query
query.kind.add().name = 'Person'

too_tall_filter = query.filter.property_filter
too_tall_filter.property.name = 'height'
too_tall_filter.operator = datastore.PropertyFilter.GREATER_THAN
too_tall_filter.value.integer_value = max_height

# Both filters are by the same property and that property is not
# multi-value, so we don't need to worry about de-duping results.
resp = self.datastore.run_query(req)
entity_results.extend(resp.batch.entity_result)

// Get the people who are too short.
Filter tooShortFilter = makeFilter(
    "height", PropertyFilter.Operator.LESS_THAN, makeValue(minHeight)).build();
Query.Builder q = Query.newBuilder();
q.addKindBuilder().setName("Person");
q.setFilter(tooShortFilter).build();

RunQueryRequest request = RunQueryRequest.newBuilder().setQuery(q).build();
RunQueryResponse response = datastore.runQuery(request);
List<EntityResult> results =
    new ArrayList<EntityResult>(response.getBatch().getEntityResultList());

// Now get the people who are too tall and add them to the result list.
// Both filters are by the same property and that property is not
// multi-value, so we don't need to worry about de-duping results.
Filter tooTallFilter = makeFilter(
    "height", PropertyFilter.Operator.GREATER_THAN, makeValue(maxHeight)).build();
q.setFilter(tooTallFilter).build();

request = RunQueryRequest.newBuilder().setQuery(q).build();
results.addAll(datastore.runQuery(request).getBatch().getEntityResultList());

Query structure

A query can specify an entity kind, zero or more filters, and zero or more sort orders.

Filters

A query's filters set constraints on the properties, keys, and ancestors of the entities to be retrieved.

Property filters

A property filter specifies

• A property name
• A comparison operator
• A property value

var heightMinQuery = {
  kinds: [{ name: 'Person' }],
  filter: {
    propertyFilter: {
      property: { name: 'height' },
      operator: 'GREATER_THAN_OR_EQUAL',
      value: { integerValue: minHeight }
    }
  }
}; 

query = datastore.Query()
query.kind.add().name = 'Person'

min_height_filter = query.filter.property_filter
min_height_filter.property.name = 'height'
min_height_filter.operator = datastore.PropertyFilter.GREATER_THAN_OR_EQUAL
min_height_filter.value.integer_value = min_height 

Query.Builder q = Query.newBuilder();
q.addKindBuilder().setName("Person");
q.setFilter(makeFilter(
    "height", PropertyFilter.Operator.GREATER_THAN_OR_EQUAL, makeValue(minHeight))); 

The property value must be supplied by the application; it cannot refer to or be calculated in terms of other properties. An entity satisfies the filter if it has a property of the given name whose value compares to the value specified in the filter in the manner described by the comparison operator.

The comparison operator can be any of the following:

Key filters

To filter on the value of an entity's key, use the special property __key__:

var keyFilterQuery = {
  kinds: [{ name: 'Person' }],
  filter: {
    propertyFilter: {
      property: { name: '__key__' },
      operator: 'GREATER_THAN',
      value: { keyValue: lastSeenKey }
    }
  }
};

query = datastore.Query()
query.kind.add().name = 'Person'

key_filter = query.filter.property_filter
key_filter.property.name = '__key__'
key_filter.operator = datastore.PropertyFilter.GREATER_THAN
key_filter.value.key_value.CopyFrom(last_seen_key)

Query.Builder q = Query.newBuilder();
q.addKindBuilder().setName("Person");
q.setFilter(makeFilter(
    "__key__", PropertyFilter.Operator.GREATER_THAN, makeValue(lastSeenKey)));

When comparing for inequality, keys are ordered by the following criteria, in order:

1. Ancestor path
2. Entity kind
3. Identifier (key name or numeric ID)

Elements of the ancestor path are compared similarly: by kind (string), then by key name or numeric ID. Kinds and key names are strings and are ordered by byte value; numeric IDs are integers and are ordered numerically. If entities with the same parent and kind use a mix of key name strings and numeric IDs, those with numeric IDs precede those with key names.

Queries on keys use indexes just like queries on properties and require custom indexes in the same cases, with a couple of exceptions: inequality filters or an ascending sort order on the key do not require a custom index, but a descending sort order on the key does. As with all queries, the development server creates appropriate entries in the index configuration file when a query that needs a custom index is tested.

Ancestor filters

You can filter your Datastore queries to a specified ancestor, so that the results returned will include only entities descended from that ancestor:

var ancestorQuery = {
  kinds: [{ name: 'Person' }],
  filter: {
    propertyFilter: {
      property: { name: '__key__' },
      operator: 'HAS_ANCESTOR',
      value: { keyValue: ancestorKey }
    }
  }
};

query = datastore.Query()
query.kind.add().name = 'Person'

key_filter = query.filter.property_filter
key_filter.property.name = '__key__'
key_filter.operator = datastore.PropertyFilter.HAS_ANCESTOR
key_filter.value.key_value.CopyFrom(ancestor_key)

Query.Builder q = Query.newBuilder();
q.addKindBuilder().setName("Person");
q.setFilter(makeFilter(
    "__key__", PropertyFilter.Operator.HAS_ANCESTOR, makeValue(ancestorKey)));

Sort orders

A query sort order specifies

• A property name
• A sort direction (ascending or descending)

For example:

// Order alphabetically by last name:
var ascendingQuery = {
  kinds: [{ name: 'Person' }],
  order: [{ property: { name: 'lastName', direction: 'ASCENDING' } }]
};

// Order by height, tallest to shortest:
var descendingQuery = {
  kinds: [{ name: 'Person' }],
  order: [{ property: { name: 'height', direction: 'DESCENDING' } }]
}; 

# Order alphabetically by last name:
last_name_asc = datastore.Query()
last_name_asc.kind.add().name = 'Person'
order = last_name_asc.order.add()
order.property.name = 'last_name'
order.direction = datastore.PropertyOrder.ASCENDING

# Order by height, tallest to shortest:
height_desc = datastore.Query()
height_desc.kind.add().name = 'Person'
order = height_desc.order.add()
order.property.name = 'height'
order.direction = datastore.PropertyOrder.DESCENDING 

// Order alphabetically by last name:
Query.Builder lastNameAsc = Query.newBuilder();
lastNameAsc.addKindBuilder().setName("Person");
lastNameAsc.addOrder(makeOrder("lastName", PropertyOrder.Direction.ASCENDING));

// Order by height, tallest to shortest:
Query.Builder heightDesc = Query.newBuilder();
heightDesc.addKindBuilder().setName("Person");
heightDesc.addOrder(makeOrder("height", PropertyOrder.Direction.DESCENDING)); 

If a query includes multiple sort orders, they are applied in the sequence specified. The following example sorts first by ascending last name and then by descending height:

var multiSortedQuery = {
  kinds: [{ name: 'Person' }],
  order: [{ property: { name: 'lastName', direction: 'ASCENDING' } },
          { property: { name: 'height', direction: 'DESCENDING' } }]
}; 

multi_sorted = datastore.Query()
multi_sorted.kind.add().name = 'Person'

order = multi_sorted.order.add()
order.property.name = 'last_name'
order.direction = datastore.PropertyOrder.ASCENDING

order = multi_sorted.order.add()
order.property.name = 'height'
order.direction = datastore.PropertyOrder.DESCENDING 

Query.Builder q = Query.newBuilder();
q.addKindBuilder().setName("Person");
q.addOrder(makeOrder("lastName", PropertyOrder.Direction.ASCENDING));
q.addOrder(makeOrder("height", PropertyOrder.Direction.DESCENDING)); 

If no sort orders are specified, the results are returned in the order they are retrieved from the Datastore.

Note: Because of the way the Datastore executes queries, if a query specifies inequality filters on a property and sort orders on other properties, the property used in the inequality filters must be ordered before the other properties.

Special query types

Some specific types of query deserve special mention:

Kindless queries

A query with no kind and no ancestor filter retrieves all of the entities of an application from the Datastore. Such kindless queries cannot include filters or sort orders on property values. They can, however, filter on entity keys by specifying __key__ as the property name:

var kindlessQuery = {
  filter: {
    propertyFilter: {
      property: { name: '__key__' },
      operator: 'GREATER_THAN',
      value: { keyValue: lastSeenKey }
    }
  }
};

query = datastore.Query()

key_filter = query.filter.property_filter
key_filter.property.name = '__key__'
key_filter.operator = datastore.PropertyFilter.GREATER_THAN
key_filter.value.key_value.CopyFrom(last_seen_key)

Query.Builder q = Query.newBuilder();
q.setFilter(makeFilter(
    "__key__", PropertyFilter.Operator.GREATER_THAN, makeValue(lastSeenKey)));

Ancestor queries

A query with an ancestor filter limits its results to the specified entity and its descendants:

var async = require('async');

var tom = { key: { path: [{ kind: 'Person', name: 'Tom' }] } };
var tomKey = tom.key;
var tomGroup = tom.key.path[0];

async.waterfall([
  function(callback) {
    var weddingPhoto = {
      key: { path: [tomGroup, { kind: 'Photo' }] },
      properties: {
        imageURL: { stringValue: 'http://domain.com/some/path/to/wedding_photo.jpg' }
      }
    };
    var babyPhoto = {
      key: { path: [tomGroup, { kind: 'Photo' }] },
      properties: {
        imageURL: { stringValue: 'http://domain.com/some/path/to/baby_photo.jpg' }
      }
    };
    var dancePhoto = {
      key: { path: [tomGroup, { kind: 'Photo' }] },
      properties: {
        imageURL: { stringValue: 'http://domain.com/some/path/to/dance_photo.jpg' }
      }
    };
    // Not a child of tom!
    var campingPhoto = {
      key: { path: [{ kind: 'Photo' }] },
      properties: {
        imageURL: { stringValue: 'http://domain.com/some/path/to/wedding_photo.jpg' }
      }
    };
    datastore.commit({
      mutation: {
        insert: [tom],
        insertAutoId: [weddingPhoto, babyPhoto, dancePhoto, campingPhoto]
      },
      mode: 'NON_TRANSACTIONAL'
    }).execute(callback);
  },
  function(result, response, callback) {
    // This returns weddingPhto, babyPhoto, and dancePhoto,
    // but not campingPhoto, because tom is not an ancestor.
    datastore.runQuery({
      query: {
        kinds: [{ name: 'Photo' }],
        filter: {
          propertyFilter: {
            property: { name: '__key__' },
            operator: 'HAS_ANCESTOR',
            value: { keyValue: tomKey }
          }
        }
      }
    }).execute(callback);
  }
], callback);

req = datastore.CommitRequest()
req.mode = datastore.CommitRequest.NON_TRANSACTIONAL

tom = req.mutation.insert.add()
path_element = tom.key.path_element.add()
path_element.kind = 'Person'
path_element.name = 'Tom'

wedding_photo = req.mutation.insert_auto_id.add()
wedding_photo.key.path_element.extend(tom.key.path_element)
wedding_photo.key.path_element.add().kind = 'Photo'
photo_url_property = wedding_photo.property.add()
photo_url_property.name = 'image_url'
photo_url_property.value.string_value = self.Url('wedding_photo.jpg')

baby_photo = req.mutation.insert_auto_id.add()
baby_photo.key.path_element.extend(tom.key.path_element)
baby_photo.key.path_element.add().kind = 'Photo'
photo_url_property = baby_photo.property.add()
photo_url_property.name = 'image_url'
photo_url_property.value.string_value = self.Url('baby_photo.jpg')

dance_photo = req.mutation.insert_auto_id.add()
dance_photo.key.path_element.extend(tom.key.path_element)
dance_photo.key.path_element.add().kind = 'Photo'
photo_url_property = dance_photo.property.add()
photo_url_property.name = 'image_url'
photo_url_property.value.string_value = self.Url('dance_photo.jpg')

camping_photo = req.mutation.insert_auto_id.add()
camping_photo.key.path_element.add().kind = 'Photo'
photo_url_property = camping_photo.property.add()
photo_url_property.name = 'image_url'
photo_url_property.value.string_value = self.Url('camping_photo.jpg')

self.datastore.commit(req)

req = datastore.RunQueryRequest()
query = req.query

query.kind.add().name = 'Photo'

key_filter = query.filter.property_filter
key_filter.property.name = '__key__'
key_filter.operator = datastore.PropertyFilter.HAS_ANCESTOR
key_filter.value.key_value.CopyFrom(tom.key)

# This returns wedding_photo, baby_photo, and dance_photo, but not
# camping_photo, because tom is not an ancestor.
resp = self.datastore.run_query(req)
results = resp.batch.entity_result

Entity tom = Entity.newBuilder().setKey(makeKey("Person", "Tom")).build();
Key tomKey = tom.getKey();

Entity.Builder weddingPhoto = Entity.newBuilder().setKey(makeKey(tomKey, "Photo"));
weddingPhoto.addProperty(makeProperty(
    "imageURL", makeValue("http://domain.com/some/path/to/wedding_photo.jpg").build()));

Entity.Builder babyPhoto = Entity.newBuilder().setKey(makeKey(tomKey, "Photo"));
babyPhoto.addProperty(makeProperty(
    "imageURL", makeValue("http://domain.com/some/path/to/baby_photo.jpg").build()));

Entity.Builder dancePhoto = Entity.newBuilder().setKey(makeKey(tomKey, "Photo"));
dancePhoto.addProperty(makeProperty(
    "imageURL", makeValue("http://domain.com/some/path/to/dance_photo.jpg").build()));

// not a child of tom!
Entity.Builder campingPhoto = Entity.newBuilder().setKey(makeKey("Photo"));
campingPhoto.addProperty(makeProperty(
    "imageURL", makeValue("http://domain.com/some/path/to/camping_photo.jpg").build()));

CommitRequest commitRequest = CommitRequest.newBuilder()
    .setMode(CommitRequest.Mode.NON_TRANSACTIONAL)
    .setMutation(Mutation.newBuilder()
        .addInsertAutoId(weddingPhoto)
        .addInsertAutoId(babyPhoto)
        .addInsertAutoId(dancePhoto)
        .addInsertAutoId(campingPhoto)
        .addInsert(tom))
    .build();
datastore.commit(commitRequest);

Query.Builder q = Query.newBuilder();
q.addKindBuilder().setName("Photo");
q.setFilter(makeFilter(
    "__key__", PropertyFilter.Operator.HAS_ANCESTOR, makeValue(tomKey)));

RunQueryRequest request = RunQueryRequest.newBuilder().setQuery(q).build();
RunQueryResponse response = datastore.runQuery(request);
// This returns weddingPhto, babyPhoto, and dancePhoto,
// but not campingPhoto, because tom is not an ancestor.
List<EntityResult> results = response.getBatch().getEntityResultList();

Kindless ancestor queries

A kindless query that includes an ancestor filter will retrieve the specified ancestor and all of its descendants, regardless of kind. This type of query does not require custom indexes. Like all kindless queries, it cannot include filters or sort orders on property values, but can filter on the entity's key:

var kindlessAncestorQuery = {
  compositeFilter: {
    operator: 'AND',
    filters: [{
      propertyFilter: {
        property: { name: '__key__' },
        operator: 'HAS_ANCESTOR',
        value: { keyValue: { path: [{ kind: 'Person', id: '33' }] } }
      }
    }, {
      propertyFilter: {
        property: { name: '__key__' },
        operator: 'GREATER_THAN',
        value: { keyValue: { path: [{ kind: 'Person', id: '33' }] } }
      }
    }]
  }
};

query = datastore.Query()
composite_filter = query.filter.composite_filter
composite_filter.operator = datastore.CompositeFilter.AND

ancestor_filter = composite_filter.filter.add().property_filter
ancestor_filter.property.name = '__key__'
ancestor_filter.operator = datastore.PropertyFilter.HAS_ANCESTOR
ancestor_filter.value.key_value.CopyFrom(ancestor_key)

# By default, ancestor queries include the specified ancestor itself.
# The following filter excludes the ancestor from the query results.
key_filter = composite_filter.filter.add().property_filter
key_filter.property.name = '__key__'
key_filter.operator = datastore.PropertyFilter.GREATER_THAN
key_filter.value.key_value.CopyFrom(last_seen_key)

Query.Builder q = Query.newBuilder();
Filter ancestorFilter = makeFilter(
    "__key__", PropertyFilter.Operator.HAS_ANCESTOR, makeValue(ancestorKey)).build();

Filter keyFilter = makeFilter(
    "__key__", PropertyFilter.Operator.GREATER_THAN, makeValue(lastSeenKey)).build();

q.setFilter(makeFilter(ancestorFilter, keyFilter));

The following example illustrates how to retrieve all entities descended from a given ancestor:

var async = require('async');

var tom = { key: { path: [{ kind: 'Person', name: 'Tom' }] } };
var tomGroup = tom.key.path[0];

async.waterfall([
  function(callback) {
    var weddingPhoto = {
      key: { path: [tomGroup, { kind: 'Photo' }] },
      properties: {
        imageURL: { stringValue: 'http://domain.com/some/path/to/wedding_photo.jpg' }
      }
    };
    var weddingVideo = {
      key: { path: [tomGroup, { kind: 'Video' }] },
      properties: {
        movieURL: { stringValue: 'http://domain.com/some/path/to/wedding_video.avi' }
      }
    };
    datastore.commit({
      mutation: {
        insert: [tom],
        insertAutoId: [weddingPhoto, weddingVideo]
      },
      mode: 'NON_TRANSACTIONAL'
    }).execute(callback);
  },
  function(result, response, callback) {
    datastore.runQuery({
      query: {
        filter: {
          compositeFilter: {
            operator: 'AND',
            filters: [{
              propertyFilter: {
                property: { name: '__key__' },
                operator: 'HAS_ANCESTOR',
                value: { keyValue: { path: [{ kind: 'Person', name: 'Tom' }] } }
              }
            }, {
              // By default, ancestor queries include the specified ancestor entity in
              // the query result.  The following filter excludes the ancestor entity
              // from the query results.
              propertyFilter: {
                property: { name: '__key__' },
                operator: 'GREATER_THAN',
                value: { keyValue: { path: [{ kind: 'Person', name: 'Tom' }] } }
              }
            }]
          }
        }
      }
    }).execute(callback);
  }
], callback);

req = datastore.CommitRequest()
req.mode = datastore.CommitRequest.NON_TRANSACTIONAL

tom = req.mutation.insert.add()
path_element = tom.key.path_element.add()
path_element.kind = 'Person'
path_element.name = 'Tom'

wedding_photo = req.mutation.insert_auto_id.add()
wedding_photo.key.path_element.extend(tom.key.path_element)
wedding_photo.key.path_element.add().kind = 'Photo'
photo_url_property = wedding_photo.property.add()
photo_url_property.name = 'image_url'
photo_url_property.value.string_value = self.Url('wedding_photo.jpg')

wedding_video = req.mutation.insert_auto_id.add()
wedding_video.key.path_element.extend(tom.key.path_element)
wedding_video.key.path_element.add().kind = 'Video'
photo_url_property = wedding_video.property.add()
photo_url_property.name = 'movie_url'
photo_url_property.value.string_value = self.Url('wedding_video.avi')

self.datastore.commit(req)

req = datastore.RunQueryRequest()
query = req.query
composite_filter = query.filter.composite_filter
composite_filter.operator = datastore.CompositeFilter.AND

ancestor_filter = composite_filter.filter.add().property_filter
ancestor_filter.property.name = '__key__'
ancestor_filter.operator = datastore.PropertyFilter.HAS_ANCESTOR
ancestor_filter.value.key_value.CopyFrom(tom.key)

# By default, ancestor queries include the specified ancestor itself.
# The following filter excludes the ancestor from the query results.
key_filter = composite_filter.filter.add().property_filter
key_filter.property.name = '__key__'
key_filter.operator = datastore.PropertyFilter.GREATER_THAN
key_filter.value.key_value.CopyFrom(tom.key)

results = self.datastore.run_query(req).batch.entity_result

Entity tom = Entity.newBuilder().setKey(makeKey("Person", "Tom")).build();
Key tomKey = tom.getKey();

Entity.Builder weddingPhoto = Entity.newBuilder().setKey(makeKey(tomKey, "Photo"));
weddingPhoto.addProperty(makeProperty(
    "imageURL", makeValue("http://domain.com/some/path/to/wedding_photo.jpg").build()));

Entity.Builder weddingVideo = Entity.newBuilder().setKey(makeKey(tomKey, "Video"));
weddingVideo.addProperty(makeProperty(
    "movieURL", makeValue("http://domain.com/some/path/to/wedding_movie.avi").build()));

CommitRequest commitRequest = CommitRequest.newBuilder()
    .setMode(CommitRequest.Mode.NON_TRANSACTIONAL)
    .setMutation(Mutation.newBuilder()
        .addInsertAutoId(weddingPhoto)
        .addInsertAutoId(weddingVideo)
        .addInsert(tom))
    .build();
datastore.commit(commitRequest);

Query.Builder q = Query.newBuilder();
Filter ancestorFilter = makeFilter(
    "__key__", PropertyFilter.Operator.HAS_ANCESTOR, makeValue(tomKey)).build();

// By default, ancestor queries include the specified ancestor itself.
// The following filter excludes the ancestor from the query results.
Filter keyFilter = makeFilter(
    "__key__", PropertyFilter.Operator.GREATER_THAN, makeValue(tomKey)).build();

q.setFilter(makeFilter(ancestorFilter, keyFilter));

RunQueryRequest request = RunQueryRequest.newBuilder().setQuery(q).build();
RunQueryResponse response = datastore.runQuery(request);

// Returns both weddingPhoto and weddingVideo,
// even though they are of different entity kinds
List<EntityResult> results = response.getBatch().getEntityResultList();

Keys-only queries

A keys-only query returns just the keys of the result entities instead of the entities themselves, at lower latency and cost than retrieving entire entities:

var keysOnlyQuery = {
  kinds: [{ name: 'Person' }],
  projection: [{ property: { name: '__key__' } }],
};
datastore.runQuery({
  query: keysOnlyQuery
}).execute(function(err, result) {
  var keys = [];
  if (!err && result.batch.entityResults) {
    result.batch.entityResults.forEach(function(er) {
      // Only the key is populated on each entity.
      keys.push(er.entity.key);
    });
  }
  callback(err, keys);
}); 

req = datastore.RunQueryRequest()
query = req.query
query.kind.add().name = 'Person'

query.projection.add().property.name = '__key__'

result_keys = []
results = self.datastore.run_query(req).batch.entity_result
for entity_result in results:
  result_keys.append(entity_result.entity.key) 

Query.Builder q = Query.newBuilder();
q.addKindBuilder().setName("Person");
q.addProjection(PropertyExpression.newBuilder().setProperty(
    PropertyReference.newBuilder().setName("__key__")));

RunQueryRequest request = RunQueryRequest.newBuilder().setQuery(q).build();
List<Key> resultKeys = new ArrayList<Key>();
for (EntityResult result : datastore.runQuery(request).getBatch().getEntityResultList()) {
  // Only the Key is populated on each entity
  resultKeys.add(result.getEntity().getKey());
} 

It is often more economical to do a keys-only query first, and then fetch a subset of entities from the results, rather than executing a general query which may fetch more entities than you actually need.

Projection queries

Sometimes all you really need from the results of a query are the values of a few specific properties. In such cases, you can use a projection query to retrieve just the properties you're actually interested in, at lower latency and cost than retrieving the entire entity; see the Projection Queries page for details.

Restrictions on queries

The nature of the index query mechanism imposes certain restrictions on what a query can do:

Entities lacking a property named in the query are ignored

Entities of the same kind need not have the same properties. To be eligible as a query result, an entity must possess a value (possibly null) for every property named in the query's filters and sort orders. If not, the entity is omitted from the indexes used to execute the query and consequently will not be included in the query's results.

Filtering on unindexed properties returns no results

A query can't find property values that aren't indexed, nor can it sort on such properties. See the Datastore Indexes page for a detailed discussion of unindexed properties.

Inequality filters are limited to at most one property

To avoid having to scan the entire index table, the query mechanism relies on all of a query's potential results being adjacent to one another in the index. To satisfy this constraint, a single query may not use inequality comparisons (LESS_THAN, LESS_THAN_OR_EQUAL, GREATER_THAN, GREATER_THAN_OR_EQUAL) on more than one property across all of its filters. For example, the following query is valid, because both inequality filters apply to the same property:

var legalInequalityFilter = {
  kinds: [{ name: 'Person' }],
  filter: {
    compositeFilter: {
      operator: 'AND',
      filters: [{
        propertyFilter: {
          property: { name: 'birthYear' },
          operator: 'GREATER_THAN_OR_EQUAL',
          value: { integerValue: minBirthYear }
        }
      }, {
        propertyFilter: {
          property: { name: 'birthYear' },
          operator: 'LESS_THAN_OR_EQUAL',
          value: { integerValue: maxBirthYear }
        }
      }]
    }
  }
};

query = datastore.Query()
query.kind.add().name = 'Person'

composite_filter = query.filter.composite_filter
composite_filter.operator = datastore.CompositeFilter.AND

birth_year_min_filter = composite_filter.filter.add().property_filter
birth_year_min_filter.property.name = 'birth_year'
birth_year_min_filter.operator = datastore.PropertyFilter.GREATER_THAN_OR_EQUAL
birth_year_min_filter.value.integer_value = min_birth_year

birth_year_max_filter = composite_filter.filter.add().property_filter
birth_year_max_filter.property.name = 'birth_year'
birth_year_max_filter.operator = datastore.PropertyFilter.LESS_THAN_OR_EQUAL
birth_year_max_filter.value.integer_value = max_birth_year

Filter birthYearMinFilter = makeFilter("birthYear",
    PropertyFilter.Operator.GREATER_THAN_OR_EQUAL,
    makeValue(minBirthYear)).build();

Filter birthYearMaxFilter = makeFilter("birthYear",
    PropertyFilter.Operator.LESS_THAN_OR_EQUAL,
    makeValue(maxBirthYear)).build();

Filter birthYearRangeFilter = makeFilter(birthYearMinFilter, birthYearMaxFilter).build();

Query.Builder q = Query.newBuilder();
q.addKindBuilder().setName("Person");
q.setFilter(birthYearRangeFilter);

However, this query is not valid, because it uses inequality filters on two different properties:

var illegalInequalityFilter = {
  kinds: [{ name: 'Person' }],
  filter: {
    compositeFilter: {
      operator: 'AND',
      filters: [{
        propertyFilter: {
          property: { name: 'birthYear' },
          operator: 'GREATER_THAN_OR_EQUAL',
          value: { integerValue: minBirthYear }
        }
      }, {
        propertyFilter: {
          property: { name: 'height' },
          operator: 'LESS_THAN_OR_EQUAL',
          value: { integerValue: maxHeight }
        }
      }]
    }
  }
};

query = datastore.Query()
query.kind.add().name = 'Person'

composite_filter = query.filter.composite_filter
composite_filter.operator = datastore.CompositeFilter.AND

birth_year_min_filter = composite_filter.filter.add().property_filter
birth_year_min_filter.property.name = 'birth_year'
birth_year_min_filter.operator = datastore.PropertyFilter.GREATER_THAN_OR_EQUAL
birth_year_min_filter.value.integer_value = min_birth_year

height_max_filter = composite_filter.filter.add().property_filter
height_max_filter.property.name = 'height'
height_max_filter.operator = datastore.PropertyFilter.LESS_THAN_OR_EQUAL
height_max_filter.value.integer_value = max_height

Filter birthYearMinFilter = makeFilter("birthYear",
    PropertyFilter.Operator.GREATER_THAN_OR_EQUAL,
    makeValue(minBirthYear)).build();

Filter heightMaxFilter = makeFilter("height",
    PropertyFilter.Operator.LESS_THAN_OR_EQUAL,
    makeValue(maxHeight)).build();

Filter invalidFilter = makeFilter(birthYearMinFilter, heightMaxFilter).build();

Query.Builder q = Query.newBuilder();
q.addKindBuilder().setName("Person");
q.setFilter(invalidFilter);

Note that a query can combine equality (EQUAL) filters for different properties, along with one or more inequality filters on a single property. Thus the following is a valid query:

var mixedFilter = {
  kinds: [{ name: 'Person' }],
  filter: {
    compositeFilter: {
      operator: 'AND',
      filters: [{
        propertyFilter: {
          property: { name: 'lastName' },
          operator: 'EQUAL',
          value: { stringValue: targetLastName }
        }
      }, {
        propertyFilter: {
          property: { name: 'city' },
          operator: 'EQUAL',
          value: { stringValue: targetCity }
        }
      }, {
        propertyFilter: {
          property: { name: 'birthYear' },
          operator: 'GREATER_THAN_OR_EQUAL',
          value: { integerValue: minBirthYear }
        }
      }, {
        propertyFilter: {
          property: { name: 'birthYear' },
          operator: 'LESS_THAN_OR_EQUAL',
          value: { integerValue: maxBirthYear }
        }
      }]
    }
  }
};

query = datastore.Query()
query.kind.add().name = 'Person'

composite_filter = query.filter.composite_filter
composite_filter.operator = datastore.CompositeFilter.AND

last_name_filter = composite_filter.filter.add().property_filter
last_name_filter.property.name = 'last_name'
last_name_filter.operator = datastore.PropertyFilter.EQUAL
last_name_filter.value.string_value = target_last_name

city_filter = composite_filter.filter.add().property_filter
city_filter.property.name = 'city'
city_filter.operator = datastore.PropertyFilter.EQUAL
city_filter.value.string_value = target_city

birth_year_min_filter = composite_filter.filter.add().property_filter
birth_year_min_filter.property.name = 'birth_year'
birth_year_min_filter.operator = datastore.PropertyFilter.GREATER_THAN_OR_EQUAL
birth_year_min_filter.value.integer_value = min_birth_year

birth_year_max_filter = composite_filter.filter.add().property_filter
birth_year_max_filter.property.name = 'birth_year'
birth_year_max_filter.operator = datastore.PropertyFilter.GREATER_THAN_OR_EQUAL
birth_year_max_filter.value.integer_value = max_birth_year

Filter lastNameFilter = makeFilter("lastName",
    PropertyFilter.Operator.EQUAL,
    makeValue(targetLastName)).build();

Filter cityFilter = makeFilter("city",
    PropertyFilter.Operator.EQUAL,
    makeValue(targetCity)).build();

Filter birthYearMinFilter = makeFilter("birthYear",
    PropertyFilter.Operator.GREATER_THAN_OR_EQUAL,
    makeValue(minBirthYear)).build();

Filter birthYearMaxFilter = makeFilter("birthYear",
    PropertyFilter.Operator.LESS_THAN_OR_EQUAL,
    makeValue(maxBirthYear)).build();

Filter validFilter =
    makeFilter(lastNameFilter, cityFilter, birthYearMinFilter, birthYearMaxFilter).build();

Query.Builder q = Query.newBuilder();
q.addKindBuilder().setName("Person");
q.setFilter(validFilter);

Ordering of query results is undefined when no sort order is specified

When a query does not specify a sort order, the results are returned in the order they are retrieved. As the Datastore implementation evolves (or if a dataset's indexes change), this order may change. Therefore, if your application requires its query results in a particular order, be sure to specify that sort order explicitly in the query.

Sort orders are ignored on properties with equality filters

Queries that include an equality filter for a given property ignore any sort order specified for that property. This is a simple optimization to save needless processing for single-valued properties, since all results have the same value for the property and so no further sorting is needed. Multiple-valued properties, however, may have additional values besides the one matched by the equality filter. Because this use case is rare and applying the sort order would be expensive and require extra indexes, the Datastore query planner simply ignores the sort order even in the multiple-valued case. This may cause query results to be returned in a different order than the sort order appears to imply.

Properties used in inequality filters must be sorted first

To retrieve all results that match an inequality filter, a query scans the index table for the first row matching the filter, then scans forward until it encounters a nonmatching row. For the consecutive rows to encompass the complete result set, they must be ordered by the property used in the inequality filter before any other properties. Thus if a query specifies one or more inequality filters along with one or more sort orders, the first sort order must refer to the same property named in the inequality filters. The following is a valid query:

var sortedInequalityFilter = {
  kinds: [{ name: 'Person' }],
  filter: {
    propertyFilter: {
      property: { name: 'birthYear' },
      operator: 'GREATER_THAN_OR_EQUAL',
      value: { integerValue: minBirthYear }
    }
  },
  order: [{ property: { name: 'birthYear', direction: 'ASCENDING' } },
          { property: { name: 'lastName', direction: 'ASCENDING' } }]
};

query = datastore.Query()
query.kind.add().name = 'Person'

birth_year_min_filter = query.filter.property_filter
birth_year_min_filter.property.name = 'birth_year'
birth_year_min_filter.operator = datastore.PropertyFilter.GREATER_THAN_OR_EQUAL
birth_year_min_filter.value.integer_value = min_birth_year

order = query.order.add()
order.property.name = 'birth_year'
order.direction = datastore.PropertyOrder.ASCENDING

order = query.order.add()
order.property.name = 'last_name'
order.direction = datastore.PropertyOrder.ASCENDING

Filter birthYearMinFilter = makeFilter("birthYear",
    PropertyFilter.Operator.GREATER_THAN_OR_EQUAL,
    makeValue(minBirthYear)).build();
Query.Builder valid = Query.newBuilder();
valid.addKindBuilder().setName("Person");
valid.setFilter(birthYearMinFilter);
valid.addOrder(makeOrder("birthYear", PropertyOrder.Direction.ASCENDING));
valid.addOrder(makeOrder("lastName", PropertyOrder.Direction.ASCENDING));

This query is not valid, because it doesn't sort on the property used in the inequality filter:

var sortedInequalityFilter = {
  kinds: [{ name: 'Person' }],
  filter: {
    propertyFilter: {
      property: { name: 'birthYear' },
      operator: 'GREATER_THAN_OR_EQUAL',
      value: { integerValue: minBirthYear }
    }
  },
  order: [{ property: { name: 'lastName', direction: 'ASCENDING' } }]
};

query = datastore.Query()
query.kind.add().name = 'Person'

birth_year_min_filter = query.filter.property_filter
birth_year_min_filter.property.name = 'birth_year'
birth_year_min_filter.operator = datastore.PropertyFilter.GREATER_THAN_OR_EQUAL
birth_year_min_filter.value.integer_value = min_birth_year

order = query.order.add()
order.property.name = 'last_name'
order.direction = datastore.PropertyOrder.ASCENDING

Filter birthYearMinFilter = makeFilter("birthYear",
    PropertyFilter.Operator.GREATER_THAN_OR_EQUAL,
    makeValue(minBirthYear)).build();
Query.Builder invalid = Query.newBuilder();
invalid.addKindBuilder().setName("Person");
invalid.setFilter(birthYearMinFilter);
invalid.addOrder(makeOrder("lastName", PropertyOrder.Direction.ASCENDING));

Similarly, this query is not valid because the property used in the inequality filter is not the first one sorted:

var sortedInequalityFilter = {
  kinds: [{ name: 'Person' }],
  filter: {
    propertyFilter: {
      property: { name: 'birthYear' },
      operator: 'GREATER_THAN_OR_EQUAL',
      value: { integerValue: minBirthYear }
    }
  },
  order: [{ property: { name: 'lastName', direction: 'ASCENDING' } },
          { property: { name: 'birthYear', direction: 'ASCENDING' } },]
};

query = datastore.Query()
query.kind.add().name = 'Person'

birth_year_min_filter = query.filter.property_filter
birth_year_min_filter.property.name = 'birth_year'
birth_year_min_filter.operator = datastore.PropertyFilter.GREATER_THAN_OR_EQUAL
birth_year_min_filter.value.integer_value = min_birth_year

last_name_order = query.order.add()
last_name_order.property.name = 'last_name'
last_name_order.direction = datastore.PropertyOrder.ASCENDING

birth_year_order = query.order.add()
birth_year_order.property.name = 'birth_year'
birth_year_order.direction = datastore.PropertyOrder.ASCENDING

Filter birthYearMinFilter = makeFilter("birthYear",
    PropertyFilter.Operator.GREATER_THAN_OR_EQUAL,
    makeValue(minBirthYear)).build();
Query.Builder invalid = Query.newBuilder();
invalid.addKindBuilder().setName("Person");
invalid.setFilter(birthYearMinFilter);
invalid.addOrder(makeOrder("lastName", PropertyOrder.Direction.ASCENDING));
invalid.addOrder(makeOrder("birthYear", PropertyOrder.Direction.ASCENDING));

Properties with multiple values can behave in surprising ways

Because of the way they're indexed, entities with multiple values for the same property can sometimes interact with query filters and sort orders in unexpected and surprising ways.

If a query has multiple inequality filters on a given property, an entity will match the query only if at least one of its individual values for the property satisfies all of the filters. For example, if an entity of kind Widget has values 1 and 2 for property x, it will not match the query:

var query = {
  kinds: [{ name: 'Widget' }],
  filter: {
    compositeFilter: {
      operator: 'AND',
      filters: [{
        propertyFilter: {
          property: { name: 'x' },
          operator: 'GREATER_THAN',
          value: { integerValue: 1 }
        }
      }, {
        propertyFilter: {
          property: { name: 'x' },
          operator: 'LESS_THAN',
          value: { integerValue: 2 }
        }
      }]
    }
  }
};

query = datastore.Query()
query.kind.add().name = 'Widget'

composite_filter = query.filter.composite_filter
composite_filter.operator = datastore.CompositeFilter.AND

x_min_filter = composite_filter.filter.add().property_filter
x_min_filter.property.name = 'x'
x_min_filter.operator = datastore.PropertyFilter.GREATER_THAN
x_min_filter.value.integer_value = 1

x_max_filter = composite_filter.filter.add().property_filter
x_max_filter.property.name = 'x'
x_max_filter.operator = datastore.PropertyFilter.LESS_THAN
x_max_filter.value.integer_value = 2

Query.Builder q = Query.newBuilder();
q.addKindBuilder().setName("Widget");
q.setFilter(makeFilter(
    makeFilter("x", PropertyFilter.Operator.GREATER_THAN, makeValue(1)).build(),
    makeFilter("x", PropertyFilter.Operator.LESS_THAN, makeValue(2)).build()));

Each of the entity's x values satisfies one of the filters, but neither single value satisfies both. Note that this does not apply to equality filters. For example, the same entity will satisfy the query

var query = {
  kinds: [{ name: 'Widget' }],
  filter: {
    compositeFilter: {
      operator: 'AND',
      filters: [{
        propertyFilter: {
          property: { name: 'x' },
          operator: 'EQUAL',
          value: { integerValue: 1 }
        }
      }, {
        propertyFilter: {
          property: { name: 'x' },
          operator: 'EQUAL',
          value: { integerValue: 2 }
        }
      }]
    }
  }
};

query = datastore.Query()
query.kind.add().name = 'Widget'

composite_filter = query.filter.composite_filter
composite_filter.operator = datastore.CompositeFilter.AND

x_filter1 = composite_filter.filter.add().property_filter
x_filter1.property.name = 'x'
x_filter1.operator = datastore.PropertyFilter.EQUAL
x_filter1.value.integer_value = 1

x_filter2 = composite_filter.filter.add().property_filter
x_filter2.property.name = 'x'
x_filter2.operator = datastore.PropertyFilter.EQUAL
x_filter2.value.integer_value = 2

Query.Builder q = Query.newBuilder();
q.addKindBuilder().setName("Widget");
q.setFilter(makeFilter(
    makeFilter("x", PropertyFilter.Operator.EQUAL, makeValue(1)).build(),
    makeFilter("x", PropertyFilter.Operator.EQUAL, makeValue(2)).build()));

even though neither of the entity's individual x values satisfies both filter conditions.

Similarly, the sort order for multiple-valued properties is unusual. Because such properties appear once in the index for each unique value, the first value seen in the index determines an entity's sort order:

• If the query results are sorted in ascending order, the smallest value of the property is used for ordering.
• If the results are sorted in descending order, the greatest value is used for ordering.
• Other values do not affect the sort order, nor does the number of values.

This has the unusual consequence that an entity with property values 1 and 9 precedes one with values 4, 5, 6, and 7 in both ascending and descending order.

Queries inside transactions must include ancestor filters

Datastore transactions operate only on entities belonging to the same entity group (descended from a common ancestor). To preserve this restriction, all queries performed within a transaction must include an ancestor filter specifying an ancestor in the same entity group as the other operations in the transaction.

Retrieving results

After constructing a query, you can specify a number of retrieval options to further control the results it returns.

To retrieve only selected properties of an entity rather than the entire entity, use a projection query. This type of query runs faster and costs less than one that returns complete entities.

Similarly, a keys-only query saves time and resources by returning just the keys to the entities it matches, rather than the full entities themselves. To create this type of query, add a projection on the __key__ property to your query object:

var keysOnlyQuery = {
  kinds: [{ name: 'Person' }],
  projection: [{ property: { name: '__key__' } }],
};
datastore.runQuery({
  query: keysOnlyQuery
}).execute(function(err, result) {
  var keys = [];
  if (!err && result.batch.entityResults) {
    result.batch.entityResults.forEach(function(er) {
      // Only the key is populated on each entity.
      keys.push(er.entity.key);
    });
  }
  callback(err, keys);
});

req = datastore.RunQueryRequest()
query = req.query
query.kind.add().name = 'Person'

query.projection.add().property.name = '__key__'

result_keys = []
results = self.datastore.run_query(req).batch.entity_result
for entity_result in results:
  result_keys.append(entity_result.entity.key)

Query.Builder q = Query.newBuilder();
q.addKindBuilder().setName("Person");
q.addProjection(PropertyExpression.newBuilder().setProperty(
    PropertyReference.newBuilder().setName("__key__")));

RunQueryRequest request = RunQueryRequest.newBuilder().setQuery(q).build();
List<Key> resultKeys = new ArrayList<Key>();
for (EntityResult result : datastore.runQuery(request).getBatch().getEntityResultList()) {
  // Only the Key is populated on each entity
  resultKeys.add(result.getEntity().getKey());
}

You can specify a limit for your query to control the maximum number of results returned in one batch. The following example retrieves the five tallest people from the Datastore:

function tallestPeople(callback) {
  var tallestQuery = {
    kinds: [{ name: 'Person' }],
    order: [{ property: { name: 'height', direction: 'DESCENDING' } }],
    limit: 5
  };
  datastore.runQuery({
    query: tallestQuery
  }).execute(function(err, result) {
    var tallPeople = [];
    if (!err && result.batch.entityResults) {
      result.batch.entityResults.forEach(function(er) {
        tallPeople.push(er.entity);
      });
    }
    callback(err, tallPeople);
  });
}

req = datastore.RunQueryRequest()
query = req.query
query.kind.add().name = 'Person'

order = query.order.add()
order.property.name = 'height'
order.direction = datastore.PropertyOrder.DESCENDING

query.limit = 5

self.datastore.run_query(req)

List<EntityResult> getTallestPeople() throws DatastoreException {
  Query.Builder q = Query.newBuilder();
  q.addKindBuilder().setName("Person");
  q.addOrder(makeOrder("height", PropertyOrder.Direction.DESCENDING));
  q.setLimit(5);

  RunQueryRequest request = RunQueryRequest.newBuilder().setQuery(q).build();
  return datastore.runQuery(request).getBatch().getEntityResultList();
}

Using an integer offset skips a specified number of results before returning the first one. Adding the following line in the example above would return the sixth through tenth tallest people instead of the five tallest:

q.offset = 5;

query.offset = 5

q.setOffset(5);

Query cursors

Query cursors allow an application to retrieve a query's results in convenient batches without incurring the overhead of a query offset. After performing a retrieval operation, the application can obtain a cursor, which is an opaque byte string marking the index position of the last result retrieved. The application can save this string (for instance in the Datastore, a cache, or embedded in a web page as a base-64 encoded HTTP GET or POST parameter), and can then use the cursor as the starting point for a subsequent retrieval operation to obtain the next batch of results from the point where the previous retrieval ended. A retrieval can also specify an end cursor, to limit the extent of the result set returned.

Limitations of cursors

Cursors are subject to the following limitations:

• A cursor can be used only by the same dataset that performed the original query, and only to continue the same query. To use the cursor in a subsequent retrieval operation, you must reconstitute the original query exactly, including the same entity kind, ancestor filter, property filters, and sort orders. It is not possible to retrieve results using a cursor without setting up the same query from which it was originally generated.
• Cursors don't always work as expected with a query that uses an inequality filter or a sort order on a property with multiple values. The de-duplication logic for such multiple-valued properties does not persist between retrievals, possibly causing the same result to be returned more than once.
• New Datastore releases may change internal implementation details, invalidating cursors that depend on them. If an application attempts to use a cursor that is no longer valid, the Datastore raises an exception.

Cursors and data updates

The cursor's position is defined as the location in the result list after the last result returned. A cursor is not a relative position in the list (it's not an offset); it's a marker to which the Datastore can jump when starting an index scan for results. If the results for a query change between uses of a cursor, the query notices only changes that occur in results after the cursor. If a new result appears before the cursor's position for the query, it will not be returned when the results after the cursor are fetched. Similarly, if an entity is no longer a result for a query but had appeared before the cursor, the results that appear after the cursor do not change. If the last result returned is removed from the result set, the cursor still knows how to locate the next result.

An interesting application of cursors is to monitor entities for unseen changes. If the app sets a timestamp property with the current date and time every time an entity changes, the app can use a query sorted by the timestamp property, ascending, with a Datastore cursor to check when entities are moved to the end of the result list. If an entity's timestamp is updated, the query with the cursor returns the updated entity. If no entities were updated since the last time the query was performed, no results are returned, and the cursor does not move.

When retrieving query results, you can use both a start cursor and an end cursor to return a continuous group of results from the Datastore. When using a start and end cursor to retrieve the results, you are not guaranteed that the size of the results will be the same as when you generated the cursors. Entities may be added or deleted from the Datastore between the time the cursors are generated and when they are used in a query.

The following example demonstrates the use of cursors for pagination:

function nextPage(pageSize, cursor, callback) {
  var query = {
    kinds: [{ name: 'Person' }],
    limit: pageSize
  };
  if (cursor) {
    query.startCursor = cursor;
  }
  datastore.runQuery({
    query: query
  }).execute(callback);
}

req = datastore.RunQueryRequest()
query = req.query
query.kind.add().name = 'Person'

query.limit = page_size
if cursor is not None:
  query.start_cursor.CopyFrom(cursor)

resp = self.datastore.run_query(req)

QueryResultBatch nextPage(int pageSize, ByteString cursor) throws DatastoreException {
  Query.Builder q = Query.newBuilder();
  q.addKindBuilder().setName("Person");
  q.setLimit(pageSize);
  if (cursor != null) {
    q.setStartCursor(cursor);
  }
  RunQueryRequest request = RunQueryRequest.newBuilder().setQuery(q).build();
  return datastore.runQuery(request).getBatch();
}

Data consistency

Datastore queries can deliver their results at either of two consistency levels:

• Strongly consistent queries guarantee the freshest results, but may take longer to complete.
• Eventually consistent queries generally run faster, but may occasionally return stale results.

In an eventually consistent query, the indexes used to gather the results are also accessed with eventual consistency. Consequently, such queries may sometimes return entities that no longer match the original query criteria, while strongly consistent queries are always transactionally consistent. See the article Transaction Isolation in App Engine for more information on how entities and indexes are updated.

Queries return their results with different levels of consistency guarantee, depending on the nature of the query:

• Ancestor queries (those within an entity group) are strongly consistent by default, but can instead be made eventually consistent by setting the Datastore read policy (see below).
• Non-ancestor queries are always eventually consistent.

To improve performance, you can set the Datastore read policy so that all reads and queries are eventually consistent. (The API also allows you to explicitly set a strong consistency policy, but this setting will have no practical effect, since non-ancestor queries are always eventually consistent regardless of policy.)

You can enable eventually consistent reads via the read options of the query object:

var query = {
  kinds: [{ name: 'Person' }],
  filter: {
    propertyFilter: {
      property: { name: '__key__' },
      operator: 'HAS_ANCESTOR',
      value: { keyValue: ancestorKey }
    }
  }
};
datastore.runQuery({
  query: query,
  readOptions: { readConsistency: 'EVENTUAL' }
}).execute(callback);

req = datastore.RunQueryRequest()
query = req.query
query.kind.add().name = 'Person'

ancestor_filter = query.filter.property_filter
ancestor_filter.property.name = '__key__'
ancestor_filter.operator = datastore.PropertyFilter.HAS_ANCESTOR
ancestor_filter.value.key_value.CopyFrom(ancestor_key)

req.read_options.read_consistency = datastore.ReadOptions.EVENTUAL

results = self.datastore.run_query(req).batch.entity_result

// Key ancestorKey = ...
Query.Builder q = Query.newBuilder();
q.addKindBuilder().setName("Person");
q.setFilter(makeFilter(
    "__key__", PropertyFilter.Operator.HAS_ANCESTOR, makeValue(ancestorKey)));
RunQueryRequest request = RunQueryRequest.newBuilder().setQuery(q).setReadOptions(
    ReadOptions.newBuilder().setReadConsistency(ReadOptions.ReadConsistency.EVENTUAL)).build();
List<EntityResult> result = datastore.runQuery(request).getBatch().getEntityResultList();

Except as otherwise noted, the content of this page is licensed under the Creative Commons Attribution 3.0 License, and code samples are licensed under the Apache 2.0 License. For details, see our Site Policies.