Metadata

Google Cloud Datastore provides programmatic access to some of its metadata to support metaprogramming, implementing backend administrative functions, simplify consistent caching, and similar purposes; you can use it, for instance, to build a custom Cloud Datastore viewer for your application. The metadata available includes information about the entity groups, namespaces, entity kinds, and properties your application uses, as well as the property representations for each property.

The Cloud Datastore Dashboard in the Cloud Platform Console also provides some metadata about your application, but the data displayed there differs in some important respects from that returned by these functions.

  • Freshness. Reading metadata using the API gets current data, whereas data in the dashboard is updated only once daily.
  • Contents. Some metadata in the dashboard is not available via the APIs; the reverse is also true.
  • Speed. Metadata gets and queries are billed in the same way as Cloud Datastore gets and queries. Metadata queries that fetch information on namespaces, kinds, and properties are generally slow to execute. As a rule of thumb, expect a metadata query that returns N entities to take about the same time as N ordinary queries each returning a single entity. Furthermore, property representation queries (non-keys-only property queries) are slower than keys-only property queries. Metadata gets of entity group metadata are somewhat faster than getting a regular entity.

Entity group metadata

The High-Replication Datastore provides access to the "version" of an entity group, a strictly positive number that is guaranteed to increase on every change to the entity group. Entity group versions can be used, e.g., to easily write code to keep a consistent cache of a complex ancestor query on an entity group.

Entity group versions are obtained by calling get() on a special pseudo-entity that contains a strictly positive __version__ property. The pseudo-entity's key can be created using the Entities.createEntityGroupKey() method:

private static long getEntityGroupVersion(DatastoreService ds, Transaction tx, Key entityKey) {
  try {
    return Entities.getVersionProperty(ds.get(tx, Entities.createEntityGroupKey(entityKey)));
  } catch (EntityNotFoundException e) {
    // No entity group information, return a value strictly smaller than any
    // possible version
    return 0;
  }
}

private static void printEntityGroupVersions(DatastoreService ds, PrintWriter writer) {
  Entity entity1 = new Entity("Simple");
  Key key1 = ds.put(entity1);
  Key entityGroupKey = Entities.createEntityGroupKey(key1);

  // Print entity1's entity group version
  writer.println("version " + getEntityGroupVersion(ds, null, key1));

  // Write to a different entity group
  Entity entity2 = new Entity("Simple");
  ds.put(entity2);

  // Will print the same version, as entity1's entity group has not changed
  writer.println("version " + getEntityGroupVersion(ds, null, key1));

  // Change entity1's entity group by adding a new child entity
  Entity entity3 = new Entity("Simple", entity1.getKey());
  ds.put(entity3);

  // Will print a higher version, as entity1's entity group has changed
  writer.println("version " + getEntityGroupVersion(ds, null, key1));
}

This example caches query results (a count of matching results) and uses entity group versions to ensure that it only uses the cached value if current:

// A simple class for tracking consistent entity group counts.
private static class EntityGroupCount implements Serializable {
  long version; // Version of the entity group whose count we are tracking
  int count;

  EntityGroupCount(long version, int count) {
    this.version = version;
    this.count = count;
  }

  // Display count of entities in an entity group, with consistent caching
  void showEntityGroupCount(
      DatastoreService ds, MemcacheService cache, PrintWriter writer, Key entityGroupKey) {
    EntityGroupCount egCount = (EntityGroupCount) cache.get(entityGroupKey);
    // Reuses getEntityGroupVersion method from the previous example.
    if (egCount != null && egCount.version == getEntityGroupVersion(ds, null, entityGroupKey)) {
      // Cached value matched current entity group version, use that
      writer.println(egCount.count + " entities (cached)");
    } else {
      // Need to actually count entities. Using a transaction to get a consistent count
      // and entity group version.
      Transaction tx = ds.beginTransaction();
      PreparedQuery pq = ds.prepare(tx, new Query(entityGroupKey));
      int count = pq.countEntities(FetchOptions.Builder.withLimit(5000));
      cache.put(
          entityGroupKey,
          new EntityGroupCount(getEntityGroupVersion(ds, tx, entityGroupKey), count));
      tx.rollback();
      writer.println(count + " entities");
    }
  }
}

__entity_group__ entities may not exist for entity groups which have never been written to.

The entity group version is guaranteed to increase on any change to the entity group; it may also (rarely) increase in the absence of user-visible changes.

Metadata queries

The Java class Entities, defined in the com.google.appengine.api.datastore package, provides three special entity kinds that are reserved for metadata queries. They are denoted by static constants of the Entities class:

Static constant Entity kind
Entities.NAMESPACE_METADATA_KIND __namespace__
Entities.KIND_METADATA_KIND __kind__
Entities.PROPERTY_METADATA_KIND __property__

These kinds will not conflict with others of the same names that may already exist in your application. By querying on these special kinds, you can retrieve entities containing the desired metadata.

The entities returned by metadata queries are generated dynamically, based on the current state of Cloud Datastore. While you can create local `Entity` objects of kinds `__namespace__`, `__kind__`, or __property__, any attempt to store them in Cloud Datastore will fail with an IllegalArgumentException .

The easiest way to issue metadata queries is with the low-level Datastore API. The following example prints the names of all namespaces in an application:

void printAllNamespaces(DatastoreService ds, PrintWriter writer) {
  Query q = new Query(Entities.NAMESPACE_METADATA_KIND);

  for (Entity e : ds.prepare(q).asIterable()) {
    // A nonzero numeric id denotes the default namespace;
    // see <a href="#Namespace_Queries">Namespace Queries</a>, below
    if (e.getKey().getId() != 0) {
      writer.println("<default>");
    } else {
      writer.println(e.getKey().getName());
    }
  }
}

Namespace queries

If your application uses the Namespaces API, you can use a namespace query to find all namespaces used in the application's entities. This allows you to perform activities such as administrative functions across multiple namespaces.

Namespace queries return entities of the special kind __namespace__ whose key name is the name of a namespace. (An exception is the default namespace designated by the empty string "": since the empty string is not a valid key name, this namespace is keyed with the numeric ID 1 instead.) Queries of this type support filtering only for ranges over the special pseudoproperty __key__, whose value is the entity's key. The results can be sorted by ascending (but not descending) __key__ value. Because __namespace__ entities have no properties, both keys-only and non-keys-only queries return the same information.

The following example returns a list of an application's namespaces in the range between two specified names, start and end:

List<String> getNamespaces(DatastoreService ds, String start, String end) {

  // Start with unrestricted namespace query
  Query q = new Query(Entities.NAMESPACE_METADATA_KIND);
  List<Filter> subFilters = new ArrayList();
  // Limit to specified range, if any
  if (start != null) {
    subFilters.add(
        new FilterPredicate(
            Entity.KEY_RESERVED_PROPERTY,
            FilterOperator.GREATER_THAN_OR_EQUAL,
            Entities.createNamespaceKey(start)));
  }
  if (end != null) {
    subFilters.add(
        new FilterPredicate(
            Entity.KEY_RESERVED_PROPERTY,
            FilterOperator.LESS_THAN_OR_EQUAL,
            Entities.createNamespaceKey(end)));
  }

  q.setFilter(CompositeFilterOperator.and(su