Datastore Metadata

Google Cloud Datastore provides programmatic access to some of its metadata to support metaprogramming, implementing backend administrative functions, simplify consistent caching, and similar purposes; you can use it, for instance, to build a custom datastore viewer for your application. The metadata available includes information about the entity groups, namespaces, entity kinds, and properties your application uses, as well as the property representations for each property.

The Cloud Datastore Dashboard in the Cloud Platform Console also provides some metadata about your application, but the data displayed there differs in some important respects from that returned by these functions.

  • Freshness. Reading metadata using the API gets current data, whereas data in the dashboard is updated only once daily.
  • Contents. Some metadata in the dashboard is not available via the APIs; the reverse is also true.
  • Speed. Metadata gets and queries are billed in the same way as datastore gets and queries. Metadata queries that fetch information on namespaces, kinds, and properties are generally slow to execute. As a rule of thumb, expect a metadata query that returns N entities to take about the same time as N ordinary queries each returning a single entity. Furthermore, property representation queries (non-keys-only property queries) are slower than keys-only property queries. Metadata gets of entity group metadata are somewhat faster than getting a regular entity.

Contents

Metadata queries

Three special entity kinds are reserved for metadata queries:

Entity Description
__namespace__ Used to find all the namespaces used in your application entities.
__kind__ Used to query a specific kind.
__property__ Used to query by a property of a kind.

These kinds will not conflict with others of the same names that may already exist in your application. By querying on these special kinds, you can retrieve entities containing the desired metadata.

The entities returned by metadata queries are generated dynamically, based on the current state of the Datastore. While you can create local entity objects of kinds __namespace__, __kind__, or __property__, any attempt to store them in the Datastore will fail.

Namespace queries

You can use a namespace query to find all namespaces used in the application's entities. This allows you to perform activities such as administrative functions across multiple namespaces.

Namespace queries return entities of the special kind __namespace__ whose key name is the name of a namespace. (An exception is the default namespace designated by the empty string "": since the empty string is not a valid key name, this namespace is keyed with the numeric ID 1 instead.) Queries of this type support filtering only for ranges over the special pseudoproperty __key__, whose value is the entity's key. The results can be sorted by ascending (but not descending) __key__ value. Because __namespace__ entities have no properties, both keys-only and non-keys-only queries return the same information.

The following example returns a list of an application's namespaces in the range between the values assigned to the startNamespace and endNamespace variables:

Java

Node.js

Go

Python

Kind queries

Kind queries return entities of kind __kind__ whose key name is the name of an entity kind. Queries of this type are implicitly restricted to the current namespace and support filtering only for ranges over the __key__ pseudoproperty. The results can be sorted by ascending (but not descending) __key__ value. Because __kind__ entities have no properties, both keys-only and non-keys-only queries return the same information.

The following example prints a list of the kinds used in an application:

Java

Node.js

Go

Python

Property queries

Property queries return entities of kind __property__ denoting the indexed properties associated with an entity kind. (Unindexed properties are not included.) The entity representing property p of kind k is built as follows:

  • The entity's key has kind __property__ and key name p.
  • The entity's parent key has kind __kind__ and key name k.
  • The entity's property_representation array property contains all the property's representations.

For example, if your datastore contains exactly two Task entities with name and done properties:

Key: 'Task:1'
name: 'Read some properties'
done: true

Key: 'Task:2'
name: 'Climb'
done: null

then the two entities returned by a __property__ query will be:

Key: '__kind__:Task/__property__:name'
property_representation: [ 'STRING' ]

Key: '__kind__:Task/__property__:done'
property_representation: [ 'BOOLEAN', 'NULL' ]

Property queries are implicitly restricted to the current namespace, and support limited filtering with an ancestor or a range over the __key__ pseudoproperty.

A keys-only property query is more efficient than a non-keys-only query, as it does not need to collect the property's representations. The following example retrieves the names of all of an application's entity kinds and the properties associated with each:

Java

Node.js

Go

Python

Property queries: property representations

Non-keys-only property queries, known as property representation queries, return additional information on the value types used for each property. The property_representation property in the entity representing property p of kind k is an array containing all representations of p's value in any entity of kind k.

Each value has the following representation (note that some value types share representations):

Value type Representation
Integer INT64
Floating-point number DOUBLE
Boolean BOOLEAN
Text string STRING
Byte string STRING
Date and time INT64
Datastore key REFERENCE
Embedded entity STRING
Array representation of the array's elements
Geographical point POINT
Null NULL

The following example finds all representations of properties of kind Task, using an ancestor property query:

Java

Node.js

Go

Python

Property queries: filtering

Property queries support ancestor filtering on a __kind__ or __property__ key, to limit the query results to a single kind or property, as seen in the property representation query above.

Property queries can also be filtered with a range over the pseudoproperty __key__, where the keys denote either __kind__ or __property__ entities. The results can be sorted by ascending (but not descending) __key__ value. Filtering is applied to kind-property pairs, ordered first by kind and second by property. For instance, suppose you have entities with these properties:

  • kind Task with properties
    • created
    • priority
    • tags
  • kind TaskList with properties
    • created

The following keys-only property query:

Java

Node.js

Not Applicable

Go

Not Applicable

Python

Not Applicable

will collect the following kind, property name pairs:

Task, priority
Task, tags
TaskList, created

Notice that the results include properties from kinds Task and TaskList, but do not include the created property of kind Task, because it falls outside the range specified for the query.

Send feedback about...

Cloud Datastore