Entities, Properties, and Keys

This solution is no longer recommended: This page describes how to use legacy App Engine APIs. Apps that use these APIs can only run in App Engine and will need to upgrade to the recommended APIs before migrating to the App Engine Java 11 runtime.

Data objects in Datastore are known as entities. An entity has one or more named properties, each of which can have one or more values. Entities of the same kind do not need to have the same properties, and an entity's values for a given property do not all need to be of the same data type. (If necessary, an application can establish and enforce such restrictions in its own data model.)

Datastore supports a variety of data types for property values. These include, among others:

• Integers
• Floating-point numbers
• Strings
• Dates
• Binary data

For a full list of types, see Properties and value types.

Each entity in Datastore has a key that uniquely identifies it. The key consists of the following components:

• The namespace of the entity, which allows for multitenancy
• The kind of the entity, which categorizes it for the purpose of Datastore queries
• An identifier for the individual entity, which can be either
  • a key name string
  • an integer numeric ID
• An optional ancestor path locating the entity within the Datastore hierarchy

Don't name a property "key." This name is reserved for a special property used to store the Model key. Though it may work locally, a property named "key" will prevent deployment to App Engine.

An application can fetch an individual entity from Datastore using the entity's key, or it can retrieve one or more entities by issuing a query based on the entities' keys or property values.

The Java App Engine SDK includes a simple API, provided in the package com.google.appengine.api.datastore, that supports the features of Datastore directly. All of the examples in this document are based on this low-level API; you can choose to use it either directly in your application or as a basis on which to build your own data management layer.

Datastore itself does not enforce any restrictions on the structure of entities, such as whether a given property has a value of a particular type; this task is left to the application.

Kinds and identifiers

Each Datastore entity is of a particular kind, which categorizes the entity for the purpose of queries: for instance, a human resources application might represent each employee at a company with an entity of kind Employee. In the Java Datastore API, you specify an entity's kind when you create it, as an argument to the Entity() constructor. All kind names that begin with two underscores (__) are reserved and may not be used.

The following example creates an entity of kind Employee, populates its property values, and saves it to Datastore:

appengine-java8/datastore/src/test/java/com/example/appengine/EntitiesTest.java

View on GitHub

Entity employee = new Entity("Employee", "asalieri");
employee.setProperty("firstName", "Antonio");
employee.setProperty("lastName", "Salieri");
employee.setProperty("hireDate", new Date());
employee.setProperty("attendedHrTraining", true);

DatastoreService datastore = DatastoreServiceFactory.getDatastoreService();
datastore.put(employee);

In addition to a kind, each entity has an identifier, assigned when the entity is created. Because it is part of the entity's key, the identifier is associated permanently with the entity and cannot be changed. It can be assigned in either of two ways:

• Your application can specify its own key name string for the entity.
• You can have Datastore automatically assign the entity an integer numeric ID.

To assign an entity a key name, provide the name as the second argument to the constructor when you create the entity:

Entity employee = new Entity("Employee", "asalieri");

To have Datastore assign a numeric ID automatically, omit this argument:

Entity employee = new Entity("Employee");

Assigning identifiers

Datastore can be configured to generate auto IDs using two different auto id policies:

• The default policy generates a random sequence of unused IDs that are approximately uniformly distributed. Each ID can be up to 16 decimal digits long.
• The legacy policy creates a sequence of non-consecutive smaller integer IDs.

If you want to display the entity IDs to the user, and/or depend upon their order, the best thing to do is use manual allocation.

Note: Instead of using key name strings or generating numeric IDs automatically, advanced applications may sometimes wish to assign their own numeric IDs manually to the entities they create. Be aware, however, that there is nothing to prevent Datastore from assigning one of your manual numeric IDs to another entity. The only way to avoid such conflicts is to have your application obtain a block of IDs with the methods DatastoreService.allocateIds() or AsyncDatastoreService.allocateIds(). Datastore's automatic ID generator will keep track of IDs that have been allocated with these methods and will avoid reusing them for another entity, so you can safely use such IDs without conflict.

Datastore generates a random sequence of unused IDs that are approximately uniformly distributed. Each ID can be up to 16 decimal digits long.

System-allocated ID values are guaranteed unique to the entity group. If you copy an entity from one entity group or namespace to another and wish to preserve the ID part of the key, be sure to allocate the ID first to prevent Datastore from selecting that ID for a future assignment.

Ancestor paths

Entities in Cloud Datastore form a hierarchically structured space similar to the directory structure of a file system. When you create an entity, you can optionally designate another entity as its parent; the new entity is a child of the parent entity (note that unlike in a file system, the parent entity need not actually exist). An entity without a parent is a root entity. The association between an entity and its parent is permanent, and cannot be changed once the entity is created. Cloud Datastore will never assign the same numeric ID to two entities with the same parent, or to two root entities (those without a parent).

An entity's parent, parent's parent, and so on recursively, are its ancestors; its children, children's children, and so on, are its descendants. A root entity and all of its descendants belong to the same entity group. The sequence of entities beginning with a root entity and proceeding from parent to child, leading to a given entity, constitute that entity's ancestor path. The complete key identifying the entity consists of a sequence of kind-identifier pairs specifying its ancestor path and terminating with those of the entity itself:

[Person:GreatGrandpa, Person:Grandpa, Person:Dad, Person:Me]

For a root entity, the ancestor path is empty and the key consists solely of the entity's own kind and identifier:

[Person:GreatGrandpa]

This concept is illustrated by the following diagram:

To designate an entity's parent, provide the parent entity's key as an argument to the Entity() constructor when creating the child entity. You can get the key by calling the parent entity's getKey() method:

appengine-java8/datastore/src/test/java/com/example/appengine/EntitiesTest.java

View on GitHub

Entity employee = new Entity("Employee");
datastore.put(employee);

Entity address = new Entity("Address", employee.getKey());
datastore.put(address);

If the new entity also has a key name, provide the key name as the second argument to the Entity() constructor and the key of the parent entity as the third argument:

Entity address = new Entity("Address", "addr1", employee.getKey());

Transactions and entity groups

Every attempt to create, update, or delete an entity takes place in the context of a transaction. A single transaction can include any number of such operations. To maintain the consistency of the data, the transaction ensures that all of the operations it contains are applied to Datastore as a unit or, if any of the operations fails, that none of them are applied. Furthermore, all strongly- consistent reads (ancestor queries or gets) performed within the same transaction observe a consistent snapshot of the data.

Note: If your application receives an exception when attempting to commit a transaction, it does not necessarily mean that the transaction has failed. It is possible to receive a DatastoreTimeoutException or DatastoreFailureException even when a transaction has been committed and will eventually be applied successfully. Whenever possible, structure your Datastore transactions so that the end result will be unaffected if the same transaction is applied more than once.

As mentioned above, an entity group is a set of entities connected through ancestry to a common root element. The organization of data into entity groups can limit what transactions can be performed:

• All the data accessed by a transaction must be contained in at most 25 entity groups.
• If you want to use queries within a transaction, your data must be organized into entity groups in such a way that you can specify ancestor filters that will match the right data.
• There is a write throughput limit of about one transaction per second within a single entity group. This limitation exists because Datastore performs masterless, synchronous replication of each entity group over a wide geographic area to provide high reliability and fault tolerance.

In many applications, it is acceptable to use eventual consistency (i.e. a non-ancestor query spanning multiple entity groups, which may at times return slightly stale data) when obtaining a broad view of unrelated data, and then to use strong consistency (an ancestor query, or a get of a single entity) when viewing or editing a single set of highly related data. In such applications, it is usually a good approach to use a separate entity group for each set of highly related data. For more information, see Structuring for Strong Consistency.

Note: Avoid storing sensitive information in the entity group key. Entity group keys may be retained after the entity group is deleted in order to provide fast and reliable service across Datastore.

Properties and value types

The data values associated with an entity consist of one or more properties. Each property has a name and one or more values. A property can have values of more than one type, and two entities can have values of different types for the same property. Properties can be indexed or unindexed (queries that order or filter on a property P will ignore entities where P is unindexed). An entity can have at most 20,000 indexed properties.

Note: Properties with multiple values can be useful, for instance, when performing queries with equality filters: an entity satisfies the query if any of its values for a property matches the value specified in the filter. For more details on multiple-valued properties, including issues you should be aware of, see the Datastore Queries page.

The following value types are supported:

Value type	Java type(s)	Sort order	Notes
Integer	short int long java.lang.Short java.lang.Integer java.lang.Long	Numeric	Stored as long integer, then converted to the field type Out-of-range values overflow
Floating-point number	float double java.lang.Float java.lang.Double	Numeric	64-bit double precision, IEEE 754
Boolean	boolean java.lang.Boolean	false<true
Text string (short)	java.lang.String	Unicode	Up to 1500 bytes Values greater than 1500 bytes throw IllegalArgumentException
Text string (long)	com.google.appengine.api.datastore.Text	None	Up to 1 megabyte Not indexed
Byte string (short)	com.google.appengine.api.datastore.ShortBlob	Byte order	Up to 1500 bytes Values longer than 1500 bytes throw IllegalArgumentException
Byte string (long)	com.google.appengine.api.datastore.Blob	None	Up to 1 megabyte Not indexed
Date and time	java.util.Date	Chronological
Geographical point	com.google.appengine.api.datastore.GeoPt	By latitude, then longitude
Postal address	com.google.appengine.api.datastore.PostalAddress	Unicode
Telephone number	com.google.appengine.api.datastore.PhoneNumber	Unicode
Email address	com.google.appengine.api.datastore.Email	Unicode
Google Accounts user	com.google.appengine.api.users.User	Email address in Unicode order
Instant messaging handle	com.google.appengine.api.datastore.IMHandle	Unicode
Link	com.google.appengine.api.datastore.Link	Unicode
Category	com.google.appengine.api.datastore.Category	Unicode
Rating	com.google.appengine.api.datastore.Rating	Numeric
Datastore key	com.google.appengine.api.datastore.Key or the referenced object (as a child)	By path elements (kind, identifier, kind, identifier...)	Up to 1500 bytes Values longer than 1500 bytes throw IllegalArgumentException
Blobstore key	com.google.appengine.api.blobstore.BlobKey	Byte order
Embedded entity	com.google.appengine.api.datastore.EmbeddedEntity	None	Not indexed
Null	null	None

Important: We strongly recommend that you avoid storing a users.User as a property value, because this includes the email address along with the unique ID. If a user changes their email address and you compare their old, stored user.User to the new user.User value, they won't match. Instead, use the User user ID value as the user's stable unique identifier.

For text strings and unencoded binary data (byte strings), Datastore supports two value types:

• Short strings (up to 1500 bytes) are indexed and can be used in query filter conditions and sort orders.
• Long strings (up to 1 megabyte) are not indexed and cannot be used in query filters and sort orders.

Note: The long byte string type is named Blob in the Datastore API. This type is unrelated to blobs as used in the Blobstore API.

When a query involves a property with values of mixed types, Datastore uses a deterministic ordering based on the internal representations:

1. Null values
2. Fixed-point numbers
   • Integers
   • Dates and times
   • Ratings
3. Boolean values
4. Byte sequences
   • Byte string
   • Unicode string
   • Blobstore keys
5. Floating-point numbers
6. Geographical points
7. Google Accounts users
8. Datastore keys

Because long text strings, long byte strings, and embedded entities are not indexed, they have no ordering defined.

Note: Integers and floating-point numbers are considered separate types in Datastore. If an entity uses a mix of integers and floats for the same property, all integers will be sorted before all floats: for example,

7 < 3.2

Working with entities

Applications can use the Datastore API to create, retrieve, update, and delete entities. If the application knows the complete key for an entity (or can derive it from its parent key, kind, and identifier), it can use the key to operate directly on the entity. An application can also obtain an entity's key as a result of a Datastore query; see the Datastore Queries page for more information.

The Java Datastore API uses methods of the DatastoreService interface to operate on entities. You obtain a DatastoreService object by calling the static method DatastoreServiceFactory.getDatastoreService():

DatastoreService datastore = DatastoreServiceFactory.getDatastoreService();

Creating an entity

You can create a new entity by constructing an instance of class Entity, supplying the entity's kind as an argument to the Entity() constructor.

After populating the entity's properties if necessary, you save it to the datastore by passing it as an argument to the DatastoreService.put() method. You can specify the entity's key name by passing it as the second argument to the constructor:

appengine-java8/datastore/src/test/java/com/example/appengine/EntitiesTest.java

View on GitHub

Entity employee = new Entity("Employee", "asalieri");
// Set the entity properties.
// ...
datastore.put(employee);

If you don't provide a key name, Datastore will automatically generate a numeric ID for the entity's key:

appengine-java8/datastore/src/test/java/com/example/appengine/EntitiesTest.java

View on GitHub

Entity employee = new Entity("Employee");
// Set the entity properties.
// ...
datastore.put(employee);

Retrieving an entity

To retrieve an entity identified by a given key, pass the Key object to the DatastoreService.get() method:

// Key employeeKey = ...;
Entity employee = datastore.get(employeeKey);

Updating an entity

To update an existing entity, modify the attributes of the Entity object, then pass it to the DatastoreService.put() method. The object data overwrites the existing entity. The entire object is sent to Datastore with every call to put().

Note: The Datastore API does not distinguish between creating a new entity and updating an existing one. If the object's key represents an entity that already exists, the put() method overwrites the existing entity. You can use a transaction to test whether an entity with a given key exists before creating one.

Deleting an entity

Given an entity's key, you can delete the entity with the DatastoreService.delete() method:

// Key employeeKey = ...;
datastore.delete(employeeKey);

Repeated properties

You can store multiple values within a single property.

appengine-java8/datastore/src/test/java/com/example/appengine/EntitiesTest.java

View on GitHub

Entity employee = new Entity("Employee");
ArrayList<String> favoriteFruit = new ArrayList<String>();
favoriteFruit.add("Pear");
favoriteFruit.add("Apple");
employee.setProperty("favoriteFruit", favoriteFruit);
datastore.put(employee);

// Sometime later
employee = datastore.get(employee.getKey());
@SuppressWarnings("unchecked") // Cast can't verify generic type.
    ArrayList<String> retrievedFruits = (ArrayList<String>) employee
    .getProperty("favoriteFruit");

Embedded entities

You may sometimes find it convenient to embed one entity as a property of another entity. This can be useful, for instance, for creating a hierarchical structure of property values within an entity. The Java class EmbeddedEntity allows you to do this:

appengine-java8/datastore/src/test/java/com/example/appengine/EntitiesTest.java

View on GitHub

// Entity employee = ...;
EmbeddedEntity embeddedContactInfo = new EmbeddedEntity();

embeddedContactInfo.setProperty("homeAddress", "123 Fake St, Made, UP 45678");
embeddedContactInfo.setProperty("phoneNumber", "555-555-5555");
embeddedContactInfo.setProperty("emailAddress", "[e-mail]");

employee.setProperty("contactInfo", embeddedContactInfo);

When an embedded entity is included in indexes, you can query on subproperties. If you exclude an embedded entity from indexing, then all subproperties are also excluded from indexing. You can optionally associate a key with an embedded entity, but (unlike a full-fledged entity) the key is not required and, even if present, cannot be used to retrieve the entity.

Instead of populating the embedded entity's properties manually, you can use the setPropertiesFrom() method to copy them from an existing entity:

appengine-java8/datastore/src/test/java/com/example/appengine/EntitiesTest.java

View on GitHub

// Entity employee = ...;
// Entity contactInfo = ...;
EmbeddedEntity embeddedContactInfo = new EmbeddedEntity();

embeddedContactInfo.setKey(contactInfo.getKey()); // Optional, used so we can recover original.
embeddedContactInfo.setPropertiesFrom(contactInfo);

employee.setProperty("contactInfo", embeddedContactInfo);

You can later use the same method to recover the original entity from the embedded entity:

appengine-java8/datastore/src/test/java/com/example/appengine/EntitiesTest.java

View on GitHub

Entity employee = datastore.get(employeeKey);
EmbeddedEntity embeddedContactInfo = (EmbeddedEntity) employee.getProperty("contactInfo");

Key infoKey = embeddedContactInfo.getKey();
Entity contactInfo = new Entity(infoKey);
contactInfo.setPropertiesFrom(embeddedContactInfo);

Batch operations

The DatastoreService methods put(), get(), and delete() (and their AsyncDatastoreService counterparts) have batch versions that accept an iterable object (of class Entity for put(), Key for get() and delete()) and use it to operate on multiple entities in a single Datastore call:

appengine-java8/datastore/src/test/java/com/example/appengine/EntitiesTest.java

View on GitHub

Entity employee1 = new Entity("Employee");
Entity employee2 = new Entity("Employee");
Entity employee3 = new Entity("Employee");
// ...

List<Entity> employees = Arrays.asList(employee1, employee2, employee3);
datastore.put(employees);

These batch operations group all the entities or keys by entity group and then perform the requested operation on each entity group in parallel. Such batch calls are faster than making separate calls for each individual entity, because they incur the overhead for only one service call. If multiple entity groups are involved, the work for all the groups is performed in parallel on the server side.

Note: A batch put() or delete() call may succeed for some entities but not others. If it is important that the call succeed completely or fail completely, use a transaction with all affected entities in the same entity group. Attempting a batch operation inside a transaction with entities or keys belonging to multiple entity groups will result in an IllegalArgumentException.

Generating keys

Applications can use the class KeyFactory to create a Key object for an entity from known components, such as the entity's kind and identifier. For an entity with no parent, pass the kind and identifier (either a key name string or a numeric ID) to the static method KeyFactory.createKey() to create the key. The following examples create a key for an entity of kind Person with key name "GreatGrandpa" or numeric ID 74219:

Key k1 = KeyFactory.createKey("Person", "GreatGrandpa");
Key k2 = KeyFactory.createKey("Person", 74219);

If the key includes a path component, you can use the helper class KeyFactory.Builder to build the path. This class's addChild method adds a single entity to the path and returns the builder itself, so you can chain together a series of calls, beginning with the root entity, to build up the path one entity at a time. After building the complete path, call getKey to retrieve the resulting key:

appengine-java8/datastore/src/test/java/com/example/appengine/EntitiesTest.java

View on GitHub

Key k =
    new KeyFactory.Builder("Person", "GreatGrandpa")
        .addChild("Person", "Grandpa")
        .addChild("Person", "Dad")
        .addChild("Person", "Me")
        .getKey();

Class KeyFactory also includes the static methods keyToString and stringToKey for converting between keys and their string representations:

appengine-java8/datastore/src/test/java/com/example/appengine/EntitiesTest.java

View on GitHub

String personKeyStr = KeyFactory.keyToString(k);

// Some time later (for example, after using personKeyStr in a link).
Key personKey = KeyFactory.stringToKey(personKeyStr);
Entity person = datastore.get(personKey);

The string representation of a key is "web-safe": it does not contain characters considered special in HTML or in URLs.

Note: The KeyFactory.keyToString method is different from Key.toString, which returns a human-readable string suitable for use in debugging and logging. If you need a string value that can be converted to a usable key, use KeyFactory.keyToString.

Note also that a key's string representation is not encrypted: a user can decode the key string to extract its components, including the kinds and identifiers of the entity and its ancestors. If it is important to conceal this information from the user, you must encrypt the key string yourself before sending it to the user.

Using an empty list

Datastore historically did not have a representation for a property representing an empty list. The Java SDK worked around this by storing empty collections as null values, so there is no way to distinguish between null values and empty lists. To maintain backward compatibility, this remains the default behavior, synopsized as follows:

• Null properties are written as null to Datastore
• Empty collections are written as null to Datastore
• A null is read as null from Datastore
• An empty collection is read as null.

Important: A read modify write of an entity with an empty list will cause that list to be turned into a null value.

However, if you change the default behavior, the SDK for Java will support storage of empty lists. We recommend you consider the implications of changing the default behavior of your application and then turn on support for empty lists.

To change default behavior so you can use empty lists, set the DATASTORE_EMPTY_LIST_SUPPORT property during your app initialization as follows:

System.setProperty(DatastoreServiceConfig.DATASTORE_EMPTY_LIST_SUPPORT, Boolean.TRUE.toString());

With this property set to true as shown above:

• Null properties are written as null to Datastore
• Empty collections are written as empty list to Datastore
• A null is read as null from Datastore
• When reading from Datastore an empty list is returned as an empty Collection.

Important: Your queries might be affected when you turn on empty list support. Null values are indexed in Datastore but empty lists are not. If you were storing empty lists as nulls and then querying for null to find them, then changing to empty list will cause these queries to not return results.