Entities, Properties, and Keys

This API is supported for first-generation runtimes and can be used when upgrading to corresponding second-generation runtimes. If you are updating to the App Engine Go 1.12+ runtime, refer to the migration guide to learn about your migration options for legacy bundled services.

Data objects in Datastore are known as entities. An entity has one or more named properties, each of which can have one or more values. Entities of the same kind do not need to have the same properties, and an entity's values for a given property do not all need to be of the same data type. (If necessary, an application can establish and enforce such restrictions in its own data model.)

Datastore supports a variety of data types for property values. These include, among others:

• Integers
• Floating-point numbers
• Strings
• Dates
• Binary data

For a full list of types, see Properties and value types.

Each entity in Datastore has a key that uniquely identifies it. The key consists of the following components:

• The namespace of the entity, which allows for multitenancy
• The kind of the entity, which categorizes it for the purpose of Datastore queries
• An identifier for the individual entity, which can be either
  • a key name string
  • an integer numeric ID
• An optional ancestor path locating the entity within the Datastore hierarchy

Don't name a property "key." This name is reserved for a special property used to store the Model key. Though it may work locally, a property named "key" will prevent deployment to App Engine.

An application can fetch an individual entity from Datastore using the entity's key, or it can retrieve one or more entities by issuing a query based on the entities' keys or property values.

The Go App Engine SDK includes a package for representing Datastore entities as Go structs, and for storing and retrieving them in Datastore.

Datastore itself does not enforce any restrictions on the structure of entities, such as whether a given property has a value of a particular type; this task is left to the application.

Kinds and identifiers

Each Datastore entity is of a particular kind, which categorizes the entity for the purpose of queries: for instance, a human resources application might represent each employee at a company with an entity of kind Employee. In the Go Datastore API, you specify an entity's kind when you create a datastore.Key. All kind names that begin with two underscores (__) are reserved and may not be used.

The following example creates an entity of kind Employee, populates its property values, and saves it to Datastore:

import (
	"context"
	"time"

	"google.golang.org/appengine/datastore"
)

type Employee struct {
	FirstName          string
	LastName           string
	HireDate           time.Time
	AttendedHRTraining bool
}

func f(ctx context.Context) {
	// ...
	employee := &Employee{
		FirstName: "Antonio",
		LastName:  "Salieri",
		HireDate:  time.Now(),
	}
	employee.AttendedHRTraining = true

	key := datastore.NewIncompleteKey(ctx, "Employee", nil)
	if _, err := datastore.Put(ctx, key, employee); err != nil {
		// Handle err
	}
	// ...
}

The Employee type declares four fields for the data model: FirstName, LastName, HireDate, and AttendedHRTraining.

In addition to a kind, each entity has an identifier, assigned when the entity is created. Because it is part of the entity's key, the identifier is associated permanently with the entity and cannot be changed. It can be assigned in either of two ways:

• Your application can specify its own key name string for the entity.
• You can have Datastore automatically assign the entity an integer numeric ID.

To assign an entity a key name, provide a non-empty stringID argument to datastore.NewKey:

// Create a key with a key name "asalieri".
key := datastore.NewKey(
	ctx,        // context.Context
	"Employee", // Kind
	"asalieri", // String ID; empty means no string ID
	0,          // Integer ID; if 0, generate automatically. Ignored if string ID specified.
	nil,        // Parent Key; nil means no parent
)

To have Datastore assign a numeric ID automatically, use an empty stringID argument:

// Create a key such as Employee:8261.
key := datastore.NewKey(ctx, "Employee", "", 0, nil)
// This is equivalent:
key = datastore.NewIncompleteKey(ctx, "Employee", nil)

Assigning identifiers

Datastore can be configured to generate auto IDs using two different auto id policies:

• The default policy generates a random sequence of unused IDs that are approximately uniformly distributed. Each ID can be up to 16 decimal digits long.
• The legacy policy creates a sequence of non-consecutive smaller integer IDs.

If you want to display the entity IDs to the user, and/or depend upon their order, the best thing to do is use manual allocation.

Note: Instead of using key name strings or generating numeric IDs automatically, advanced applications may sometimes wish to assign their own numeric IDs manually to the entities they create. Be aware, however, that there is nothing to prevent Datastore from assigning one of your manual numeric IDs to another entity. The only way to avoid such conflicts is to have your application obtain a block of IDs with the datastore.AllocateIDs function. Datastore's automatic ID generator will keep track of IDs that have been allocated with this function and will avoid reusing them for another entity, so you can safely use such IDs without conflict.

Datastore generates a random sequence of unused IDs that are approximately uniformly distributed. Each ID can be up to 16 decimal digits long.

Ancestor paths

Entities in Cloud Datastore form a hierarchically structured space similar to the directory structure of a file system. When you create an entity, you can optionally designate another entity as its parent; the new entity is a child of the parent entity (note that unlike in a file system, the parent entity need not actually exist). An entity without a parent is a root entity. The association between an entity and its parent is permanent, and cannot be changed once the entity is created. Cloud Datastore will never assign the same numeric ID to two entities with the same parent, or to two root entities (those without a parent).

An entity's parent, parent's parent, and so on recursively, are its ancestors; its children, children's children, and so on, are its descendants. A root entity and all of its descendants belong to the same entity group. The sequence of entities beginning with a root entity and proceeding from parent to child, leading to a given entity, constitute that entity's ancestor path. The complete key identifying the entity consists of a sequence of kind-identifier pairs specifying its ancestor path and terminating with those of the entity itself:

[Person:GreatGrandpa, Person:Grandpa, Person:Dad, Person:Me]

For a root entity, the ancestor path is empty and the key consists solely of the entity's own kind and identifier:

[Person:GreatGrandpa]

This concept is illustrated by the following diagram:

To designate an entity's parent, use the parent argument to datastore.NewKey. The value of this argument should be the parent entity's key.. The following example creates an entity of kind Address and designates an Employee entity as its parent:

// Create Employee entity
employee := &Employee{ /* ... */ }
employeeKey, err := datastore.Put(ctx, datastore.NewIncompleteKey(ctx, "Employee", nil), employee)

// Use Employee as Address entity's parent
// and save Address entity to datastore
address := &Address{ /* ... */ }
addressKey := datastore.NewIncompleteKey(ctx, "Address", employeeKey)
_, err = datastore.Put(ctx, addressKey, address)

Transactions and entity groups

Every attempt to create, update, or delete an entity takes place in the context of a transaction. A single transaction can include any number of such operations. To maintain the consistency of the data, the transaction ensures that all of the operations it contains are applied to Datastore as a unit or, if any of the operations fails, that none of them are applied. Furthermore, all strongly- consistent reads (ancestor queries or gets) performed within the same transaction observe a consistent snapshot of the data.

Note: If your application receives an error when attempting to commit a transaction, it does not necessarily mean that the transaction has failed. It is possible to receive an error even when a transaction has been committed and will eventually be applied successfully. Whenever possible, structure your Datastore transactions so that the end result will be unaffected if the same transaction is applied more than once.

As mentioned above, an entity group is a set of entities connected through ancestry to a common root element. The organization of data into entity groups can limit what transactions can be performed:

• All the data accessed by a transaction must be contained in at most 25 entity groups.
• If you want to use queries within a transaction, your data must be organized into entity groups in such a way that you can specify ancestor filters that will match the right data.
• There is a write throughput limit of about one transaction per second within a single entity group. This limitation exists because Datastore performs masterless, synchronous replication of each entity group over a wide geographic area to provide high reliability and fault tolerance.

In many applications, it is acceptable to use eventual consistency (i.e. a non-ancestor query spanning multiple entity groups, which may at times return slightly stale data) when obtaining a broad view of unrelated data, and then to use strong consistency (an ancestor query, or a get of a single entity) when viewing or editing a single set of highly related data. In such applications, it is usually a good approach to use a separate entity group for each set of highly related data. For more information, see Structuring for Strong Consistency.

Note: Avoid storing sensitive information in the entity group key. Entity group keys may be retained after the entity group is deleted in order to provide fast and reliable service across Datastore.

Properties and value types

The data values associated with an entity consist of one or more properties. Each property has a name and one or more values. A property can have values of more than one type, and two entities can have values of different types for the same property. Properties can be indexed or unindexed (queries that order or filter on a property P will ignore entities where P is unindexed). An entity can have at most 20,000 indexed properties.

Note: Properties with multiple values can be useful, for instance, when performing queries with equality filters: an entity satisfies the query if any of its values for a property matches the value specified in the filter. For more details on multiple-valued properties, including issues you should be aware of, see the Datastore Queries page.

The following value types are supported:

Value type	Go type(s)	Sort order	Notes
Integer	int int8 int16 int32 int64	Numeric	64-bit integer, signed
Floating-point number	float32 float64	Numeric	64-bit double precision, IEEE 754
Boolean	bool	false<true
String (short)	string	Unicode	Up to 1500 bytes. Values longer than 1500 bytes result in an error at runtime.
String (long)	string (with noindex)	None	Up to 1 megabyte Not indexed
Byte slice (short)	datastore.ByteString	Byte order	Up to 1500 bytes. Values longer than 1500 bytes result in an error at runtime.
Byte slice (long)	[]byte	None	Up to 1 megabyte Not indexed
Date and time	time.Time	Chronological
Geographical point	appengine.GeoPoint	By latitude, then longitude
Datastore key	*datastore.Key	By path elements (kind, identifier, kind, identifier...)
Blobstore key	appengine.BlobKey	Byte order

You can also use a struct or slice to aggregate properties. See the Datastore reference for more details.

When a query involves a property with values of mixed types, Datastore uses a deterministic ordering based on the internal representations:

1. Null values
2. Fixed-point numbers
   • Integers
   • Dates and times
3. Boolean values
4. Byte sequences
   • Byte slices (short)
   • Unicode string
   • Blobstore keys
5. Floating-point numbers
6. Geographical points
7. Datastore keys

Because long byte slices and long strings are not indexed, they have no ordering defined.

Note: Integers and floating-point numbers are considered separate types in Datastore. If an entity uses a mix of integers and floats for the same property, all integers will be sorted before all floats: for example,

7 < 3.2

Working with entities

Applications can use the Datastore API to create, retrieve, update, and delete entities. If the application knows the complete key for an entity (or can derive it from its parent key, kind, and identifier), it can use the key to operate directly on the entity. An application can also obtain an entity's key as a result of a Datastore query; see the Datastore Queries page for more information.

Creating an entity

In Go, you create a new entity by constructing an instance of a Go struct, populating its fields, and calling datastore.Put to save it to Datastore. Only exported fields (beginning with an upper case letter) will be saved to Datastore. You can specify the entity's key name by passing a non-empty stringID argument to datastore.NewKey:

employee := &Employee{
	FirstName: "Antonio",
	LastName:  "Salieri",
	HireDate:  time.Now(),
}
employee.AttendedHRTraining = true
key := datastore.NewKey(ctx, "Employee", "asalieri", 0, nil)
_, err = datastore.Put(ctx, key, employee)

If you provide an empty key name, or use datastore.NewIncompleteKey, Datastore will automatically generate a numeric ID for the entity's key:

employee := &Employee{
	FirstName: "Antonio",
	LastName:  "Salieri",
	HireDate:  time.Now(),
}
employee.AttendedHRTraining = true
key := datastore.NewIncompleteKey(ctx, "Employee", nil)
_, err = datastore.Put(ctx, key, employee)

Retrieving an entity

To retrieve an entity identified by a given key, pass the *datastore.Key as an argument to the datastore.Get function. You can generate the *datastore.Key using the datastore.NewKey function.

employeeKey := datastore.NewKey(ctx, "Employee", "asalieri", 0, nil)
addressKey := datastore.NewKey(ctx, "Address", "", 1, employeeKey)
var addr Address
err = datastore.Get(ctx, addressKey, &addr)

datastore.Get populates an instance of the appropriate Go struct.

Updating an entity

To update an existing entity, modify the attributes of the struct, then call datastore.Put. The data overwrites the existing entity. The entire object is sent to Datastore with every call to datastore.Put.

Note: The Datastore API does not distinguish between creating a new entity and updating an existing one. If the key represents an entity that already exists, the datastore.Put function overwrites the existing entity. You can use a transaction to test whether an entity with a given key exists before creating one.

Deleting an entity

Given an entity's key, you can delete the entity with the datastore.Delete function:

key := datastore.NewKey(ctx, "Employee", "asalieri", 0, nil)
err = datastore.Delete(ctx, key)

Batch operations

datastore.Put, datastore.Get and datastore.Delete have bulk variants called datastore.PutMulti, datastore.GetMulti and datastore.DeleteMulti. They permit acting on multiple entities in a single Datastore call:

// A batch put.
_, err = datastore.PutMulti(ctx, []*datastore.Key{k1, k2, k3}, []interface{}{e1, e2, e3})

// A batch get.
var entities = make([]*T, 3)
err = datastore.GetMulti(ctx, []*datastore.Key{k1, k2, k3}, entities)

// A batch delete.
err = datastore.DeleteMulti(ctx, []*datastore.Key{k1, k2, k3})

Batch operations do not change your costs. You will be charged for every key in a batched operation, whether or not each key exists. The size of the entities involved in an operation does not affect the cost.

Note: A call to datastore.PutMulti or datastore.DeleteMulti may succeed for some entities but not others. If it is important that the call succeed completely or fail completely, you must use a transaction, and all affected entities must be in the same entity group.