Firestore in Datastore mode (Datastore) supports a variety of data types for property values. These include, among others:
- Integers
- Floating-point numbers
- Strings
- Dates
- Binary data
For a full list of types, see properties and value types.
Properties and value types
The data values associated with an entity consist of one or more properties. Each property has a name and one or more values. A property can have values of more than one type, and two entities can have values of different types for the same property. Properties can be indexed or unindexed (queries that order or filter on a property P will ignore entities where P is unindexed). An entity can have at most 20,000 indexed properties.
The following value types are supported:
When a query involves a property with values of mixed types, Datastore uses a deterministic ordering based on the internal representations:
- Null values
- Fixed-point numbers
- Integers
- Dates and times
- Boolean values
- Byte sequences
- Unicode string
- Blobstore keys
- Floating-point numbers
- Datastore keys
Because long text strings and long byte strings are not indexed, they have no ordering defined.
Property Types
NDB supports the following property types:
Property type | Description |
---|---|
IntegerProperty |
64-bit signed integer |
FloatProperty |
Double-precision floating-point number |
BooleanProperty |
Boolean |
StringProperty |
Unicode string; up to 1500 bytes, indexed |
TextProperty |
Unicode string; unlimited length, not indexed |
BlobProperty |
Uninterpreted byte string: if you set indexed=True , up to 1500 bytes, indexed;if indexed is False (the default), unlimited length, not indexed.Optional keyword argument: compressed . |
DateTimeProperty |
Date and time (see Date and Time Properties) |
DateProperty |
Date (see Date and Time Properties) |
TimeProperty |
Time (see Date and Time Properties) |
GeoPtProperty |
Geographical location. This is a ndb.GeoPt object. The object has attributes lat and lon , both floats. You can construct one with two floats like ndb.GeoPt(52.37, 4.88) or with a string ndb.GeoPt("52.37, 4.88") . (This is actually the same class as db.GeoPt ) |
KeyProperty |
Datastore key Optional keyword argument: kind=kind, to require that keys assigned to this property always have the indicated kind. May be a string or a Model subclass. |
BlobKeyProperty |
Blobstore key Corresponds to BlobReferenceProperty in the old db API, but the property value is a BlobKey instead of a BlobInfo ; you can construct a BlobInfo from it using BlobInfo(blobkey) |
UserProperty |
User object. |
StructuredProperty |
Includes one kind of model inside another, by value (see Structured Properties) |
LocalStructuredProperty |
Like StructuredProperty , but on-disk representation is an opaque blob and is not indexed (see Structured Properties).Optional keyword argument: compressed . |
JsonProperty |
Value is a Python object (such as a list or a dict or a string) that is serializable using Python's json module; Datastore stores the JSON serialization as a blob. Unindexed by default.Optional keyword argument: compressed . |
PickleProperty |
Value is a Python object (such as a list or a dict or a string) that is serializable using Python's pickle protocol; Datastore stores the pickle serialization as a blob. Unindexed by default. Optional keyword argument: compressed . |
GenericProperty |
Generic value Used mostly by the Expando class, but also usable explicitly. Its type may be any of int , long , float , bool , str , unicode , datetime , Key , BlobKey , GeoPt , User , None . |
ComputedProperty |
Value computed from other properties by a user-defined function. (See Computed Properties.) |
Some of these properties have an optional keyword argument,
compressed
. If the property has compressed=True
, then its data is compressed
with gzip on disk. It takes up less space but needs CPU to encode/decode on
write and read operations.
Both compression and decompression are "lazy"; a compressed property value will only be decompressed the first time you access it. If you read an entity containing a compressed property value and write it back without accessing the compressed property, it won't be decompressed and compressed at all. The in-context cache participates in this lazy scheme as well, but memcache always stores the compressed value for compressed properties.
Because of the extra CPU time needed for compression, it is usually best to use compressed properties only if the data would be too big to fit without it. Remember that gzip-based compression is typically not effective for images and other media data, since those formats already are compressed using a media-specific compression algorithm (e.g. JPEG for images).
Property Options
Most property types support some standard arguments. The first is an optional positional argument specifying the property's Datastore name. You can use this to give the property a different name in Datastore than from the application's viewpoint. A common use for this is to reduce space in Datastore, letting Datastore use abbreviated property names while your code uses longer, more meaningful ones. For example,
This is particularly useful for repeated properties for which you expect many values per entity.
In addition, most property types support the following keyword arguments:
Argument | Type | Default | Description |
---|---|---|---|
indexed |
bool |
Usually True |
Include property in Datastore's indexes; if False , values cannot be queried but writes are faster. Not all property types support indexing; setting indexed to True fails for these.Unindexed properties cost fewer write ops than indexed properties. |
repeated |
bool |
False |
Property value is a Python list containing values of the underlying type (see Repeated Properties). Cannot be combined with required=True or default=True . |
required |
bool |
False |
Property must have a value specified. |
default |
Property's underlying type | None | Default value of property if none explicitly specified. |
choices |
List of values of underlying type | None |
Optional list of allowable values. |
validator |
Function | None |
Optional function to validate and possibly coerce the value. Will be called with arguments (prop, value) and should either return the (possibly coerced) value or raise an exception. Calling the function again on a coerced value should not modify the value further. (For example, returning value.strip() or value.lower() is fine, but not value + '$' .) May also return None , which means "no change". See also Writing Property Subclasses |
verbose_name |
string | None |
Optional HTML label to use in web form frameworks like jinja2. |
Repeated Properties
Any property with repeated=True
becomes a repeated property. The property
takes a list of values of the underlying type, rather than a single value. For
example, the value of a property defined with IntegerProperty(repeated=True)
is a list of integers.
Datastore may see multiple values for such a property. A separate index record is created for each value. This affects query semantics; see Querying for Repeated Properties for an example.
This example uses a repeated property:
...
This creates a Datastore entity with the following contents:
When querying for the tags
property, this entity will satisfy a query for
either 'python'
or 'ruby'
.
When updating a repeated property, you can either assign it a new list or mutate
the existing list in place. When you assign a new list, the types of the list
items are validated immediately. Invalid item types (for example, assigning
[1, 2]
to art.tags
above) raise an exception. When you mutate the list,
the change is not validated immediately. Instead, the value will be validated
when you write the entity to Datastore.
Datastore preserves the order of the list items in a repeated property, so you can assign some meaning to their ordering.
Date and Time Properties
Three property types are available for storing date- and time-related values:
DateProperty
TimeProperty
DateTimeProperty
These take values belonging to the corresponding classes (date
, time
,
datetime
) of the standard Python datetime
module. The most general of the
three is DateTimeProperty
, which denotes both a calendar date and a time of
day; the others are occasionally useful for special purposes requiring just a
date (such as a date of birth) or just a time (such as a meeting time). For
technical reasons, DateProperty
and TimeProperty
are subclasses of
DateTimeProperty
, but you shouldn't depend on this inheritance relationship
(and note that it differs from the inheritance relationships between the
underlying classes defined by the datetime
module itself).
Each of these properties has two extra Boolean keyword options:
Option | Description |
---|---|
auto_now_add |
Set property to current date/time when entity is created. You can manually override this property. When the entity is updated, the property doesn't change. For that behavior, use auto_now . |
auto_now |
Set property to current date/time when entity is created and whenever it is updated. |
These options cannot be combined with repeated=True
. Both of them default to
False
; if both are set to True
, auto_now
takes precedence. It is possible
to override the value for a property with auto_now_add=True
, but not for one
with auto_now=True
. The automatic value is not generated until the entity is
written; that is, these options don't provide dynamic defaults. (These details
differ from the old db API.)
Structured Properties
You can structure a model's properties. For example, you can define a model
class Contact containing a list of addresses, each with internal structure.
Structured properties (type StructuredProperty
``) let you do this; for
example:
...
...
This creates a single Datastore entity with the following properties:
Reading back such an entity reconstructs the original Contact
entity exactly.
Although the Address
instances are defined using the same syntax as for model
classes, they are not full-fledged entities. They don't have their own keys in
Datastore. They cannot be retrieved independently of the
Contact
entity to which they belong. An application can, however, query for
the values of their individual fields; see
Filtering for Structured Property Values.
Note that address.type
, address.street
, and address.city
are viewed as
parallel arrays from Datastore's viewpoint, but the NDB library
hides this aspect and constructs the corresponding list of Address
instances.
You can specify the usual property options for structured properties
(except indexed
). The Datastore name is the second
positional argument in this case (the first being the model class used to define
the substructure).
When you don't need to query for a substructure's internal properties, you can
use a local structured property (LocalStructuredProperty
) instead. If you
replace StructuredProperty
with LocalStructuredProperty
in the example
above, the behavior of the Python code is the same, but Datastore sees just an opaque blob for each address. The guido
entity created in the
example would be stored as follows:
name = 'Guido'
address = <opaque blob for {'type': 'home', 'city': 'Amsterdam'}>
address = <opaque blob for {'type': 'work', 'city': 'SF',
'street': 'Spear St'}>
The entity will be read back correctly. Since properties of this type are always unindexed, you cannot query for address values.
Computed Properties
Computed properties (ComputedProperty
) are read-only properties whose value
is computed from other property values by an application-supplied function. Note
that a computed property only supports the types that are supported by generic
properties! The computed value is written to Datastore so that
it can be queried and displayed in the Datastore viewer, but
the stored value is ignored when the entity is read back from Datastore; rather, the value is recomputed by calling the function
whenever the value is requested. For example:
...
This stores an entity with the following property values:
If we change name to 'Nickie' and ask for the value of name_lower
, it returns
'nickie':
Google Protocol RPC Message Properties
The Google Protocol RPC
library uses
Message
objects for structured data; they can represent RPC requests, responses, or
other things. NDB provides an API for storing Google Protocol RPC Message
objects as entity properties. Suppose you define a Message
subclass:
...
You can store Note
objects in Datastore as entity property
values by using NDB's msgprop
API.
...
...
If you want to query for field names, they must be indexed. You can specify a
list of field names that will be indexed with the indexed_fields
parameter to
MessageProperty
.
MessageProperty
supports many, but not all, Property options. It
supports:
name
repeated
required
default
choices
validator
verbose_name
Message properties do not support the indexed
property option; you can't index
Message
values. (You can index fields of a message as described above.)
Nested messages (using MessageField
) also work:
...
MessageProperty
has a special property option, protocol
, which specifies how
the message object is serialized to Datastore. The values are
protocol names as used by protorpc.remote.Protocols
class. Supported protocol
names are protobuf
and protojson
; the default is protobuf
.
msgprop
also defines EnumProperty
, a property type which can be used to
store a protorpc.messages.Enum
value in an entity. Example:
...
...
EnumProperty
stores the value as an integer; in fact, EnumProperty
is a
subclass of IntegerProperty
. This implies that you can rename your enum values
without having to modify already-stored entities, but you cannot renumber them.
The EnumProperty supports the following property options:
name
indexed
repeated
required
default
choices
validator
verbose_name
About NDB entity models
An NDB entity model can define properties. Entity properties are a bit like data members of Python classes, a structured way to hold data; they are also a bit like fields in a database schema.
A typical application defines a data model by defining a class that inherits
from Model
with some property class attributes. For example:
...
Here, username
, userid
, and email
are properties of Account
.
There are several other property types. Some are handy for representing dates and times and have convenient auto-update features.
An application can adjust a property's behavior by specifying options on the property; these can ease validation, set defaults, or change query indexing.
A model can have more complex properties. Repeated properties are list-like. Structured properties are object-like. Read-only computed properties are defined via functions; this makes it easy to define a property in terms of one or more other properties. Expando models can define properties dynamically.