Note: Python 2.7 has reached end of support on January 31, 2024. Your existing Python 2.7 applications will continue to run and receive traffic. However, App Engine might block re-deployment of applications that use runtimes after their end of support date. We recommend that you migrate to the latest supported version of Python.

Datastore Functions

Note: Developers building new applications are strongly encouraged to use the NDB Client Library, which has several benefits compared to this client library, such as automatic entity caching via the Memcache API. If you are currently using the older DB Client Library, read the DB to NDB Migration Guide

The functions described on this page are defined in the google.appengine.ext.db package.

Functions

allocate_ids (model, count)

Allocates a batch of IDs in the Datastore for a Datastore kind and parent combination.

IDs allocated in this manner will not be used by the Datastore's automatic ID sequence generator and can be used in entity keys without conflict.

Arguments

model: The model key for which to allocate an ID batch. This is a regular key, but only the parent and kind of the key are necessary to determine which ID sequence to use.
count: The number of IDs to allocate.

Returns a tuple of the first and last IDs that it allocates. For example, if you allocated 10 IDs using this function you would get a return in the format (1, 10), not a full list of created IDs.

Example of allocating and using IDs:

# allocate for MyModel without an instance
handmade_key = db.Key.from_path('MyModel', 1)
first_batch = db.allocate_ids(handmade_key, 10)
first_range = range(first_batch[0], first_batch[1] + 1)

# or allocate using an existing key
model_instance = MyModel.all().get()
second_batch = db.allocate_ids(model_instance.key(), 10)
second_range = range(second_batch[0], second_batch[1] + 1)

# and then use them! woo!
my_id = second_range.pop(0)
new_key = db.Key.from_path('MyModel', my_id)
new_instance = MyModel(key=new_key)
new_instance.put()
assert new_instance.key().id() == my_id

# the Datastore will not assign ids in first_batch or second_batch
another_instance = MyModel()
another_instance.put()
assert another_instance.key().id() not in first_range
assert another_instance.key().id() not in second_range

allocate_ids_async (model, count)

Asynchronously allocates a batch of IDs in the Datastore for a Datastore kind and parent combination.

This function is identical to allocate_ids() except that it returns an asynchronous object. You can call get_result() on the return value to block on the call and return the result.

Arguments

model: A db.Model instance, key, or string to serve as a template specifying the ID sequence in which to allocate IDs. Returned ids should only be used in entities with the same parent (if any) and kind as this key.
count: The number of IDs to allocate.

Returns a tuple of the first and last IDs that it allocates. For example, if you allocated 10 IDs using this function you would get a return value in the format (1, 10), not a full list of created IDs.

allocate_id_range (model, start, end, **kwargs)

Allocates a range of IDs with specific endpoints. Once these IDs have been allocated, you can manually assign them to newly created entities.

The Datastore's automatic ID allocator never assigns a key that has already been allocated (either through automatic ID allocation or through an explicit `allocate_ids` call). As a result, entities written to the given key range will never be overwritten. However, writing entities with manually assigned keys in this range may overwrite existing entities (or new entities written by a separate request), depending on the key range state returned.

Use this function only if you have an existing numeric id range that you want to reserve (for example, bulk loading entities that already have IDs). If you don't care about which IDs you receive, use allocate_ids() instead.

Arguments

model: A db.Model instance, key, or string to serve as a template specifying the ID sequence in which to allocate IDs. Returned ids should only be used in entities with the same parent (if any) and kind as this key.
start: The first ID to allocate, a number.
end: The last ID to allocate, a number.

Returns one of (KEY_RANGE_EMPTY, KEY_RANGE_CONTENTION, KEY_RANGE_COLLISION). If not KEY_RANGE_EMPTY, this represents a potential issue with using the allocated key range.

create_transaction_options (**kwargs)

Creates a transaction options object (class TransactionOptions) for controlling transaction execution. You pass the resulting object as the first argument to the run_in_transaction_options() function.

Arguments

propagation

What to do if this transactional function is called from within another transaction:

ALLOWED: If already in a transaction, continue using it; if not, start one.
Note: If a function using this policy throws an exception, it is probably not safe to catch the exception and commit the outer transaction; the function may have left the outer transaction in a bad state.
MANDATORY: Continue in the existing transaction, if any; if not, throw a BadRequestError exception.
Note: If a function using this policy throws an exception, it is probably not safe to catch the exception and commit the outer transaction; the function may have left the outer transaction in a bad state.
INDEPENDENT: Create a new transaction, pausing any existing transaction.
Note: A function using this policy should not return any entities read in the new transaction, as the entities are not transactionally consistent with the outer transaction.
NESTED: (Not yet supported) Create a nested transaction within an existing one.

xg

If True, allow cross-group (XG) transactions. Raises a BadArgumentError exception if set to a nonboolean value.

retries

Number of retries to attempt on failure of transaction commit.

deadline

Maximum time, in seconds, to wait for Datastore to return a result before aborting with an error. Accepts either an integer or a floating-point value. Cannot be set higher than the default value (60 seconds), but can be adjusted downward to ensure that a particular operation fails quickly (for instance, to return a faster response to the user, retry the operation, try a different operation, or add the operation to a task queue).

The following example creates the options for a subsequent cross-group (XG) transaction:

from google.appengine.ext import db

xg_on = db.create_transaction_options(xg=True)

def my_txn():
  x = MyModel(a=3)
  x.put()
  y = MyModel(a=7)
  y.put()

db.run_in_transaction_options(xg_on, my_txn)

delete (models, deadline=60)

Deletes one or more model instances from the Datastore.

Arguments

models: A model instance, an entity key, or a list (or other iterable) of model instances or entity keys to delete.
deadline: Maximum time, in seconds, to wait for Datastore to return a result before aborting with an error. Accepts either an integer or a floating-point value. Cannot be set higher than the default value (60 seconds), but can be adjusted downward to ensure that a particular operation fails quickly (for instance, to return a faster response to the user, retry the operation, try a different operation, or add the operation to a task queue).

As with put(), if multiple keys are given, they may be in more than one entity group.

An exception will always be raised if any error occurs during the operation, even if some of the entities actually were deleted. If the call returns without raising an exception, then all of the entities were deleted successfully.

Caution: Deleting multiple entities in a single operation does not guarantee that the deletions will take place atomically unless the operation is performed inside a transaction. Other processes querying the Datastore may see inconsistent results even when the query is performed with strong consistency.

delete_async (models, deadline=60)

Asynchronously deletes one or more model instances from the Datastore.

This function is identical to delete() except that it returns an asynchronous object. You can call get_result() on the return value to block on the call.

Arguments

models: A model instance, an entity key, or a list (or other iterable) of model instances or entity keys to delete.
deadline: Maximum time, in seconds, to wait for Datastore to return a result before aborting with an error. Accepts either an integer or a floating-point value. Cannot be set higher than the default value (60 seconds), but can be adjusted downward to ensure that a particular operation fails quickly (for instance, to return a faster response to the user, retry the operation, try a different operation, or add the operation to a task queue).

As with put(), if multiple keys are given, they may be in more than one entity group.

This function returns an object that lets you block on the result of the call.

get (keys, read_policy=STRONG_CONSISTENCY, deadline=60)

Fetch the specific Model instance(s) with the given key(s) from the Datastore.

Arguments

keys

Key of entity to be retrieved, a string representation of the key, or a list of keys or their string representations.

read_policy

Read policy specifying desired level of data consistency:

STRONG_CONSISTENCY: Guarantees the freshest results, but limited to a single entity group.
EVENTUAL_CONSISTENCY: Can span multiple entity groups, but may occasionally return stale results. In general, eventually consistent queries run faster than strongly consistent queries, but there is no guarantee.

Note: Global (non-ancestor) queries ignore this argument.

deadline

If keys consists of a single key (or its string representation), this function returns the model instance associated with the key if the key exists in the Datastore, otherwise None. If keys is a list, the return value is a corresponding list of model instances, with None values where no entity exists for a given key.

See also Model.get().

get_indexes ()

Returns a list of composite indexes belonging to the calling application.

The following example illustrates how to get and use the indexes:

def get_index_state_as_string(index_state):
  return {db.Index.BUILDING:'BUILDING', db.Index.SERVING:'SERVING',
          db.Index.DELETING:'DELETING', db.Index.ERROR:'ERROR'}[index_state]

def get_sort_direction_as_string(sort_direction):
  return {db.Index.ASCENDING:'ASCENDING',
          db.Index.DESCENDING:'DESCENDING'}[sort_direction]


def dump_indexes():
  for index, state in db.get_indexes():
    print "Kind: %s" % index.kind()
    print "State: %s" % get_index_state_as_string(state)
    print "Is ancestor: %s" % index.has_ancestor()
    for property_name, sort_direction in index.properties():
      print "  %s:%s" % (property_name,
                         get_sort_direction_as_string(sort_direction))

get_indexes_async ()

Asynchronously returns a list of composite indexes belonging to the calling application.

is_in_transaction ()

Returns a boolean indicating whether the current scope is executing in a transaction.

model_to_protobuf (model_instance)

Creates the protocol buffer serialization of a Model instance. A protocol buffer is Google's serialization format used for remote procedure calls, and can be useful for serializing Datastore objects for backup and restore purposes.

Caution: This function uses a different (older) format for protocol buffers than the open-source protocol buffer format, and is not compatible with the open-source implementation.

Argument

model_instance: The instance of class Model (or a subclass) to serialize.

Returns the protocol buffer serialization of the object, as a byte string.

model_from_protobuf (pb)

Creates a Model instance based on a protocol buffer serialization; see model_to_protobuf() for more information.

Argument

pb: The protocol buffer serialization, as returned by model_to_protobuf().

Returns an object of the appropriate kind class. If the kind class does not exist, raises a KindError exception. If the object is not valid according to the model, raises a BadValueError exception.

You can save the new object to the Datastore just like any other Model instance, such as by calling its put() method. The object retains the key it had when the protocol buffer was created. If an object with that key already exists in the Datastore, saving the deserialized object overwrites the existing object.

Caution: If the object's key uses a system-assigned ID and that ID has not already been allocated for the given path and kind, the save will succeed, but the ID is not reserved. An object created in the future may be assigned that ID, and would overwrite the earlier object. For safety, restore objects only in the same application where they existed when they were serialized.

model_is_projection (model_instance)

Returns True if the specified query (model_instance) is a projection query instead of a query for a full entity.

Argument

model_instance: The query you are checking to determine whether it is a projection query.

Returns True if the query is a projection query, False if it is not.

put (models, deadline=60)

Writes one or more model instances to the Datastore.

Arguments

models: A model instance or a list of model instances to store.
deadline: Maximum time, in seconds, to wait for Datastore to return a result before aborting with an error. Accepts either an integer or a floating-point value. Cannot be set higher than the default value (60 seconds), but can be adjusted downward to ensure that a particular operation fails quickly (for instance, to return a faster response to the user, retry the operation, try a different operation, or add the operation to a task queue).

If multiple model instances are given, they may be in more than one entity group.

An exception will always be raised if any error occurs during the operation, even if some of the entities actually were written. If the call returns without raising an exception, then all of the entities were written successfully.

If models consists of a single model instance, this function returns the corresponding Key object. If models is a list, the return value is a list of corresponding Key objects.

Caution: Writing multiple entities in a single operation does not guarantee that the writes will take place atomically unless the operation is performed inside a transaction. Other processes querying the Datastore may see inconsistent results even when the query is performed with strong consistency.

put_async (models, deadline=60)

Writes one or more model instances to the Datastore.

This function is identical to put() except that it returns an asynchronous object. You can call get_result() on the return value to block on the call and get the results.

Arguments

models: A model instance or a list of model instances to store.
deadline: Maximum time, in seconds, to wait for Datastore to return a result before aborting with an error. Accepts either an integer or a floating-point value. Cannot be set higher than the default value (60 seconds), but can be adjusted downward to ensure that a particular operation fails quickly (for instance, to return a faster response to the user, retry the operation, try a different operation, or add the operation to a task queue).

If multiple model instances are given, they may be in more than one entity group.

This function returns an asynchronous object on which get_result() can be called. The results returned are the same as for put().

query_descendants (model_instance)

Returns a query for all the descendants of a model instance.

Argument

model_instance: The model instance whose descendants you want to find.

run_in_transaction (function, *args, **kwargs)

Executes a function containing Datastore updates in a single transaction. If any code raises an exception during the transaction, all updates made in the transaction are rolled back. Alternatively, you can use the @db.transactional() decorator.

Arguments

function: Function to execute.
args: Positional arguments to pass to the function.
kwargs: Keyword arguments to pass to the function.

If the function returns a value, run_in_transaction() returns the value to the caller.

If the function raises an exception, the transaction is rolled back. If the exception is a Rollback exception, it is not re-raised; any other exception is re-raised to the caller.

The Datastore uses optimistic locking and retries for transactions. If the transaction prepared by the function cannot be committed, run_in_transaction() calls the function again, retrying the transaction up to 3 times. (To use a different number of retries, use run_in_transaction_custom_retries().) Because the transaction function may be called more than once for a single transaction, the function should not have side effects, including modifications to arguments.

If the transaction cannot be committed, such as because of a high rate of contention, a TransactionFailedError exception is raised.

from google.appengine.ext import db

class Counter(db.Model):
  name = db.StringProperty()
  count = db.IntegerProperty(default=0)

def decrement(key, amount=1):
  counter = db.get(key)
  counter.count -= amount
  if counter.count < 0:        # Don't let counter go negative
    raise db.Rollback()
  db.put(counter)

q = db.GqlQuery("SELECT * FROM Counter WHERE name = :1", "foo")
counter = q.get()
db.run_in_transaction(decrement, counter.key(), amount=5)

run_in_transaction_custom_retries (retries, function, *args, **kwargs)

Executes a function containing Datastore updates in a single transaction, retrying the transaction a specified number of times in the event of contention. If any code raises an exception during the transaction, all updates made in the transaction are rolled back.

Other than the ability to specify the number of retries, this function behaves identically to run_in_transaction().

Arguments

retries: Maximum number of times to call the function in the event of contention within the entity group (more than one user attempting to modify the group simultaneously).
function: Function to execute.
args: Positional arguments to pass to the function.
kwargs: Keyword arguments to pass to the function.

run_in_transaction_options (options, function, *args, **kwargs)

Executes a function containing Datastore updates in a single transaction using the options specified in a transaction options object. If any code raises an exception during the transaction, all Datastore updates made in the transaction are rolled back.

For cross-group (XG) transactions, the xg parameter in the transaction options object must be set to True.

Arguments

options: The transaction options object containing the settings used by this transaction. To enable XG transactions, its xg parameter must be set to True.
function: Function to execute.
args: Positional arguments to pass to the function.
kwargs: Keyword arguments to pass to the function.

If the function returns a value, run_in_transaction_options() returns the value to the caller.

If the function raises an exception, the transaction is rolled back. If the exception is a Rollback exception, it is not re-raised; any other exception is re-raised to the caller.

The Datastore uses optimistic locking and retries for transactions. If the transaction prepared by the function cannot be committed, run_in_transaction_options() calls the function again, retrying the transaction up to the number of retries specified in the transaction options object. Because the transaction function may be called more than once for a single transaction, the function should not have side effects, including modifications to arguments.

If the transaction cannot be committed, such as because of a high rate of contention, a TransactionFailedError exception is raised.

The following example shows how to use this function to run a cross-group transaction:

from google.appengine.ext import db

xg_options = db.create_transaction_options(xg=True)

def my_txn():
  x = MyModel(a=3)
  x.put()
  y = MyModel(a=7)
  y.put()

db.run_in_transaction_options(xg_options, my_txn)

to_dict (model_instance, dictionary=None)

Creates and returns a dictionary representation of a model instance.

Arguments

model_instance: Model instance to copy.
dictionary: If present, dictionary into which to merge the model's data. Model values overwrite values in the dictionary; dictionary entries not corresponding to fields in the model instance are preserved.

Decorators

@db.transactional (propagation=ALLOWED, xg=False, retries=3, deadline=60)

Makes a function run inside a db transaction. Thus, instead of calling run_in_transaction(func), you can call func().

Arguments

propagation

What to do if this transactional function is called from within another transaction:

ALLOWED: If already in a transaction, continue using it; if not, start one.
Note: If a function using this policy throws an exception, it is probably not safe to catch the exception and commit the outer transaction; the function may have left the outer transaction in a bad state.
MANDATORY: Continue in the existing transaction, if any; if not, throw a BadRequestError exception.
Note: If a function using this policy throws an exception, it is probably not safe to catch the exception and commit the outer transaction; the function may have left the outer transaction in a bad state.
INDEPENDENT: Create a new transaction, pausing any existing transaction.
Note: A function using this policy should not return any entities read in the new transaction, as the entities are not transactionally consistent with the outer transaction.
NESTED: (Not yet supported) Create a nested transaction within an existing one.

xg

If True, allow cross-group (XG) transactions. Raises a BadArgumentError exception if set to a nonboolean value.

retries

Number of retries to attempt on failure of transaction commit.

deadline

@db.non_transactional (allow_existing=True)

Ensures that a function is run outside a db transaction, even if called from within a transaction.

Argument

allow_existing: If True, allow function to be called from within an existing transaction; if False, throw a BadRequestError exception instead.