This page describes how to create tasks and place them in push queues. When you want to process a task, you must create a new task object and place it on a queue. You can explicitly specify the service and handler that process the task, and optionally pass task-specific data along to the handler. You can also fine-tune the configuration for the task, like scheduling a time in the future when it should be executed or limiting the number of times you want the task to be retried if it fails.
Creating a new task
To create and enqueue a task, call the taskqueue.add() function.
The following code creates a task that targets the service named worker
and
invokes its handler by setting the url /update-counter
:
Alternatively, you can create a Task object and call its add() method.
Specifying the worker service
When a task is popped off its queue, the Task Queue service sends it on to a worker service. Every task has a target and a url, which determine what service and handler will ultimately perform the task.
target
The target specifies the service that will receive the HTTP request to perform the task. It is a string that specifies a service/version/instance in any one of the canonical forms. The most often-used forms are:
service
version.service
instance.version.service
The target string is prepended to the domain name of your app. There are three ways to set the target for a task:
Declare the target when you construct the task. You can set the target explicitly using the
target
parameter in the taskqueue.add() function. See the example above.Include a
target
directive when you define a queue in thequeue.yaml
, as in the definition ofqueue-blue
. All tasks added to a queue with atarget
will use that target, even if a different target was assigned to the task at construction time.If no target is specified according to either of the previous two methods, then the task's target is the version of the service that enqueues it. Note that if you enqueue a task from the default service and version in this manner, and the default version changes before the task executes, it will run in the new default version.
url
The url
selects one of the handlers in the target service, which will
perform the task.
The url
should match one of the handler URL patterns in the target
service. The url
can include query parameters if the method specified in the task is
GET
or PULL
. If no url
is specified the default URL
/_ah/queue/[QUEUE_NAME]
is used, where [QUEUE_NAME]
is the name of
the task's queue.
Passing data to the handler
You can pass data to the handler as query parameters in the task's URL, but
only if the method specified in the task is GET
or PULL
.
payload
, which delivers task data in the body of the HTTP request.params
These three calls are equivalent:
taskqueue.add(method=GET, url='/update-counter?key=blue', target='worker')
taskqueue.add(url='/update-counter', params={'key': 'blue'}, target='worker')
taskqueue.add(url='/update-counter', payload="{'key': 'blue'}", target='worker')
Naming a task
When you create a new task, App Engine assigns the task a unique name by
default. However, you can assign your own name to a task by using the name
parameter. An advantage of assigning your own task names is that named tasks are
de-duplicated, which means you can use task names to
guarantee
that a task is only added once. De-duplication continues for 9 days after the
task is completed or deleted.
Note that de-duplication logic introduces significant performance overhead, resulting in increased latencies and potentially increased error rates associated with named tasks. These costs can be magnified significantly if task names are sequential, such as with timestamps. So, if you assign your own names, we recommend using a well-distributed prefix for task names, such as a hash of the contents.
If you assign your own names to tasks, note that the maximum name length is 500 characters, and the name can contain uppercase and lowercase letters, numbers underscores, and hyphens.
taskqueue.add(url='/url/path', name='first-try')
Adding tasks asynchronously
By default, the calls that add tasks to queues are synchronous. For most scenarios, synchronous calls work fine. Adding a task to a queue is usually a fast operation. There are a small percentage of add task operations that can take significantly longer, but the median time to add a task is less than 5 ms.
Add task operations to different queues cannot be batched, so the Task Queue API also provides asynchronous calls that give you the ability to add these tasks in parallel, further minimizing this latency. This is useful if you are building an extremely latency sensitive application that needs to perform several add task operations to different queues at the same time.
If you want to make asynchronous calls to a task queue, use the asynchronous
methods provided by the
Queue class and an
RPC object. Call get_result()
on
the returned RPC
object to force the request to complete. When asynchronously
adding tasks in a transaction, you should call get_result()
on the RPC
object before committing the transaction to ensure that the request has finished
.
Enqueuing tasks in Cloud Datastore transactions
You can enqueue a task as part of a Datastore transaction, such that the task is only enqueued—and guaranteed to be enqueued—if the transaction is committed successfully. Tasks added in a transaction are considered to be a part of it and have the same level of isolation and consistency.
An application cannot insert more than five transactional tasks into task queues during a single transaction. Transactional tasks must not have user-specified names.
The following code sample demonstrates how to insert transactional tasks into a push queue as part of a Datastore transaction:
from google.appengine.api import taskqueue
from google.appengine.ext import ndb
@ndb.transactional
def do_something_in_transaction():
taskqueue.add(url='/path/to/my/worker', transactional=True)
#...
do_something_in_transaction()
Using the deferred task library instead of a worker service
Setting up a handler for each distinct task (as described in the previous
sections) can be cumbersome, as can serializing and deserializing complex
arguments for the task—particularly if you have many diverse but small tasks
that you want to run on the queue. The Python SDK includes a library
(google.appengine.ext.deferred
) exposing a simple function that allows you to
bypass all the work of setting up dedicated task handlers and serializing and
deserializing your parameters.
To use this library, you need to add the deferred
builtin to app.yaml
. For
more information, see the
Built-in Handlers section of
the app.yaml
reference.
To use the deferred
library, simply pass the function and its arguments to
deferred.defer()
:
import logging
from google.appengine.ext import deferred
def do_something_expensive(a, b, c=None):
logging.info("Doing something expensive!")
# Do your work here
# Somewhere else
deferred.defer(do_something_expensive, "Hello, world!", 42, True)
The deferred
library packages your function call and its arguments, then adds
it to the task queue. When the task is executed, the deferred
library executes
do_something_expensive("Hello, world!", 42, True)
.
Working with tasks in a multi-tenant application
By default, push queues use the current namespace as set in the namespace manager at the time the task is created. If your application uses multitenancy, see the Namespaces Python 2 API.
What's next
- Learn how to create task handlers.