Task Queue Overview

Python |Java |PHP |Go

This page describes what task queues are, and when and how to use them. The Task Queue API lets applications perform work, called tasks, asynchronously outside of a user request. If an app needs to execute work in the background, it adds tasks to task queues. The tasks are executed later, by scalable App Engine worker modules in your application.

Push queues and pull queues

Task queues come in two flavors, push and pull. The manner in which the Task Queue service dispatches task requests to worker modules is different for the different queues.

Push queues dispatch requests at a reliable, steady rate. They guarantee reliable task execution. Because you can control the rate at which tasks are sent from the queue, you can control the workers' scaling behavior and hence your costs.

Because tasks are executed as App Engine requests targeted at modules, they are subject to stringent deadlines. Tasks handled by automatic scaling modules must finish in ten minutes. Tasks handled by basic and manual scaling modules can run for up to 24 hours.

Pull queues are not supported in PHP.

Tip: In some cases Google Cloud Pub/Sub is a good alternative to pull queues.

All task queue tasks are performed asynchronously. The application that creates the task is not notified whether or not the task completed, or if it was successful. The task queue service provides a retry mechanism, so if a task fails it can be retried a finite number of times.

Use cases

Push queues

One typical push queue use case is a "slow" operation. Consider a social network messaging system. Every time a user sends a message, the network needs to update the followers of the sender. This can be a very time-consuming operation. The application can enqueue a task for every message which will be delivered to a worker asynchronously. When the worker receives the task request, it can retrieve the sender's list of followers and update the DB for each one. The worker can be made even more efficient by enqueuing another push task for each database update.

Another use for push queues is scheduled tasks. Imagine an application that implements an ad campaign. A group of tasks written to send out emails can be added to a push queue with instructions to withhold the tasks until a specified time in the future. When the due date arrives, the Task Queue service will begin to issue requests to execute the tasks.

What's next

Send feedback about...

App Engine standard environment for PHP