Task Queue Overview

Python |Java |PHP |Go

This page describes what task queues are, and when and how to use them. The Task Queue API lets applications perform work, called tasks, asynchronously outside of a user request. If an app needs to execute work in the background, it adds tasks to task queues. The tasks are executed later, by scalable App Engine worker modules in your application.

Push queues and pull queues

Task queues come in two flavors, push and pull. The manner in which the Task Queue service dispatches task requests to worker modules is different for the different queues.

Push queues dispatch requests at a reliable, steady rate. They guarantee reliable task execution. Because you can control the rate at which tasks are sent from the queue, you can control the workers' scaling behavior and hence your costs.

Because tasks are executed as App Engine requests targeted at modules, they are subject to stringent deadlines. Tasks handled by automatic scaling modules must finish in ten minutes. Tasks handled by basic and manual scaling modules can run for up to 24 hours.

Pull queues do not dispatch tasks at all. They depend on other worker modules to "lease" tasks from the queue on their own initiative. Pull queues give you more power and flexibility over when and where tasks are processed, but they also require you to to do more queue management. When a task is leased the leasing program declares a deadline. The task must either complete or be deleted by the time the deadline arrives, otherwise the Task Queue service will allow it to be leased to another worker.

Tip: In some cases Google Cloud Pub/Sub is a good alternative to pull queues.

All task queue tasks are performed asynchronously. The application that creates the task is not notified whether or not the task completed, or if it was successful. The task queue service provides a retry mechanism, so if a task fails it can be retried a finite number of times.

Use cases

Push queues

One typical push queue use case is a "slow" operation. Consider a social network messaging system. Every time a user sends a message, the network needs to update the followers of the sender. This can be a very time-consuming operation. The application can enqueue a task for every message which will be delivered to a worker asynchronously. When the worker receives the task request, it can retrieve the sender's list of followers and update the DB for each one. The worker can be made even more efficient by enqueuing another push task for each database update.

Another use for push queues is scheduled tasks. Imagine an application that implements an ad campaign. A group of tasks written to send out emails can be added to a push queue with instructions to withhold the tasks until a specified time in the future. When the due date arrives, the Task Queue service will begin to issue requests to execute the tasks.

Pull queues

Pull queues work well when you need to batch tasks together for efficient execution. One solution takes advantage of the ability to attach a tag to a pull task. Workers can lease a group of tasks that have the same tag. A typical example is an app that maintains leaderboards for many different games, with many players and groups constantly in play. Every time there is a new high score, the app can enqueue a pull task with the score and the player, and use the game ID as a task tag. A worker periodically "wakes up", leases a group of tasks with the same game ID, and updates the leaderboard. You can lease tasks explicity, using a specified tag value, or let the service decide which group of similarly tagged tasks to send.

Batching with tags can be very powerful. Since tags can be dynamically generated while your app is running, a worker can handle new game IDs with no special effort. If you used queues with names to batch scores by game ID, you’d have to update the app code on both ends.

What's next

Send feedback about...