Creating Push Queues

This page describes how to create and customize a push queue, and how to examine the contents of a queue.

Using queue.xml to create queues

To process a task, you must add it to a push queue. App Engine provides a default push queue. If you add all your tasks to the default queue you do not need to create a queue explicitly. If you want to customize the performance settings of the default queue, or supply two or more queues to organize task requests, you can create as many push queues as you need.

You cannot create queues dynamically in your program. You must define them in a queue.xml file, which you upload to App Engine.

For example, this queue.xml file defines two queues:

<?xml version="1.0" encoding="UTF-8"?>
  <queue-entries>
    <queue>
      <name>queue-blue</name>
      <target>v2.task-module</target>
    </queue>
    <queue>
      <name>queue-red</name>
      <rate>1/s</rate>
    </queue>
  </queue-entries>

All tasks added to queue-blue are sent to the target module v2.task-module. The refresh rate of queue-red is changed from the default 5/s to 1/s. Tasks will be dequeued and sent to their targets at the rate of 1 task per second.

There are many other parameters that can be added to the queue.xml file to customize the behavior of a push queue. For more information, see the queue.xml reference.

Defining the push queue processing rate

You can control the rate at which tasks are processed in each of your queues by defining other directives, such as <rate>, <bucket-size>, and <max-concurrent-requests>.

The task queue uses token buckets to control the rate of task execution. Each named queue has a token bucket that holds a certain number of tokens, defined by the <bucket-size> directive. Each time your application executes a task, it uses a token. Your app continues processing tasks in the queue until the queue's bucket runs out of tokens. App Engine refills the bucket with new tokens continuously based on the <rate> that you specified for the queue.

If your queue contains tasks to process, and the queue's bucket contains tokens, App Engine processes as many tasks as there are tokens remaining in the bucket. This can lead to bursts of processing, consuming system resources and competing with user-serving requests.

If you want to prevent too many tasks from running at once or to prevent datastore contention, you use <max-concurrent-requests>.

The following samples shows how to set <max-concurrent-requests> to limit tasks and also shows how to adjust the bucket size and rate based on your application's needs and available resources:

<?xml version="1.0" encoding="UTF-8"?>
<queue-entries>
  <queue>
    <name>optimize-queue</name>
    <rate>20/s</rate>
    <bucket-size>40</bucket-size>
    <max-concurrent-requests>10</max-concurrent-requests>
  </queue>
</queue-entries>

Setting storage limits for all queues

You can use queue.xml to define the total amount of storage that task data can consume over all queues. To define the total storage limit, include an element named <total-storage-limit> at the top level:

<?xml version="1.0" encoding="UTF-8"?>
<queue-entries>
  <total-storage-limit>120M</total-storage-limit>
  <queue>
    <name>foor</name>
    <rate>35/s</rate>
  </queue>
</queue-entries>

The value is a number followed by a unit: B for bytes, K for kilobytes, M for megabytes, G for gigabytes, T for terabytes. For example, 100K specifies a limit of 100 kilobytes. If adding a task would cause the queue to exceed its storage limit, the call to add the task will fail. The default limit is 500M (500 megabytes) for free apps. For billed apps there is no limit until you explicitly set one. You can use this limit to protect your app from a fork bomb programming error in which each task adds multiple other tasks during its execution. If your app is receiving errors for insufficient quota when adding tasks, increasing the total storage limit can help. If you are using this feature, we strongly recommend setting a limit that corresponds to the storage required for several days' worth of tasks. In this way, your app is robust to its queues being temporarily backed up and can continue to accept new tasks while working through the backlog while still being protected from a fork bomb programming error.

Configuring the maximum number of concurrent requests

You can further control the processing rate by setting <max-concurrent-requests>, which limits the number of tasks that can execute simultaneously.

If your application queue has a rate of 20/s and a bucket size of 40, tasks in that queue execute at a rate of 20/s and can burst up to 40/s briefly. These settings work fine if task latency is relatively low; however, if latency increases significantly, you'll end up processing significantly more concurrent tasks. This extra processing load can consume extra instances and slow down your application.

For example, let's assume that your normal task latency is 0.3 seconds. At this latency, you'll process at most around 40 tasks simultaneously. But if your task latency increases to 5 seconds, you could easily have over 100 tasks processing at once. This increase forces your application to consume more instances to process the extra tasks, potentially slowing down the entire application and interfering with user requests.

You can avoid this possibility by setting <max-concurrent-requests> to a lower value. For example, if you set <max-concurrent-requests> to 10, our example queue maintains about 20 tasks/second when latency is 0.3 seconds. However, when the latency increases over 0.5 seconds, this setting throttles the processing rate to ensure that no more than 10 tasks run simultaneously.

<?xml version="1.0" encoding="utf-8"?>
<queue-entries>
  <queue>
    <name>optimize-queue</name>
    <rate>20/s</rate>
    <bucket-size>40</bucket-size>
    <max-concurrent-requests>10</max-concurrent-requests>
  </queue>
</queue-entries>

Monitoring queues in the Cloud Platform Console

The Task Queues page in the Cloud Platform Console. displays information about all the task queues in your application.

  1. Visit the Task Queues page in the Cloud Platform Console and select the Push Queues tab in the menu bar at the top of the page.

    Go to the Push Queues tab

  2. The Push Queues tab lists all of the queues in the application. Clicking on a queue name brings up the Task Queue Details page where you can see all of the tasks in the selected queue.

What's next