queue.yaml Reference

Applications define task queues in a configuration file called queue.yaml. You can use queue.yaml to configure push queues . This configuration file is optional for push queues, which have a default queue. Pull queues must be specifically configured in queue.yaml.

Example

The following a basic example that defines a named queue and overrides the default processing rate:

queue:
- name: my-push-queue
  rate: 1/s

The following is a more complex example of a queue.yaml configuration that demonstrates setting up task retries and modifying the default processing rate.

queue:
- name: fooqueue
  rate: 1/s
  retry_parameters:
    task_retry_limit: 7
    task_age_limit: 2d
- name: barqueue
  rate: 1/s
  retry_parameters:
    min_backoff_seconds: 10
    max_backoff_seconds: 200
    max_doublings: 0
- name: bazqueue
  rate: 1/s
  retry_parameters:
    min_backoff_seconds: 10
    max_backoff_seconds: 200
    max_doublings: 3

Syntax

The queue.yaml file is a YAML file whose root directive is queue. This directive contains zero or more named queues. Each queue definition can specify the following elements:

Element Description
acl (pull queues)

Experimental. Creates an access control list (ACL) for Pull Queues using the Task Queue REST API. The ACL is composed of the specified email addresses. Accepts email addresses only from a Google Account. Enter each email address on its own line as follows:

queue:
- name: my-pull-queue
  mode: pull
  acl:
  # can list, get, lease, delete, and update tasks:
  - user_email: bar@foo.com
  # can insert tasks:
  - writer_email: user@gmail.com
  # can insert tasks, in addition to rights granted by
  # being a user_email above:
  - writer_email: bar@foo.com

For more information, see Defining Pull Queues.

bucket_size (push queues)

Optional. Task queues use a "token bucket" algorithm for dequeueing tasks. The bucket size limits how fast the queue is processed when many tasks are in the queue and the rate is high. The maximum value for bucket size is 100. This allows you to have a high rate so processing starts shortly after a task is enqueued, but still limit resource usage when many tasks are enqueued in a short period of time.

If no bucket_size is specified for a queue, the default value is 5.

For more information on the algorithm, see the Wikipedia article on token buckets.

max_concurrent_requests (push queues)

Optional. Sets the maximum number of tasks that can be executed simultaneously from the specified queue. The value is an integer. By default, the limit is 1000 tasks per queue.

Restricting the number of concurrent tasks gives you more control over the queue's rate of execution and can prevent too many tasks from running at once. It can also prevent datastore contention and make resources available for other queues or online processing.

mode

Optional. Identifies the queue mode. This setting defaults to push, which identifies a queue as a push queue. If you wish to use pull queues, set the mode to pull.

name

Required. The name of the queue.

A queue name can contain uppercase and lowercase letters, numbers, and hyphens. The maximum length for a queue name is 100 characters.

All apps have a push queue named default. This queue has a preset rate of 5 tasks per second, but you can change this rate by defining a default queue in queue.yaml. If you do not configure a default queue in queue.yaml, the default queue doesn't display in the Google Cloud Platform Console until the first time it is used. You can customize the settings for this queue by defining a queue named default.

rate (push queues)

Optional. How often tasks are processed on this queue. The value is a number followed by a slash and a unit of time, where the unit is s for seconds, m for minutes, h for hours, or d for days. For example, the value 5/m says tasks will be processed at a rate of 5 times per minute.

If the number is 0 (such as 0/s), the queue is considered "paused," and no tasks are processed.

retry_parameters

Optional. Configures retry attempts for failed tasks. This addition allows you to specify the maximum number of times to retry failed tasks in a specific queue. You can also set a time limit for retry attempts and control the interval between attempts.

The retry parameters can contain the following subelements:

task_retry_limit
The maximum number of retry attempts for a failed task. If specified with task_age_limit, App Engine retries the task until both limits are reached. If 0 is specified, the task will not be retried.
task_age_limit (push queues)
The time limit for retrying a failed task, measured from when the task was first run. The value is a number followed by a unit of time, where the unit is s for seconds, m for minutes, h for hours, or d for days. For example, the value 5d specifies a limit of five days after the task's first execution attempt. If specified with task_retry_limit, App Engine retries the task until both limits are reached.
min_backoff_seconds (push queues)
The minimum number of seconds to wait before retrying a task after it fails.
max_backoff_seconds (push queues)
The maximum number of seconds to wait before retrying a task after it fails.
max_doublings (push queues)
The maximum number of times that the interval between failed task retries will be doubled before the increase becomes constant. The constant is: 2**(max_doublings - 1) * min_backoff_seconds.
target (push queues)

Optional. A string naming a module/version, a frontend version, or a backend, on which to execute all of the tasks enqueued onto this queue.

The string is prepended to the domain name of your app when constructing the HTTP request for a task. For example, if your app ID is my-app and you set the target to my-version.my-service, the URL hostname will be set to my-version.my-service.my-app.appspot.com.

If target is unspecified, then tasks are invoked on the same version of the application where they were enqueued. So, if you enqueued a task from the default application version without specifying a target on the queue, the task is invoked in the default application version. Note that if the default application version changes between the time that the task is enqueued and the time that it executes, then the task will run in the new default version.

If you are using modules along with a dispatch file, your task's HTTP request might be intercepted and re-routed to another module.

Send feedback about...

App Engine standard environment for Go