Using Queue Management versus queue.yaml

This page explains the differences between using the Cloud Tasks API to manage queues and using the upload of a Cloud Tasks queue.yaml or queue.xml file to accomplish the same ends. It also discusses some of the pitfalls of mixing mechanisms and how to deal with common problems.

Introduction

The Cloud Tasks API provides an App Engine-independent interface to the App Engine Task Queue service. As part of that interface, it provides the ability to manage queues, including doing so via the console or thegcloud command. Queues that are created by Cloud Tasks API are accessible from the App Engine SDK and vice versa. To maintain compatibility, it is possible to use the configuration file used by the App Engine SDK, queue.yaml, to also create and configure queues to be used via the Cloud Tasks API. However, mixing configuration via file with configuration via the Cloud Tasks API can produce unexpected consequences.

Pitfalls of mixing queue.yaml with Cloud Tasks queue management methods

For the underlying service,queue.yaml files are definitive. Uploading a queue.yaml that omits existing queues in your project, no matter how they were created, causes those queues to be disabled, or paused. Thus if you use the Cloud Tasks API to call CreateQueue or UpdateQueue and then upload a queue.yaml file that omits them, the queues that were created in the Cloud Tasks calls are disabled.

Consider the following scenario:

  1. Call CreateQueue to create a queue named "cloud-tasks-queue".
  2. Upload a queue.yaml file with the following contents:

    queue:
    - name: queue-yaml-queue
    

What is the current state of queues in this project? The queue named "cloud-tasks-queue" and any other queues that existed prior are in DISABLED state, and the queue named "queue-yaml-queue" is in RUNNING state.

This behavior might be surprising if you create queues through the Cloud Tasks API. The instructions below explain how to resume a disabled queue.

Similarly, if a queue is disabled in the Cloud Tasks API but later appears in an uploaded queue.yaml file, that queue is resumed.

If a queue is deleted with the DeleteQueue method and later appears in a queue.yaml file, the queue.yaml upload can fail because queue names are not allowed to be reused for several days after deletion.

Best practices

If you are new to Cloud Tasks or App Engine, use the Cloud Tasks API exclusively to manage your queues and avoid the use of queue.yaml and queue.xml altogether. Cloud Tasks queue management methods give users more choice in creating, updating, and deleting queues.

If, however, you are an existing queue.yaml or queue.xml user, you should only consider switching to queue management methods if you understand the pitfalls of mixing queue.yaml with Cloud Tasks queue management methods.

To help enforce the use of only one configuration method, consider using groups and permissions to control access to queue management activities. See Securing Queue Configuration for instructions.

Debugging

You can inspect your project's Admin Activity audit logs to retrieve the history of queue configuration changes including queue creations, updates, and deletions:

    gcloud logging read \
      'protoPayload.methodName=
       (com.google.appengine.legacy.queue_created OR
        com.google.appengine.legacy.queue_updated OR
        google.cloud.tasks.v2.CloudTasks.CreateQueue OR
        google.cloud.tasks.v2.CloudTasks.UpdateQueue OR
        google.cloud.tasks.v2.CloudTasks.DeleteQueue)'

For example, if an existing queue is disabled by a queue.yaml upload, a "Disabled queue '[QUEUE_NAME]'" message would appear in the audit log via the com.google.appengine.legacy.queue_updated method.

How to resume a queue disabled by a queue.yaml upload

If you mix queue.yaml with Cloud Tasks queue management methods, uploading a queue.yaml file might accidentally disable a queue created through the Cloud Tasks API.

To resume the queue, you can either call ResumeQueue on the queue or add it to queue.yaml and upload. Be aware that if you had previously set a custom processing rate in the queue.yaml configuration for the queue, ResumeQueue resets the queue to the default rate. This is reflected in the maxDispatchesPerSecond field of the response to ResumeQueue.

Additional information about Cloud Tasks queue management methods

Queue configuration and queue startup delay

Changes to queue configuration can require several minutes to take effect. For example, after calling CreateQueue or UpdateQueue, several minutes might pass before you can successfully call CreateTask on that queue.

Cloud Tasks and the default App Engine queue

The App Engine queue named "default" is given special treatment in the App Engine SDK and in the Cloud Tasks API.

If the default queue does not already exist, it is created in the following situations:

  1. When a task is first added to the default queue using the App Engine SDK.
  2. When a queue.yaml file that specifies a default queue is uploaded.
  3. When CreateQueue or UpdateQueue is called to create the default queue.

To preserve compatibility with App Engine, Cloud Tasks enforces the following restrictions:

  1. If a queue named "default" is created, it must be a queue using App Engine tasks.
  2. Once created, users cannot delete the default queue.

In the Cloud Tasks API, the following also applies to the default queue:

  1. The Cloud Tasks API does not automatically create the default queue or any other queues.
  2. Just like any other queue, calling GetQueue on the default queue results in a not found error if the call is made before the queue is created.
  3. Similarly, the default queue does not appear in the output of ListQueues before it is created.
  4. The configuration of the default queue can be changed with the UpdateQueue call.

What's next

Was this page helpful? Let us know how we did:

Send feedback about...

Cloud Tasks Documentation