Cloud Tasks overview

This page describes what Cloud Tasks tasks and queues are, and when and how to use them. Cloud Tasks lets you separate out pieces of work that can be performed independently, outside of your main application flow, and send them off to be processed, asynchronously, using handlers that you create. These independent pieces of work are called tasks. For example, you need to update a database as part of processing a user request, but updates can be time-consuming. Offloading that detail as a task allows you to return from the request more quickly.

The offloaded task is added to a queue, which persists the task until it is successfully executed. The queue can also act as a kind of dispatch flow control, based on your initial configuration. You create and configure the queue which is then managed by the Cloud Tasks service. Once tasks are added, the queue dispatches them and makes sure they are reliably processed by your workers. Complexities associated with that process, such as user-facing latency costs, server crashes, resource consumption limitations, and retry management, are handled by the service.

The tasks themselves are made up of a unique name and configuration information and, optionally, any data from the initial request, called the payload, necessary to process the request. As the payload is sent in the request body, tasks that include payloads must use POST or PUT as their HTTP method.

Users of App Engine standard environment first generation runtimes should access the Cloud Tasks service via the App Engine Task Queue API. For more information on using this method, see the Java 8, Python 2.7, Go 1.9, or PHP 5.5 documentation. Users of App Engine standard environment second generation runtimes, flexible environment, and those on other platforms can now use Cloud Tasks API .

To access the Cloud Tasks service using Cloud Tasks API , you must have a project that contains an App Engine app that hosts the Cloud Tasks queues that you create. This app is located in a specific region which serves as the LOCATION_ID parameter for your Cloud Tasks requests, so make a note of it. Note that two locations, called europe-west and us-central in App Engine commands, are called, respectively, europe-west1 and us-central1 in Cloud Tasks commands. The app serves as the location for whatever queues the developer creates. The underlying Cloud Tasks service itself runs in that same location.

Use cases

Typical use cases include:

  • Speeding user response times by delegating potentially slow background operations like database updates to a worker
  • Preserving requests in the context of unexpected production incidents
  • Helping smooth traffic spikes by removing non-user-facing tasks from the main user flow
  • Managing third-party API call rates

Cloud Tasks queues with HTTP targets

In the case of generic HTTP Targets, the Cloud Tasks service forwards the task request to the worker, located at any generic HTTP endpoint, such as Cloud Functions, Cloud Run, GKE, Compute Engine, or even an on-prem web server, based on how the task is configured. These queues dispatch requests at a reliable, configurable rate. They guarantee reliable task execution - upon success, all workers must send an HTTP response code (200-299) to the Cloud Tasks service before a 10 minute deadline. If a different response is sent, or no response, the task is retried.

HTTP-based queues

The target must manage scaling workers and cleaning up tasks once they are complete.

If your target requires authentication, you must set up two service accounts, one for your application, the client, and one for the queue itself. Both accounts must have been granted the requisite permissions, and an identifier for the client service account must be included in the task request. See Creating HTTP Target Tasks for more information.

Cloud Tasks queues with App Engine targets

In the case of App Engine targets, the Cloud Tasks service also forwards the task request to the worker, located within App Engine, based on how the task (or, less commonly, the queue itself) is configured. These queues dispatch requests at a reliable, configurable rate. They guarantee reliable task execution - upon success, all workers must send an HTTP response code (200-299) to the Cloud Tasks service, in this instance before a deadline based on the instance scaling type of the service: 10 minutes for automatic scaling or up to 24 hours for manual scaling. If a different response is sent, or no response, the task is retried.

App Engine-based queues

Because the handlers are part of App Engine, the Cloud Tasks service itself can do much of the process management for the task, scaling workers up and down in relation to traffic and deleting tasks when they are completed.

Workflows

The general workflow is as follows:

  1. You create a worker to process the tasks.
  2. You create a queue.
  3. You create tasks programmatically and add them to the queue.
  4. The Cloud Tasks service returns an OK to the originating application. This indicates that the task has been successfully written to Cloud Tasks storage, making the create task request both highly available and durable.
  5. The tasks are passed to the worker.
  6. The worker processes the task.
  7. To complete the sequence, the worker returns a 2xx success status code to the Cloud Tasks service.

Once the task has been handed off to the queue, no data is made available to the initial request.

Features

Using Cloud Tasks, you can dispatch asynchronous work items with the following controls:

  • Schedule specific delivery times
  • Manage delivery rates
  • Configure retry behavior
  • Access and manage individual tasks in a queue
  • Enable task deduplication

Terms

The following table shows key terms that describe aspects of Cloud Tasks behavior.

Term Definition
queue A set of tasks with the same target type managed by a single configuration.
target type Where and how a task is processed.
worker A service which processes tasks.
attempt An attempt to run a task.
attempt dispatch The moment when Cloud Tasks has sent the task to its target.
attempt response A response from a worker which indicates that the work associated with the task completed successfully or failed.
retry Multiple attempts to run a task. The number of retries is set via the RetryConfig.
rate limits The rate limits for a queue.

Metrics

The following predefined Cloud Task metrics are available via Stackdriver.

Metric type
Display name
Kind, Type, Unit
Description
Labels
api/request_count
API requests
DELTA, INT64, 1
Count of Cloud Tasks API calls.
api_method: The API method called (e.g. CreateTask).
response_code: Canonical response code as string (e.g. 'ok').
queue/depth
BETA Queue depth
GAUGE, INT64, 1
The number of tasks in the queue. Sampled every 60 seconds. After sampling, data is not visible for up to 120 seconds.
queue/task_attempt_count
Task attempt count
DELTA, INT64, 1
Count of task attempts broken down by response code.
response_code: Canonical response code as string (e.g. 'ok').
queue/task_attempt_delays
Task attempt delays
DELTA, DISTRIBUTION, ms
Delay between each scheduled attempt time and actual attempt time.