Announcing Cloud Tasks, a task queue service for App Engine flex and second generation runtimes
Morgan Hallmon
Product Manager
No application is an island—each one has dependencies and exists alongside other services, sharing resources and data. Today we are excited to announce Cloud Tasks, a fully managed, asynchronous task execution service that allows any application, running anywhere to use our standard API to add a task to the queue. Cloud Tasks enables serverless execution of your tasks via App Engine standard or flexible environments. With Cloud Tasks, you can offload long running and background activities, decouple services from one another, and make your applications much more resilient to failures. It is available today in beta for all Google Cloud Platform (GCP) users.
Cloud Tasks provides all the benefits of a distributed task queue: task offloading, loose coupling between services, and enhanced system reliability:
With task offloading, heavyweight and long running processes can be dispatched to a task queue, allowing your application to be more responsive for your users.
With loose coupling, services don’t talk to one another via direct request/response, but rather asynchronously. This allows them to scale independently.
Higher reliability and resilience comes from the fact that your tasks are persisted in storage and retried automatically, making your infrastructure resilient to intermittent failures.
Finally, Cloud Tasks does all this in a fully managed serverless fashion, with no manual intervention needed from the developer or operator. Customers also pay only for the operations they run. GCP takes care of all the provisioning of resources, replication and scaling required to operate Cloud Tasks. As a developer you simply add tasks to the queue and Cloud Tasks handles the rest.
Local Insights, a provider of aggregated real-estate information based in San Francisco, uses App Engine, and relies on Cloud Tasks as the backbone of its data transport architecture:
"Local Insights is the CarFax for properties. We deal with millions of real estate records every day, and we rely on Cloud Tasks to distribute the load of updating both our cold storage and user-facing search platform. Cloud Tasks seamlessly scales but also allows for throttling so as not to overwhelm our servers. It retries on failure, so our systems are sure to ingest important data, and because it's serverless, we spend more time delivering great products instead of handling DevOps. Cloud Tasks has become a critical component of our tech stack." - Mike Hardy, Founder, Local Insights
Made for second generation runtimes
We recently introduced second generation runtimes for App Engine. These runtimes allow you to easily build applications using up-to-date versions of most popular programming languages, with no restrictions on frameworks or supported libraries. Our hope is that these runtimes make App Engine more open, extensible and idiomatic. Cloud Tasks brings the long-held benefits of task queues to these new runtimes. In addition, it works with the App Engine flexible environment, supporting many thousands of existing applications.
How Cloud Tasks works:
Cloud Tasks operates in a push queue format, where tasks are dispatched to the worker services via HTTP requests according to a routing configuration. The worker then attempts to process them. If a task fails, an HTTP error is sent back to Cloud Tasks which then pauses and retries executing the service until a maximum number of attempts is reached. Once the execution is successful, the worker sends a 2xx success status code to Cloud Tasks.
The Cloud Tasks service handles all of the process management for the task, scaling workers up and down in relation to traffic and deleting tasks as they are completed.
Tasks Queue vs. Cloud Tasks
For many years, App Engine customers have leveraged Task Queue, an App Engine service that enables asynchronous task execution. With today’s release, all management of task queues is now centralized in the Cloud Tasks console, command line and API. However, existing App Engine customers can choose to continue managing their tasks via the App Engine SDK for convenience.
