This page describes some of the issues and limitations that you might encounter when using Cloud Tasks.
Execution order
With the exception of tasks scheduled to run in the future, task queues are completely platform-independent about execution order. There are no guarantees or best effort attempts made to execute tasks in any particular order. Specifically: there are no guarantees that old tasks will execute unless a queue is completely emptied. A number of common cases exist where newer tasks are executed sooner than older tasks, and the patterns surrounding this can change without notice.
Duplicate execution
Cloud Tasks aims for a strict "execute exactly once" semantic. However, in situations where a design trade-off must be made between guaranteed execution and duplicate execution, the service errs on the side of guaranteed execution. As such, a non-zero number of duplicate executions do occur. Developers should take steps to ensure that duplicate execution is not a catastrophic event. In production, more than 99.999% of tasks are executed only once.
Resource limitations
The most common source of backlogs in immediate processing queues is exhausting resources on the target instances. If a user is attempting to execute 100 tasks per second on frontend instances that can only process 10 requests per second, a backlog will build. This typically manifests in one of two ways, either of which can generally be resolved by increasing the number of instances processing requests.
Backoff errors and enforced rates
Servers that are being overloaded can start to return backoff errors: HTTP 503
(for App Engine targets), or HTTP 429 or 5xx (for external targets).
Cloud Tasks reacts to these errors by slowing down execution until
errors stop. This system throttling prevents the worker from overloading. Note
that user-specified settings are not changed.
System throttling happens in the following circumstances:
Cloud Tasks backs off on all errors. Normally the backoff specified in
rate limitsis used. But if the worker returns HTTP429 Too Many Requests,503 Service Unavailable, or the rate of errors is high, Cloud Tasks uses a higher backoff rate. The retry specified in theRetry-AfterHTTP response header is considered.To prevent traffic spikes and to smooth sudden increases in traffic, dispatches ramp up slowly when the queue is newly created or idle, and if large numbers of tasks suddenly become available to dispatch (due to spikes in create task rates, the queue being unpaused, or many tasks that are scheduled at the same time).
Latency spikes and max concurrent
Overloaded servers can also respond with large increases in latency.
In this situation, requests remain open for longer. Because queues run with a
maximum concurrent number of tasks, this can result in queues being unable to
execute tasks at the expected rate. Increasing the
max_concurrent_tasks
for the affected queues can help in situations where the value has been set too
low, introducing an artificial rate limit. But increasing max_concurrent_tasks
is unlikely to relieve any underlying resource pressure.
Ramp up issues with long running tasks
Cloud Tasks queues ramp up their output in part based on the number of successful previously dispatched tasks. If the task handler takes a considerable period of time - on the order of minutes - to complete a task and return a success response, there can be a lag in the queue ramp up rate.
Viewing more than 5,000 tasks
If you have more than 5,000 tasks, some tasks are not visible in the Google Cloud console. Use gcloud CLI to view all tasks.