Common pitfalls and limitations

You might encounter the following issues and limitations when using Cloud Tasks:

Execution order

With the exception of tasks scheduled to run in the future, task queues are completely agnostic about execution order. There are no guarantees or best effort attempts made to execute tasks in any particular order. Specifically: there are no guarantees that old tasks will execute unless a queue is completely emptied. A number of common cases exist where newer tasks are executed sooner than older tasks, and the patterns surrounding this can change without notice.

Duplicate execution

Cloud Tasks aims for a strict "execute exactly once" semantic. However, in situations where a design trade-off must be made between guaranteed execution and duplicate execution, the service errs on the side of guaranteed execution. As such, a non-zero number of duplicate executions do occur. Developers should take steps to ensure that duplicate execution is not a catastrophic event. In production, more than 99.999% of tasks are executed only once.

Resource limitations

The most common source of backlogs in immediate processing queues is exhausting resources on the target instances. If a user is attempting to execute 100 tasks per second on frontend instances that can only process 10 requests per second, a backlog will build. This typically manifests in one of two ways, either of which can generally be resolved by increasing the number of instances processing requests.

Backoff errors and enforced rates

Servers that are being overloaded can start to return backoff errors in the form of HTTP response code 503. Cloud Tasks will react to these errors by slowing down execution until errors stop. This can be observed by looking at the "enforced rate" field in the Cloud Console.

Go to the Cloud Tasks page

Latency spikes and max concurrent

Overloaded servers can also respond with large increases in latency. In this situation, requests remain open for longer. Because queues run with a maximum concurrent number of tasks, this can result in queues being unable to execute tasks at the expected rate. Increasing the max_concurrent_tasks for the affected queues can help in situations where the value has been set too low, introducing an artificial rate limit. But increasing max_concurrent_tasks is unlikely to relieve any underlying resource pressure.