Queue workflow executions using Cloud Tasks

You can use Cloud Tasks to queue a workflow and execute it asynchronously.

Cloud Tasks offers the following controls:

  • Schedule specific delivery times
  • Manage delivery rates
  • Configure retry behavior
  • Access and manage individual tasks in a queue
  • Enable task deduplication

These controls can be useful, for example, to manage requests that trigger a workflow and that could unexpectedly exceed Workflows limits. By configuring a Cloud Tasks queue with rate limits and retries, you can ensure that all such requests result in workflow executions.

For more information, see Understand Cloud Tasks.

Note the following:

  • You should base the rate limit on the Workflows limit for Execution API write requests. If necessary, you can request an increase to most quotas in the Google Cloud console. Find out more about quota increase requests.

  • Cloud Tasks is designed to provide "at least once" delivery; however, Workflows doesn't ensure exactly-once processing of duplicate requests from Cloud Tasks.

  • You can remove execution delays and maximize throughput by enabling execution backlogging in Workflows. Backlogged executions automatically run as soon as execution concurrency quota becomes available. For more information, see Manage execution backlogging.

Before you begin

  1. If you don't already have a workflow that you would like to queue, create one.

  2. Enable the Cloud Tasks API.

    Enable the API

Queue a task to execute a workflow

  1. Create a service account so that Cloud Tasks can make requests to the Workflows API:

    gcloud iam service-accounts create SERVICE_ACCOUNT_NAME

    Replace SERVICE_ACCOUNT_NAME with a name that is between 6 and 30 characters. It can contain lowercase alphanumeric characters and dashes. After you create a service account, you cannot change its name.

  2. To allow the principal that will run your Cloud Tasks commands the ability to act as an Identity and Access Management (IAM) service account, grant a role that allows the principal to impersonate the service account.

  3. Grant your new service account the workflows.invoker role so that the account has permission to trigger your workflow:

    gcloud projects add-iam-policy-binding PROJECT_NAME \
    --member serviceAccount:SERVICE_ACCOUNT_NAME@PROJECT_NAME.iam.gserviceaccount.com \
    --role roles/workflows.invoker

    Replace the following:

    • PROJECT_NAME: the name of your Google Cloud project.
    • SERVICE_ACCOUNT_NAME: the name of the service account that you previously created.

  4. If you don't already have a Cloud Tasks queue, create one.

  5. Create a task that targets your workflow's HTTP endpoint, using the service account you previously created to authenticate:

    gcloud tasks create-http-task \
    --queue="QUEUE_ID" \
    --url="https://workflowexecutions.googleapis.com/v1/projects/PROJECT_NAME/locations/REGION_NAME/workflows/WORKFLOW_NAME/executions" \
    --body-content="{\"argument\": \"DOUBLE_ESCAPED_JSON_STRING\"}" \
    --oauth-service-account-email="SERVICE_ACCOUNT_NAME@PROJECT_NAME.iam.gserviceaccount.com"

    Replace the following:

    • QUEUE_ID: the identifier you assigned to your Cloud Tasks queue; for example, my-queue.
    • PROJECT_NAME: the name of your Google Cloud project.
    • REGION_NAME: the region your workflow is in, such as us-central1.
    • WORKFLOW_NAME: the name of the workflow you want to create a task for.
    • DOUBLE_ESCAPED_JSON_STRING: a JSON encoding of any arguments you are passing. The double quotation marks inside the quoted string are escaped using backslashes (\). For example: --body-content="{\"argument\": \"{\\\"foo\\\": \\\"bar\\\"}\"}"
    • SERVICE_ACCOUNT_NAME: the name of the service account that you previously created.

    Your task is created and added to your Cloud Tasks queue. The task persists in the queue until the workflow execution starts and returns an HTTP 2xx status code, at which point Cloud Tasks considers the task complete.

What's next