Executing asynchronous tasks

You can use Cloud Tasks to securely enqueue a task to be asynchronously processed by a Cloud Run service. Typical use cases include:

  • Preserving requests through unexpected production incidents
  • Smoothing traffic spikes by delaying work that is not user-facing
  • Speeding user response time by delegating slow background operations to be handled by another service, such as database updates or batch processing
  • Limiting the call rate to backing services like databases and third-party APIs

This page shows how to enqueue tasks that are securely pushed via the HTTPS protocol to a private Cloud Run service. It describes required behavior for the private Cloud Run service, required service account permissions, task queue creation, and task creation.

Before you start

Enable the Cloud Tasks API on the project you are using.

Deploying a Cloud Run service to handle tasks

To deploy a service that accepts tasks sent to the task queue, deploy the service in the same way as any other Cloud Run service. The Cloud Run service must return an HTTP 200 code to confirm proper processing of the task.

Tasks will be pushed to this Cloud Run service as HTTPS requests by Cloud Tasks.

Creating a task queue

To create a task queue, use the command

gcloud tasks queues create QUEUE-ID

replacing QUEUE-ID with the name you want to give to your task queue: it must be unique in your project. If you are prompted to create an App Engine app in your project, respond y to create it. Cloud Tasks uses this for the queue: make sure you choose the same location as you are using for your Cloud Run service.

The default task queue configuration should work in most cases. However, you can optionally set different rate limits and retry parameters if you want.

Creating a service account to associate with the tasks

You must create a service account that will be associated with the enqueued tasks. This service account must have the Cloud Run Invoker IAM role to allow the task queue to push tasks to the Cloud Run (fully managed) service. .

Console

  1. Visit the Create service account key page in the Cloud Console.

    Create service account page

  2. From the Service account list, select New service account.

  3. In the Service account name field, enter the name you want to use for the service account.

  4. Click Create.

  5. Copy the service account email to use in the following steps.

  6. Click Continue if prompted to specify permissions.

  7. Visit the Cloud Run Services page in the Cloud Console.

    Go to the Services page

  8. Select your service in the displayed list.

  9. If necessary, click the Show Info Panel/Hide Info Panel toggle in the far right of the page to show information.

  10. Locate the Permissions tab, and in that tab, click Add Member.

  11. Paste your service account email into the New members field.

  12. From the Role dropdown menu, select Cloud Run > Cloud Run Invoker.

  13. Click Save.

Command line

  1. Create the service account:

    gcloud iam service-accounts create SERVICE-ACCOUNT_NAME \
       --display-name "DISPLAYED-SERVICE-ACCOUNT_NAME"

    Replace

    • SERVICE-ACCOUNT_NAME with a lower case name unique within your Google Cloud project, for example my-invoker-service-account-name.
    • DISPLAYED-SERVICE-ACCOUNT-NAME with the name you want to display for this service account, for example, in the console, for example, My Invoker Service Account.
  2. For Cloud Run, give your service account permission to invoke your service:

    gcloud run services add-iam-policy-binding SERVICE \
       --member=serviceAccount:SERVICE-ACCOUNT_NAME@PROJECT-ID.iam.gserviceaccount.com \
       --role=roles/run.invoker

    Replace

    • SERVICE with the name of the service you want to be invoked by Cloud Tasks.
    • SERVICE-ACCOUNT_NAME with the name of the service account.
    • PROJECT-ID with your Google Cloud project ID.

Creating HTTP tasks with authentication tokens

When you create a task to send to the task queue, you specify the project, the location, queue name, the email of the previously created service account to associate with tasks, the URL of the private Cloud Run service that will run the task, and any other data you need to send.

Refer to the Cloud Tasks API documentation for details on the task request body. Note that requests that contain data payloads must use the HTTP PUT or POST method.

The code that enqueues the tasks must have the necessary IAM permissions to do so, such as the Cloud Tasks Enqueuer role. Your code will have the necessary IAM permissions if you use the default service account on Cloud Run (fully managed).

The following examples create task requests that also include the creation of a header token. OIDC tokens are used in the examples. To use an OAuth token, replace the OIDC parameter with the language appropriate OAuth parameter in constructing the request.

Python

"""Create a task for a given queue with an arbitrary payload."""

from google.cloud import tasks_v2
from google.protobuf import timestamp_pb2

# Create a client.
client = tasks_v2.CloudTasksClient()

# TODO(developer): Uncomment these lines and replace with your values.
# project = 'my-project-id'
# queue = 'my-queue'
# location = 'us-central1'
# url = 'https://example.com/task_handler'
# payload = 'hello'

# Construct the fully qualified queue name.
parent = client.queue_path(project, location, queue)

# Construct the request body.
task = {
        'http_request': {  # Specify the type of request.
            'http_method': 'POST',
            'url': url,  # The full url path that the task will be sent to.
            'oidc_token': {
                'service_account_email': service_account_email
            }
        }
}

if payload is not None:
    # The API expects a payload of type bytes.
    converted_payload = payload.encode()

    # Add the payload to the request.
    task['http_request']['body'] = converted_payload

if in_seconds is not None:
    # Convert "seconds from now" into an rfc3339 datetime string.
    d = datetime.datetime.utcnow() + datetime.timedelta(seconds=in_seconds)

    # Create Timestamp protobuf.
    timestamp = timestamp_pb2.Timestamp()
    timestamp.FromDatetime(d)

    # Add the timestamp to the tasks.
    task['schedule_time'] = timestamp

if task_name is not None:
    # Add the name to tasks.
    task['name'] = task_name

# Use the client to build and send the task.
response = client.create_task(parent, task)

print('Created task {}'.format(response.name))
return response

Note the requirements.txt file:

google-cloud-tasks==1.3.0
googleapis-common-protos==1.6.0

Java

import com.google.cloud.tasks.v2.CloudTasksClient;
import com.google.cloud.tasks.v2.HttpMethod;
import com.google.cloud.tasks.v2.HttpRequest;
import com.google.cloud.tasks.v2.OidcToken;
import com.google.cloud.tasks.v2.QueueName;
import com.google.cloud.tasks.v2.Task;
import com.google.protobuf.ByteString;
import java.nio.charset.Charset;

public class CreateHttpTaskWithToken {
  /**
   * Create a task with a HTTP target and authorization token using the Cloud Tasks client.
   *
   * @param projectId the Id of the project.
   * @param queueId the name of your Queue.
   * @param locationId the GCP region of your queue.
   * @param serviceAccountEmail your Cloud IAM service account
   * @throws Exception on Cloud Tasks Client errors.
   */
  public static void createTask(
      String projectId, String locationId, String queueId, String serviceAccountEmail)
      throws Exception {

    // Instantiates a client.
    try (CloudTasksClient client = CloudTasksClient.create()) {
      String url =
          "https://example.com/taskhandler"; // The full url path that the request will be sent to
      String payload = "Hello, World!"; // The task HTTP request body

      // Construct the fully qualified queue name.
      String queuePath = QueueName.of(projectId, locationId, queueId).toString();

      // Add your service account email to construct the OIDC token.
      // in order to add an authentication header to the request.
      OidcToken.Builder oidcTokenBuilder =
          OidcToken.newBuilder().setServiceAccountEmail(serviceAccountEmail);

      // Construct the task body.
      Task.Builder taskBuilder =
          Task.newBuilder()
              .setHttpRequest(
                  HttpRequest.newBuilder()
                      .setBody(ByteString.copyFrom(payload, Charset.defaultCharset()))
                      .setHttpMethod(HttpMethod.POST)
                      .setUrl(url)
                      .setOidcToken(oidcTokenBuilder)
                      .build());

      // Send create task request.
      Task task = client.createTask(queuePath, taskBuilder.build());
      System.out.println("Task created: " + task.getName());
    }
  }
}

Note the pom.xml file:

<?xml version="1.0" encoding="UTF-8"?>
<!--
Copyright 2018 Google LLC

 Licensed under the Apache License, Version 2.0 (the "License");
 you may not use this file except in compliance with the License.
 You may obtain a copy of the License at

       http://www.apache.org/licenses/LICENSE-2.0

 Unless required by applicable law or agreed to in writing, software
 distributed under the License is distributed on an "AS IS" BASIS,
 WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 See the License for the specific language governing permissions and
 limitations under the License.
-->
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <version>1.0-SNAPSHOT</version>
  <groupId>com.example.task</groupId>
  <artifactId>tasks-samples</artifactId>

  <!--
    The parent pom defines common style checks and testing strategies for our samples.
    Removing or replacing it should not affect the execution of the samples in anyway.
  -->
  <parent>
    <groupId>com.google.cloud.samples</groupId>
    <artifactId>shared-configuration</artifactId>
    <version>1.0.11</version>
  </parent>

  <properties>
    <maven.compiler.target>1.8</maven.compiler.target>
    <maven.compiler.source>1.8</maven.compiler.source>
    <failOnMissingWebXml>false</failOnMissingWebXml>
  </properties>

  <dependencies>
    <!-- Compile/runtime dependencies -->
    <dependency>
      <groupId>com.google.cloud</groupId>
      <artifactId>google-cloud-tasks</artifactId>
      <version>1.28.2</version>
    </dependency>
    <dependency>
      <groupId>com.google.protobuf</groupId>
      <artifactId>protobuf-java</artifactId>
      <version>3.11.4</version>
    </dependency>

    <!-- Test dependencies -->
    <dependency>
      <groupId>junit</groupId>
      <artifactId>junit</artifactId>
      <version>4.13</version>
    </dependency>
    <dependency>
      <groupId>com.google.truth</groupId>
      <artifactId>truth</artifactId>
      <version>1.0</version>
      <scope>test</scope>
    </dependency>
  </dependencies>

  <build>
    <plugins>
      <plugin>
        <artifactId>maven-assembly-plugin</artifactId>
        <version>3.2.0</version>
        <configuration>
          <descriptorRefs>
            <descriptorRef>jar-with-dependencies</descriptorRef>
          </descriptorRefs>
        </configuration>
        <executions>
          <execution>
            <id>make-assembly</id>
            <phase>package</phase>
            <goals>
              <goal>single</goal>
            </goals>
          </execution>
        </executions>
      </plugin>
    </plugins>
  </build>
</project>

Go

import (
	"context"
	"fmt"

	cloudtasks "cloud.google.com/go/cloudtasks/apiv2"
	taskspb "google.golang.org/genproto/googleapis/cloud/tasks/v2"
)

// createHTTPTaskWithToken constructs a task with a authorization token
// and HTTP target then adds it to a Queue.
func createHTTPTaskWithToken(projectID, locationID, queueID, url, email, message string) (*taskspb.Task, error) {
	// Create a new Cloud Tasks client instance.
	// See https://godoc.org/cloud.google.com/go/cloudtasks/apiv2
	ctx := context.Background()
	client, err := cloudtasks.NewClient(ctx)
	if err != nil {
		return nil, fmt.Errorf("NewClient: %v", err)
	}

	// Build the Task queue path.
	queuePath := fmt.Sprintf("projects/%s/locations/%s/queues/%s", projectID, locationID, queueID)

	// Build the Task payload.
	// https://godoc.org/google.golang.org/genproto/googleapis/cloud/tasks/v2#CreateTaskRequest
	req := &taskspb.CreateTaskRequest{
		Parent: queuePath,
		Task: &taskspb.Task{
			// https://godoc.org/google.golang.org/genproto/googleapis/cloud/tasks/v2#HttpRequest
			MessageType: &taskspb.Task_HttpRequest{
				HttpRequest: &taskspb.HttpRequest{
					HttpMethod: taskspb.HttpMethod_POST,
					Url:        url,
					AuthorizationHeader: &taskspb.HttpRequest_OidcToken{
						OidcToken: &taskspb.OidcToken{
							ServiceAccountEmail: email,
						},
					},
				},
			},
		},
	}

	// Add a payload message if one is present.
	req.Task.GetHttpRequest().Body = []byte(message)

	createdTask, err := client.CreateTask(ctx, req)
	if err != nil {
		return nil, fmt.Errorf("cloudtasks.CreateTask: %v", err)
	}

	return createdTask, nil
}

Node.js

// Imports the Google Cloud Tasks library.
const {CloudTasksClient} = require('@google-cloud/tasks');

// Instantiates a client.
const client = new CloudTasksClient();

// TODO(developer): Uncomment these lines and replace with your values.
// const project = 'my-project-id';
// const queue = 'my-queue';
// const location = 'us-central1';
// const url = 'https://example.com/taskhandler';
// const serviceAccountEmail = 'client@<project-id>.iam.gserviceaccount.com';
// const payload = 'Hello, World!';

// Construct the fully qualified queue name.
const parent = client.queuePath(project, location, queue);

const task = {
  httpRequest: {
    httpMethod: 'POST',
    url,
    oidcToken: {
      serviceAccountEmail,
    },
  },
};

if (payload) {
  task.httpRequest.body = Buffer.from(payload).toString('base64');
}

if (inSeconds) {
  // The time when the task is scheduled to be attempted.
  task.scheduleTime = {
    seconds: inSeconds + Date.now() / 1000,
  };
}

console.log('Sending task:');
console.log(task);
// Send create task request.
const request = {parent, task};
const [response] = await client.createTask(request);
const name = response.name;
console.log(`Created task ${name}`);

Note the package.json file:

{
  "name": "appengine-cloudtasks",
  "description": "Google App Engine Cloud Tasks example.",
  "license": "Apache-2.0",
  "author": "Google Inc.",
  "private": true,
  "engines": {
    "node": ">=8"
  },
  "scripts": {
    "test": "mocha",
    "start": "node server.js",
    "lint": "eslint '**/*.js'"
  },
  "dependencies": {
    "@google-cloud/tasks": "^1.7.3",
    "body-parser": "^1.18.3",
    "express": "^4.16.3",
    "yargs": "^15.0.0"
  },
  "devDependencies": {
    "chai": "^4.2.0",
    "mocha": "^7.0.0",
    "uuid": "^3.3.2"
  }
}

What's next