Getting Started: Task Queues

Learn how to use task queues and the App Engine Image API to resize images.

Task queues execute code outside of direct user interaction, allowing tasks to happen in the background. This guide uses a Task Queue to perform tasks after adding an image to Cloud Storage. The tasks to be performed in the Task Queue are:

  1. Retrieve the image file that just got uploaded to Cloud Storage.
  2. Resize it to a thumbnail image using the Image API.
  3. Store the resulting thumbnail in Cloud Storage.

The App Engine Java 8 runtime also supports Java's native image manipulation classes such as AWT and Java2D.

Before you begin

  1. Configure your development environment and create your App Engine project.

  2. This guide uses the Apache Commons IOUtils library. To include the IOUtils library to your App Engine project:

    Add to your pom.xml:

    <dependency>
  <groupId>commons-io</groupId>
  <artifactId>commons-io</artifactId>
  <version>2.5</version>
</dependency>

Importing libraries

The sample code provided with this guide uses the following imports:

import com.google.appengine.api.images.Image;
import com.google.appengine.api.images.ImagesService;
import com.google.appengine.api.images.ImagesServiceFactory;
import com.google.appengine.api.images.Transform;
import org.apache.commons.io.IOUtils;

Creating a task queue

While App Engine provides a default task queue, you can create different task queues for different types of work. For example, you can create one task queue to resize images and another to update your app's database.

To add queues, create the queue.xml file in your App Engine project's WEB-INF directory. A taskqueue must specify a name and an execution rate:

<?xml version="1.0" encoding="UTF-8"?>
  <queue-entries>
    <queue>
      <name>resize-image</name>
      <rate>60/h</rate>
    </queue>
  </queue-entries>

This example queue, named resize-image, defines a rate of execution of 60 times per hour, or once a minute. To see the full list queue options see the queue.xml reference.

A task queue has two components, the task requester and a task handler. The requestor adds a task to the queue and sends it to the task handler.

Adding tasks to a queue

To add a task to a queue:

  1. Create a task queue object using QueueFactory.getQueue(), making sure you specify the queue name defined in the queue.xml:

    Queue imageResizeQueue; // Taskqueue queue

@Override
public void init() throws ServletException {

  // Setup Cloud Storage service
  gcsService =
      GcsServiceFactory.createGcsService(
          new RetryParams.Builder()
              .initialRetryDelayMillis(10)
              .retryMaxAttempts(10)
              .totalRetryPeriodMillis(15000)
              .build());

  // Initialize the queue object with the specific queue
  imageResizeQueue = QueueFactory.getQueue([QUEUE-NAME]);

  // Cloud SQL connection setup
  try {
    final String url = System.getProperty("cloudsql"); // Cloud SQL server URI

    try {
      conn = DriverManager.getConnection(url); // Connect to the database

      Statement createTable; // SQL statement

      // Batch SQL table creation commands
      createTable.addBatch(createContentTableSql);
      createTable.addBatch(createUserTableSql);
      createTable.addBatch(createImageTableSql);
      createTable.addBatch(createBlogPostImageTableSql);
      conn.createTable.executeBatch(); // Execute batch

    } catch (SQLException e) {
      throw new ServletException("Unable to connect to Cloud SQL", e);
    }

  } finally {
    // Nothing really to do here.
  }

}

  2. Add tasks to Queue object. As shown in the code sample, imageResizeQueue.add() adds a task to the imageResizeQueue object:

    try {
  // Add a queued task to create a thumbnail of the uploaded image
  imageResizeQueue.add(
      TaskOptions.Builder.withUrl("/tasks/imageresize").param("filename", filename));
}

    Specify the URI of the task handler using TaskOptions.Builder.withUrl(), along with and any parameters sent to the handler.

    In this example, the URI is /tasks/imageresize and the parameter is a variable called filename that contains the filename of the image to be processed.

Creating a task handler

Once you have added a task to the queue, the task handler mapped to the URI /tasks/imageresize will run. A task handler is a Java Servlet that attempts to perform the task until it is successful.

In this example, the task handler does three tasks:

  • Retrieve the image specified by the caller from Cloud Storage.

  • Transform the image using the App Engine Image API, in this sample, to a thumbnail image.

  • Store the transformed image (thumbnail) in Cloud Storage.

To create the task handler:

  1. Add an annotation that maps the handler to the URI /tasks/imageresize:

     @WebServlet(name = "imageResize", description = "Task queue handler", urlPatterns = "/tasks/imageresize")
 public class imageResize extends HttpServlet {

   // Task handler functionality

 }

  2. Set up a connection to Cloud Storage as documented in the Using Cloud Storage guide and retrieve the image from Cloud Storage:

     public void init() throws ServletException {

  // initiate GcsService
  GcsService gcsService =
    GcsServiceFactory.createGcsService(
        new RetryParams.Builder()
            .initialRetryDelayMillis(10)
            .retryMaxAttempts(10)
            .totalRetryPeriodMillis(15000)
            .build());
}

  3. Handle the incoming Task Queue request, using the supplied file name to retrieve the image from Cloud Storage:

    protected void doPost(HttpServletRequest req, HttpServletResponse resp)
    throws ServletException, IOException {

  String filename = req.getParameter("filename"); // Get the filename passed from the task requestor
  GcsFilename gcsFile = new GcsFilename(bucket, filename); // Create a valid Cloud Storage filename

  GcsInputChannel readChannel = gcsService.openPrefetchingReadChannel(gcsFile, 0, BUFFER_SIZE); // Read the file from Cloud Storage

  4. Use the ImagesService object to do the image resizing:

    // Get an instance of the ImagesService we can use to transform images.
ImagesService imagesService = ImagesServiceFactory.getImagesService();

// Make an image directly from a byte array, and transform it.
Image image =
    ImagesServiceFactory.makeImage(IOUtils.toByteArray(Channels.newInputStream(readChannel)));
Transform resize = ImagesServiceFactory.makeResize(100, 50); // resize image to 100x50
Image resizedImage = imagesService.applyTransform(resize, image);

// Write the transformed image back to a Cloud Storage object.
gcsService.createOrReplace(
    new GcsFilename(bucket, "thumbnail_" + filename),
    new GcsFileOptions.Builder().acl("public-read").build(),
    ByteBuffer.wrap(resizedImage.getImageData()));

    The above snippet uses the Image API makeResize() method to resize the image to a thumbnail. To do this, it reads the image from Cloud Storage into an InputChannel and convert it to a ByteArray using IOUtils.toByteArray().

    After applying the transformation, the new image has the string thumbnail_ appended to its filename, permission set to be publicly readable and written to Cloud Storage.

Securing task handler URLs

You should secure tasks that perform sensitive operations such as modifying data so external users cannot call it directly. You can do this by restricting task access to App Engine administrators, which prevents users from accessing task URLs . Note that this restriction does not apply to task requests coming from your App Engine app.

In the current example, the task handlers have URLs in the /tasks/ folder. To restrict access to the /tasks/ folder to App Engine administrators, add the following to the project's web.xml.

<security-constraint>
    <web-resource-collection>
        <web-resource-name>tasks</web-resource-name>
        <url-pattern>/tasks/*</url-pattern>
    </web-resource-collection>
    <auth-constraint>
        <role-name>admin</role-name>
    </auth-constraint>
</security-constraint>

Removing a single task from a queue

To remove a single task from a queue, use deleteTask():

private void removeTask(Queue queue, String taskName) {
  queue.deleteTask(taskName); // remove a task from a queue
}

Removing all tasks from a queue

To remove all tasks from a queue, use purge(). The purge can take up to a minute to remove all tasks in the queue.

private void purgeQueue(Queue queue) {
  queue.purge(); // remove all tasks from a queue
}

It can take a minute to remove all tasks from a queue, so you should wait a few seconds before adding new tasks to the queue.

Deleting a task queue

To delete a task queue, remove the entry from the project's queue.xml file and redeploy.

Deploying to App Engine

You can deploy your app to App Engine using Maven.

Go to the root directory of your project and type:

mvn package appengine:deploy -Dapp.deploy.projectId=PROJECT_ID

Replace PROJECT_ID with the ID of your Google Cloud project. If your pom.xml file already specifies your project ID, you don't need to include the -Dapp.deploy.projectId property in the command you run.

After Maven deploys your app, open a web browser tab automatically at your new app by typing:

gcloud app browse

What's next

This guide shows how to use a task queue to create an image thumbnail and store it in Cloud Storage. It can be used with other storage services such as Cloud Datastore or Cloud SQL.

Previous
Using Cloud SQL