Getting Started: Task Queues

Learn how to use task queues and the App Engine Image API to resize images.

Task queues execute code outside of direct user interaction, allowing tasks to happen in the background. This guide builds on the sample code used in the Using Cloud Storage guide, which stores files in Cloud Storage. The current guide extends that Cloud Storage sample by adding a Task Queue for performing tasks after an image is stored in Cloud Storage. The tasks to be performed in the Task Queue are:

1. Retrieve the image file that just got uploaded to Cloud Storage.
2. Resize it to a thumbnail image using the Image API.
3. Store the resulting thumbnail in Cloud Storage.

The App Engine Java 8 runtime also supports Java's native image manipulation classes such as AWT and Java2D.

Before you begin

1. Configure your development environment and create your App Engine project.

2. This guide uses the Apache Commons IOUtils library. To include the IOUtils library to your App Engine project:

   Maven

   Add to your pom.xml:

   <dependency>
     <groupId>commons-io</groupId>
     <artifactId>commons-io</artifactId>
     <version>2.5</version>
   </dependency>
   

   Gradle

   Add to your build.gradle:

   compile group: 'commons-io', name: 'commons-io', version: '2.5'
   

Importing libraries

The sample code provided with this guide uses the following imports:

import com.google.appengine.api.images.Image;
import com.google.appengine.api.images.ImagesService;
import com.google.appengine.api.images.ImagesServiceFactory;
import com.google.appengine.api.images.Transform;
import org.apache.commons.io.IOUtils;

Creating a task queue

App Engine has a default task queue, however you can create your own by adding an entry in the queue.xml.

To add queues, create the queue.xml file in your App Engine project's WEB-INF directory. A basic taskqueue can only consist of a name and an execution rate:

<?xml version="1.0" encoding="UTF-8"?>
  <queue-entries>
    <queue>
      <name>resize-image</name>
      <rate>60/h</rate>
    </queue>
  </queue-entries>

This example queue, named resize-image, defines a rate of execution of 60 times per hour, or once a minute. To see the full list queue options see the queue.xml reference.

A task queue has two components, the task requestor and a task hander. The requestor adds a task to the queue and sends it to the task handler.

Adding tasks to a queue

To add a task to a queue:

1. Create a task queue object using QueueFactory.getQueue(), making sure you specify the queue name defined in the queue.xml:

   Queue imageResizeQueue; // Taskqueue queue
   
   @Override
   public void init() throws ServletException {
   
     // Setup Cloud Storage service
     gcsService =
         GcsServiceFactory.createGcsService(
             new RetryParams.Builder()
                 .initialRetryDelayMillis(10)
                 .retryMaxAttempts(10)
                 .totalRetryPeriodMillis(15000)
                 .build());
   
     // Initialize the queue object with the specific queue
     imageResizeQueue = QueueFactory.getQueue([QUEUE-NAME]);
   
     // Cloud SQL connection setup
     try {
       final String url = System.getProperty("cloudsql"); // Cloud SQL server URI
   
       try {
         conn = DriverManager.getConnection(url); // Connect to the database
   
         Statement createTable; // SQL statement
   
         // Batch SQL table creation commands
         createTable.addBatch(createContentTableSql);
         createTable.addBatch(createUserTableSql);
         createTable.addBatch(createImageTableSql);
         createTable.addBatch(createBlogPostImageTableSql);
         conn.createTable.executeBatch(); // Execute batch
   
       } catch (SQLException e) {
         throw new ServletException("Unable to connect to Cloud SQL", e);
       }
   
     } finally {
       // Nothing really to do here.
     }
   
   }
   

2. Add tasks to Queue object. As shown in the code sample, imageResizeQueue.add() adds a task to the imageResizeQueue object:

   try {
     // Add a queued task to create a thumbnail of the uploaded image
     imageResizeQueue.add(
         TaskOptions.Builder.withUrl("/tasks/imageresize").param("filename", filename));
   }
   

   Specify the URI of the task handler using TaskOptions.Builder.withUrl(), along with and any parameters sent to the handler.

   In this example, the URI is /tasks/imageresize and the paramater is a variable called filename that contains the filename of the image to be processed.

Creating a task handler

Once you have added a task to the queue, the task handler mapped to the URI /tasks/imageresize will run. A task handler is a Java Servlet that attempts to perform the task until it is successful.

In this example, the task handler does three tasks:

• Retrieve the image specified by the caller from Cloud Storage.

• Transform the image using the App Engine Image API, in this sample, to a thumbnail image.

• Store the transformed image (thumbnail) in Cloud Storage.

To create the task handler:

1. Add an annotation that maps the handler to the URI /tasks/imageresize:

    @WebServlet(name = "imageResize", description = "Task queue handler", urlPatterns = "/tasks/imageresize")
    public class imageResize extends HttpServlet {
   
      // Task handler functionality
   
    }
   

2. Set up a connection to Cloud Storage as documented in the Using Cloud Storage guide and retrieve the image from Cloud Storage:

    public void init() throws ServletException {
   
     // initiate GcsService
     GcsService gcsService =
       GcsServiceFactory.createGcsService(
           new RetryParams.Builder()
               .initialRetryDelayMillis(10)
               .retryMaxAttempts(10)
               .totalRetryPeriodMillis(15000)
               .build());
   }
   

3. Handle the incoming Task Queue request, using the supplied file name to retrieve the image from Cloud Storage:

   protected void doPost(HttpServletRequest req, HttpServletResponse resp)
       throws ServletException, IOException {
   
     String filename = req.getParameter("filename"); // Get the filename passed from the task requestor
     GcsFilename gcsFile = new GcsFilename(bucket, filename); // Create a valid Cloud Storage filename
   
     GcsInputChannel readChannel = gcsService.openPrefetchingReadChannel(gcsFile, 0, BUFFER_SIZE); // Read the file from Cloud Storage
   

4. Use the ImagesService object to do the image resizing:

   // Get an instance of the ImagesService we can use to transform images.
   ImagesService imagesService = ImagesServiceFactory.getImagesService();
   
   // Make an image directly from a byte array, and transform it.
   Image image =
       ImagesServiceFactory.makeImage(IOUtils.toByteArray(Channels.newInputStream(readChannel)));
   Transform resize = ImagesServiceFactory.makeResize(100, 50); // resize image to 100x50
   Image resizedImage = imagesService.applyTransform(resize, image);
   
   // Write the transformed image back to a Cloud Storage object.
   gcsService.createOrReplace(
       new GcsFilename(bucket, "thumbnail_" + filename),
       new GcsFileOptions.Builder().acl("public-read").build(),
       ByteBuffer.wrap(resizedImage.getImageData()));
   

   The above snippet uses the Image API makeResize() method to resize the image to a thumbnail. To do this, it reads the image from Cloud Storage into an InputChannel and convert it to a ByteArray using IOUtils.toByteArray().

   After applying the transformation, the new image has the string thumbnail_ appended to its filename, permission set to be publicly readable and written to Cloud Storage.

Securing task handler URLs

You should secure tasks that perform sensitive operations such as modifying data so external users cannot call it directly. You can do this by restricting task access to App Engine administrators, which prevents users from accessing task URLs . Note that this restriction does not apply to task requests coming from your App Engine app.

In the current example, the task handlers have URLs in the /tasks/ folder. To restrict access to the /tasks/ folder to App Engine administrators, add the following to the project's web.xml.

<security-constraint>
    <web-resource-collection>
        <web-resource-name>tasks</web-resource-name>
        <url-pattern>/tasks/*</url-pattern>
    </web-resource-collection>
    <auth-constraint>
        <role-name>admin</role-name>
    </auth-constraint>
</security-constraint>

Removing a single task from a queue

To remove a single task from a queue, use deleteTask():

private void removeTask(Queue queue, String taskName) {
  queue.deleteTask(taskName); // remove a task from a queue
}

Removing all tasks from a queue

To remove all tasks from a queue, use purge(). The purge can take up to a minute to remove all tasks in the queue.

private void purgeQueue(Queue queue) {
  queue.purge(); // remove all tasks from a queue
}

It can take a minute to remove all tasks from a queue, so you should wait a few seconds before adding new tasks to the queue.

Deleting a task queue

To delete a task queue, remove the entry from the project's queue.xml file and redeploy.

Deploying to App Engine

You can deploy your app to App Engine using Maven.

Go to the root directory of your project and type:

mvn appengine:deploy

After Maven deploys your app, open a web browser tab automatically at your new app by typing:

gcloud app browse

What's next

This guide shows how to use a task queue to create an image thumbnail and store it in Cloud Storage. It can be used with other storage services such as Cloud Datastore or Cloud SQL.