Using App Engine APIs for Cloud Storage

This document describes how to store and retrieve data using the App Engine client library for Cloud Storage. It assumes that you know how to build an App Engine application as described in the Quickstart for Java 8 App Engine standard environment. It also assumes that you understand the basic concepts of using Cloud Storage in App Engine.

Downloading the App Engine client library for Cloud Storage

You can download the library using popular tools like Apache Maven, Apache Ivy, or Git, or you can download the library manually from the Maven repository. Choose your preferred method:

Git

If you have Git installed, you can clone the Google Cloud Storage client library's GitHub repository as follows:

git clone https://github.com/GoogleCloudPlatform/appengine-gcs-client.git

Maven

Maven users should include the following in their application's pom.xml file:

<dependency>
    <groupId>com.google.appengine.tools</groupId>
    <artifactId>appengine-gcs-client</artifactId>
    <version>0.8</version>
</dependency>

Ivy

Ivy users should include the following in their application's ivy.xml file:

<dependency org="com.google.appengine.tools"
            name="appengine-gcs-client"
            rev="latest.integration" />

Manual download

Visit the library's Maven repository and download the latest class, source, and JavaDoc JAR files:

In addition, you will need to download the following dependencies and include them in your application:

ACLs and the App Engine client library

An app using the client library cannot change the bucket ACL, but it can specify an ACL that controls access to the objects it creates. The available ACL settings are described under documentation for the FcsFileOptions class.

Cloud Storage and subdirectories

The App Engine client library for Cloud Storage lets you supply subdirectory delimiters when you create an object, but there are no true subdirectories in Cloud Storage. Instead, a subdirectory in Cloud Storage is a part of the object filename.

For example, you might assume that creating an object somewhere/over/the/rainbow.mp3 would store the file rainbow.mp3 in the subdirectory somewhere/over/the/. Instead, the object name is set to somewhere/over/the/rainbow.mp3.

Using the App Engine client library with the development app server

You can use the client library with the development server. However because there is no local emulation of Cloud Storage, all requests to read and write files must be sent over the Internet to an actual Cloud Storage bucket.

To use the client library with the development app server, run dev_appserver.py with the flag --default_gcs_bucket_name [BUCKET_NAME], replacing [BUCKET_NAME] with the name of the Cloud Storage bucket you are using.

This flag controls the bucket that will be returned when your application calls file.DefaultBucketName(ctx).

Reading and Writing to Google Cloud Storage

Required imports

The following snippet shows the imports you need for Cloud Storage access via the client library:

java/example/src/main/java/com/google/appengine/demos/GcsExampleServlet.java 
import com.google.appengine.tools.cloudstorage.GcsFileOptions;
import com.google.appengine.tools.cloudstorage.GcsFilename;
import com.google.appengine.tools.cloudstorage.GcsInputChannel;
import com.google.appengine.tools.cloudstorage.GcsOutputChannel;
import com.google.appengine.tools.cloudstorage.GcsService;
import com.google.appengine.tools.cloudstorage.GcsServiceFactory;
import com.google.appengine.tools.cloudstorage.RetryParams;

Specifying the Cloud Storage bucket

The following snippet shows one way to allow the user to specify a bucket name when writing a file to a bucket:

function uploadFile() {
  var bucket = document.forms["putFile"]["bucket"].value;
  var filename = document.forms["putFile"]["fileName"].value;
  if (bucket == null || bucket == "" || filename == null || filename == "") {
    alert("Both Bucket and FileName are required");
    return false;
  } else {
    var postData = document.forms["putFile"]["content"].value;
    document.getElementById("content").value = null;

    var request = new XMLHttpRequest();
    request.open("POST", "/gcs/" + bucket + "/" + filename, false);
    request.setRequestHeader("Content-Type", "text/plain;charset=UTF-8");
    request.send(postData);
  }
}

The snippet prepends the required /gcs/ to the user supplied bucket and file name.

Writing to Cloud Storage

To write a file to Cloud Storage:

java/example/src/main/java/com/google/appengine/demos/GcsExampleServlet.java 
@Override
public void doPost(HttpServletRequest req, HttpServletResponse resp) throws IOException {
  GcsFileOptions instance = GcsFileOptions.getDefaultInstance();
  GcsFilename fileName = getFileName(req);
  GcsOutputChannel outputChannel;
  outputChannel = gcsService.createOrReplace(fileName, instance);
  copy(req.getInputStream(), Channels.newOutputStream(outputChannel));
}

This sample writes a new file to Cloud Storage, or, if a file with the same name already exists, it overwrites it. This is useful because once you write a file to Cloud Storage, you cannot modify it. To change the file, you must make your modifications to a copy of the file, then overwrite the old file.

Reading from Cloud Storage

To read a file from Cloud Storage:

java/example/src/main/java/com/google/appengine/demos/GcsExampleServlet.java 
@Override
public void doGet(HttpServletRequest req, HttpServletResponse resp) throws IOException {
  GcsFilename fileName = getFileName(req);
  if (SERVE_USING_BLOBSTORE_API) {
    BlobstoreService blobstoreService = BlobstoreServiceFactory.getBlobstoreService();
    BlobKey blobKey = blobstoreService.createGsBlobKey(
        "/gs/" + fileName.getBucketName() + "/" + fileName.getObjectName());
    blobstoreService.serve(blobKey, resp);
  } else {
    GcsInputChannel readChannel = gcsService.openPrefetchingReadChannel(fileName, 0, BUFFER_SIZE);
    copy(Channels.newInputStream(readChannel), resp.getOutputStream());
  }
}

In the line containing gcsService.openPrefetchingReadChannel, notice the use of prefetching. This buffers data in memory and prefetches it before it is required to avoid blocking on the read call.

Modifying objects in Cloud Storage

To modify an existing object in Cloud Storage, write the modified object with the same filename as the original.

Deleting objects from Cloud Storage

The following procedure shows how to delete objects in an app that stores the full Cloud Storage object name (filename) in a Cloud SQL database:

final String bucket = "CLOUD-STORAGE-BUCKET"; // Cloud Storage bucket name
Map<String, String[]> userData = req.getParameterMap();

String[] image = userData.get("id"); // Grab the encoded image ID
String decodedId = new String(Base64.getUrlDecoder().decode(image[0])); // decode the image ID
int imageId = Integer.parseInt(decodedId);

// Grab the filename and build out a Cloud Storage filepath in preparation for deletion
try (PreparedStatement statementDeletePost = conn.prepareStatement(imageFilenameSql)) {
  statementDeletePost.setInt(1, imageId); // cast String to Int
  ResultSet rs = statementDeletePost.executeQuery(); // remove image record
  rs.next(); // move the cursor

  GcsFilename filename = new GcsFilename(bucket, rs.getString("filename"));
  if (gcsService.delete(filename)) {

    // Remove all records of image use in the blog
    // Use of foreign keys with cascading deletes will cause removal from blogpostImages table
    PreparedStatement statementDeleteImageRecord = conn.prepareStatement(deleteSql);
    statementDeleteImageRecord.setInt(1, imageId);
    statementDeleteImageRecord.executeUpdate();

    final String confirmation =
        "Image ID "
            + imageId
            + " has been deleted and record of its use in blog posts have been removed.";

    req.setAttribute("confirmation", confirmation);
    req.getRequestDispatcher("/confirm.jsp").forward(req, resp);
  } else {
    final String confirmation = "File marked for deletion does not exist.";

    req.setAttribute("confirmation", confirmation);
    req.getRequestDispatcher("/confirm.jsp").forward(req, resp);
  }
} catch (SQLException e) {
  throw new ServletException("SQL error", e);
}

The code above decodes the Base64 encoded image ID and retrieves the filename of an image identified by its image_id from the images table. The filename is converted to a valid Cloud Storage filename using GcsFilename.

The file is deleted from the bucket using gcsService.delete. Finally, it removes the record of its usage in the blogpostImage table.

What's next