Getting Started: Cloud Storage

Learn how to store data in Google Cloud Storage from a Java app on App Engine.

Cloud Storage is used to store binary large objects (BLOB) such as images, PDFs, or video files and is one storage option available with App Engine. Compare Cloud Storage, Cloud Datastore, and Cloud SQL and choose the one that meets your app's requirements.

This guide extends the App Engine blog app to store text data in Cloud SQL and images in Cloud Storage.

Before you begin

Configure your development environment and create your App Engine project.

Creating a Cloud Storage Bucket

Your app will store data in a Cloud Storage bucket, to create the bucket:

  1. In the GCP Console, go to the Cloud Storage browser.

    Go to the Cloud Storage browser

  2. Click Create bucket.
  3. In the Create bucket dialog, specify the following attributes:
  4. Click Create.

Downloading the App Engine Cloud Storage library

Your application must use the App Engine Tools library to use Cloud Storage.

Maven

Include the following in your application's pom.xml file:

<project>

 [...]

 <dependency>
   <groupId>com.google.appengine.tools</groupId>
   <artifactId>appengine-gcs-client</artifactId>
   <version>0.7</version>
 </dependency>

 [...]

</project>

Gradle

Include the following in your application's build.gradle file:

compile 'com.google.appengine.tools:appengine-gcs-client:0.7'

Importing libraries

The sample code provided with this guide uses the following imports from the App Engine Tools library:

import com.google.appengine.tools.cloudstorage.GcsFileOptions;
import com.google.appengine.tools.cloudstorage.GcsFilename;
import com.google.appengine.tools.cloudstorage.GcsOutputChannel;
import com.google.appengine.tools.cloudstorage.GcsService;
import com.google.appengine.tools.cloudstorage.GcsServiceFactory;
import com.google.appengine.tools.cloudstorage.RetryParams;

Uploading a file to Cloud Storage

The following snippet extends the code shown for Handling Form Data to include a file selector so the user can choose images to upload.

<form method="POST" action="/create" enctype="multipart/form-data">

  <div>
    <label for="title">Title</label>
    <input type="text" name="title" id="title" size="40" value="" />
  </div>

  <div>
    <label for="author">Author</label>
    <input type="text" name="author" id="author" size="40" value="" />
  </div>

  <div class="form-group">
    <label for="description">Post content</label>
    <textarea name="description" id="description" rows="10" cols="50"></textarea>
  </div>

  <div class="form-group">
    <label for="filename">Upload image</label>
    <!-- Allow only image file types to be selected -->
    <input type="file" name="image" accept="image/*">
  </div>

  <button type="submit">Save</button>

</form>

The POST method sends the data to /create, where a servlet stores the text input and the filename to a Cloud SQL table and the image to Cloud Storage, as shown later in this guide.

Storing files in Cloud Storage

To store files in Cloud Storage:

  1. Use the @MultipartConfig annotation, which is required for handling multi-part HTML form data such as files:

    @MultipartConfig(
      maxFileSize = 10 * 1024 * 1024, // max size for uploaded files
      maxRequestSize = 20 * 1024 * 1024, // max size for multipart/form-data
      fileSizeThreshold = 5 * 1024 * 1024 // start writing to Cloud Storage after 5MB
    )
    
  2. Create a GcsService object using GcsServiceFactory.

    private final GcsService gcsService =
        GcsServiceFactory.createGcsService(
            new RetryParams.Builder()
                .initialRetryDelayMillis(10)
                .retryMaxAttempts(10)
                .totalRetryPeriodMillis(15000)
                .build());
    

    When creating a GcsService object, you must specify the backoff parameters. In this example, the service will do 10 retries within 15 seconds.

  3. Create variables for the buffer size and Cloud Storage bucket for gcsService object:

    private static final int BUFFER_SIZE = 2 * 1024 * 1024;
    private final String bucket = "[CLOUD-STORAGE-BUCKET-NAME]";
    

    Replace CLOUD-STORAGE-BUCKET-NAME with the name of your Cloud Storage bucket.

  4. Extract the filename from the file to be uploaded, create a unique Cloud Storage filename from it, and store the file in Cloud Storage:

    private String storeImage(Part image) throws IOException {
    
      String filename = uploadedFilename(image); // Extract filename
      GcsFileOptions.Builder builder = new GcsFileOptions.Builder();
    
      builder.acl("public-read"); // Set the file to be publicly viewable
      GcsFileOptions instance = GcsFileOptions.getDefaultInstance();
      GcsOutputChannel outputChannel;
      GcsFilename gcsFile = new GcsFilename(bucket, filename);
      outputChannel = gcsService.createOrReplace(gcsFile, instance);
      copy(filePart.getInputStream(), Channels.newOutputStream(outputChannel));
    
      return filename; // Return the filename without GCS/bucket appendage
    }
    

    The storeImage() method sets the file permissions to public-read to make it publicly visible. The image is written to Cloud Storage using the copy() function from the App Engine Tools library.

    private String uploadedFilename(final Part part) {
    
      final String partHeader = part.getHeader("content-disposition");
    
      for (String content : part.getHeader("content-disposition").split(";")) {
        if (content.trim().startsWith("filename")) {
          // Append a date and time to the filename
          DateTimeFormatter dtf = DateTimeFormat.forPattern("-YYYY-MM-dd-HHmmssSSS");
          DateTime dt = DateTime.now(DateTimeZone.UTC);
          String dtString = dt.toString(dtf);
          final String fileName =
              dtString + content.substring(content.indexOf('=') + 1).trim().replace("\"", "");
    
          return fileName;
        }
      }
      return null;
    }
    

    In the snippet above, uploadedFilename() extracts the filename from the HTTP headers and appends a timestamp to create a unique filename.

    The storeImage() method writes to Cloud Storage using copy:

    private void copy(InputStream input, OutputStream output) throws IOException {
    
      try {
        byte[] buffer = new byte[BUFFER_SIZE];
        int bytesRead = input.read(buffer);
        while (bytesRead != -1) {
          output.write(buffer, 0, bytesRead);
          bytesRead = input.read(buffer);
        }
      } finally {
        input.close();
        output.close();
      }
    
    }
    

    The copy() method takes an InputStream object, which is the image, and an OutputStream object which in this example is createOrReplace(), a method that writes the file to Cloud Storage.

  5. Save the Cloud Storage filename in a database for later lookups. The following code uses Cloud SQL to a create a table to store image filenames.

    final String createImageTableSql =
        "CREATE TABLE IF NOT EXISTS images ( image_id INT NOT NULL "
            + "AUTO_INCREMENT, filename VARCHAR(256) NOT NULL, "
            + "PRIMARY KEY (image_id) )";
    
    conn.createStatement().executeUpdate(createImageTableSql);
    
  6. Associate each image with the blog post that uses the image. The sample uses the blogpostImages table for this:

    final String createBlogPostImageTableSql =
          "CREATE TABLE IF NOT EXISTS blogpostImages ( post_id INT NOT NULL, "
              + "image_id INT NOT NULL, "
              + "INDEX postImages (post_id, image_id), "
              + "FOREIGN KEY (post_id) REFERENCES posts(post_id) ON DELETE CASCADE, "
              + "FOREIGN KEY (image_id) REFERENCES images(image_id) ON DELETE CASCADE )";
    
    conn.createStatement().executeUpdate(createBlogPostImageTableSql);
    

    Notice the table has two foreign keys, post_id and image_id that are set to be deleted if either the parent blog post or image has been deleted.

    if (imageFile.getSize() != 0) { // check if an image has been uploaded
    
      // Grab the last autogenerated post ID to associate with image
      ResultSet lastId = conn.prepareStatement(getLastIdSql).executeQuery();
      lastId.next(); // move the cursor
      int postId = lastId.getInt(1); // store the post's ID to associate with image
    
      // Store the image file
      try {
        String filename = storeImage(imageFile); // Store the image and get the filename
        // Store a record of the filename in the database
        try (PreparedStatement statementCreateImage = conn.prepareStatement(createImageSql)) {
          statementCreateImage.setString(1, filename);
          statementCreateImage.executeUpdate();
    
          lastId = conn.prepareStatement(getLastIdSql).executeQuery();
          lastId.next(); // Move the cursor
          int imageId = lastId.getInt(1); // Store the post's ID for insertion later
    
          // Associate image with blog post
          PreparedStatement statementBlogImage = conn.prepareStatement(imageBlogPostSql);
          statementBlogImage.setInt(1, postId);
          statementBlogImage.setInt(2, imageId);
          statementBlogImage.executeUpdate();
    
        } catch (SQLException e) {
          throw new ServletException("SQL error when storing image details", e);
        }
      } catch (IOException e) {
        throw new IOException("Cloud Storage error when storing file", e);
      }
    }
    

Retrieving objects from Cloud Storage

The previous sections in this guide show how to store publicly viewable images in Cloud Storage. Cloud Storage generates links that are usable in HTML <img> tags. These image links have the URI https://storage.googleapis.com/[BUCKET-NAME]/[OBJECT-NAME].

To retrieve images associated with a particular blog post, you need to retrieve the related URIs from a database, as shown in the following SQL query for Cloud SQL:

final String selectSql =
      "SELECT images.filename "
          + "FROM images, blogpostImages "
          + "WHERE (images.image_id = blogpostImages.image_id) AND (blogpostImages.post_id = ?)";

The selectSql query returns the filename for an image associated with a particular blog post ID. Appending the URI to the filename will create a valid path to the publicly viewable image that can be used in the src attribute in an HTML <img> tag.

Modifying objects in Cloud Storage

To modify an existing object in Cloud Storage, write the modified object with the same filename as the original.

Deleting objects from Cloud Storage

The following procedure shows how to do deletions in an app that stores the full Cloud Storage object name (filename) in a Cloud SQL database:

final String bucket = "CLOUD-STORAGE-BUCKET"; // Cloud Storage bucket name
Map<String, String[]> userData = req.getParameterMap();

String[] image = userData.get("id"); // Grab the encoded image ID
String decodedId = new String(Base64.getUrlDecoder().decode(image[0])); // decode the image ID
int imageId = Integer.parseInt(decodedId);

// Grab the filename and build out a Cloud Storage filepath in preperation for deletion
try (PreparedStatement statementDeletePost = conn.prepareStatement(imageFilenameSql)) {
  statementDeletePost.setInt(1, imageId); // cast String to Int
  ResultSet rs = statementDeletePost.executeQuery(); // remove image record
  rs.next(); // move the cursor

  GcsFilename filename = new GcsFilename(bucket, rs.getString("filename"));
  if (gcsService.delete(filename)) {

    // Remove all records of image use in the blog
    // Use of foreign keys with cascading deletes will cause removal from blogpostImages table
    PreparedStatement statementDeleteImageRecord = conn.prepareStatement(deleteSql);
    statementDeleteImageRecord.setInt(1, imageId);
    statementDeleteImageRecord.executeUpdate();

    final String confirmation =
        "Image ID "
            + imageId
            + " has been deleted and record of its use in blog posts have been removed.";

    req.setAttribute("confirmation", confirmation);
    req.getRequestDispatcher("/confirm.jsp").forward(req, resp);
  } else {
    final String confirmation = "File marked for deletion does not exist.";

    req.setAttribute("confirmation", confirmation);
    req.getRequestDispatcher("/confirm.jsp").forward(req, resp);
  }
} catch (SQLException e) {
  throw new ServletException("SQL error", e);
}

The code above decodes the Base64 encoded image ID and retrieves the filename of an image identified by its image_id from the images table. The filename is converted to a valid Cloud Storage filename using GcsFilename.

The file is deleted from the bucket using gcsService.delete. Finally, it removes the record of its usage in the blogpostImage table.

Deploying to App Engine

You can deploy your app to App Engine using Maven.

Go to the root directory of your project and type:

mvn appengine:deploy

After Maven deploys your app, open a web browser tab automatically at your new app by typing:

gcloud app browse

What's next

Storing files in Cloud Storage in App Engine allows you to work with large files and user assets that your application might require. In some situations, you might need to perform tasks that take longer than the default time out or that you want to run asynchronously.

Next, learn about using task queues to perform asynchronous tasks by following an example of using the Images API to resize the images uploaded in this guide.

Send feedback about...

App Engine standard environment for Java