Reading and Writing to Google Cloud Storage

This document describes how to store and retrieve data using Cloud Storage in an App Engine app using the App Engine client library for Cloud Storage. It assumes that you completed the tasks described in Setting Up for Google Cloud Storage to activate a Cloud Storage bucket and download the client libraries. It also assumes that you know how to build an App Engine application, as described in the Quickstart for Go 1.11 App Engine standard environment.

Required imports

The imports in the file required for App Engine and for Cloud Storage are:

  • google.golang.org/appengine,
  • google.golang.org/appengine/file
  • cloud.google.com/go/storage

as shown in the following snippet:

docs/appengine/storage/app.go 
import (
	"bytes"
	"fmt"
	"io"
	"io/ioutil"
	"net/http"
	"strings"

	"cloud.google.com/go/storage"
	"golang.org/x/net/context"
	"google.golang.org/api/iterator"
	"google.golang.org/appengine"
	"google.golang.org/appengine/file"
	"google.golang.org/appengine/log"
)

Specifying the Cloud Storage bucket

Before you can execute any Cloud Storage operation, you must supply the bucket name. The easiest way to do this is to use the default bucket for your project, which can be obtained from the App Engine context, as shown in this snippet:

docs/appengine/storage/app.go 
// Use `dev_appserver.py --default_gcs_bucket_name GCS_BUCKET_NAME`
// when running locally.
bucket, err := file.DefaultBucketName(ctx)
if err != nil {
	log.Errorf(ctx, "failed to get default GCS bucket name: %v", err)
}

Writing to Cloud Storage

To write a file to Cloud Storage:

docs/appengine/storage/app.go 
// createFile creates a file in Google Cloud Storage.
func (d *demo) createFile(fileName string) {
	fmt.Fprintf(d.w, "Creating file /%v/%v\n", d.bucketName, fileName)

	wc := d.bucket.Object(fileName).NewWriter(d.ctx)
	wc.ContentType = "text/plain"
	wc.Metadata = map[string]string{
		"x-goog-meta-foo": "foo",
		"x-goog-meta-bar": "bar",
	}
	d.cleanUp = append(d.cleanUp, fileName)

	if _, err := wc.Write([]byte("abcde\n")); err != nil {
		d.errorf("createFile: unable to write data to bucket %q, file %q: %v", d.bucketName, fileName, err)
		return
	}
	if _, err := wc.Write([]byte(strings.Repeat("f", 1024*4) + "\n")); err != nil {
		d.errorf("createFile: unable to write data to bucket %q, file %q: %v", d.bucketName, fileName, err)
		return
	}
	if err := wc.Close(); err != nil {
		d.errorf("createFile: unable to close bucket %q, file %q: %v", d.bucketName, fileName, err)
		return
	}
}

When the file is created, the sample specifies Cloud Storage headers (x-goog-meta-foo and x-goog-meta-bar). This optional code introduces the notion of using Cloud Storage headers, which you can apply to

  • Affect request behavior
  • Specify access to the file in the bucket different from the defaults (see x-goog-acl)
  • Write file metadata.

The x-goog-meta- headers shown above are custom file metadata that you can set; these headers are always returned with the file. Note that the space available for custom headers and their data is limited to a few kilobytes, so use these carefully.

Because the code sample doesn't set x-goog-acl, the default Cloud Storage ACL of public read is applied to the object when it is written to the bucket.

Finally, notice the call to close the file after you finish the write. If you don't do this, the file is not written to Cloud Storage. Be aware that after you call Close, you cannot append to the file.

Reading from Cloud Storage

To read a file from Cloud Storage:

docs/appengine/storage/app.go 
// readFile reads the named file in Google Cloud Storage.
func (d *demo) readFile(fileName string) {
	io.WriteString(d.w, "\nAbbreviated file content (first line and last 1K):\n")

	rc, err := d.bucket.Object(fileName).NewReader(d.ctx)
	if err != nil {
		d.errorf("readFile: unable to open file from bucket %q, file %q: %v", d.bucketName, fileName, err)
		return
	}
	defer rc.Close()
	slurp, err := ioutil.ReadAll(rc)
	if err != nil {
		d.errorf("readFile: unable to read data from bucket %q, file %q: %v", d.bucketName, fileName, err)
		return
	}

	fmt.Fprintf(d.w, "%s\n", bytes.SplitN(slurp, []byte("\n"), 2)[0])
	if len(slurp) > 1024 {
		fmt.Fprintf(d.w, "...%s\n", slurp[len(slurp)-1024:])
	} else {
		fmt.Fprintf(d.w, "%s\n", slurp)
	}
}

Listing bucket contents

This sample code shows how to list the contents of the bucket:

docs/appengine/storage/app.go 
// listBucket lists the contents of a bucket in Google Cloud Storage.
func (d *demo) listBucket() {
	io.WriteString(d.w, "\nListbucket result:\n")

	query := &storage.Query{Prefix: "foo"}
	it := d.bucket.Objects(d.ctx, query)
	for {
		obj, err := it.Next()
		if err == iterator.Done {
			break
		}
		if err != nil {
			d.errorf("listBucket: unable to list bucket %q: %v", d.bucketName, err)
			return
		}
		d.dumpStats(obj)
	}
}

Deleting files in Cloud Storage

The code below demonstrates how to delete a file from Cloud Storage using the ObjectHandle.delete() method.

docs/appengine/storage/app.go 

// deleteFiles deletes all the temporary files from a bucket created by this demo.
func (d *demo) deleteFiles() {
	io.WriteString(d.w, "\nDeleting files...\n")
	for _, v := range d.cleanUp {
		fmt.Fprintf(d.w, "Deleting file %v\n", v)
		if err := d.bucket.Object(v).Delete(d.ctx); err != nil {
			d.errorf("deleteFiles: unable to delete bucket %q, file %q: %v", d.bucketName, v, err)
			return
		}
	}
}

This example cleans up the files that were written to the bucket in the Writing to Cloud Storage section.

What's next