App Engine and Google Cloud Storage Sample

This tutorial shows everything you need to do to enable your App Engine Go app to access Cloud Storage. This non-interactive sample shows you how to create, write, read, and list files in a Cloud Storage bucket.

When this sample runs, it executes a script that performs several actions against your Cloud Storage bucket and then outputs details to your browser.

Objectives

  • Deploy and run the sample app in Google App Engine.
  • Learn how to create a file and then write that file to a Cloud Storage bucket.
  • Learn how to read files from a Cloud Storage bucket.
  • Learn how to copy files in a Cloud Storage bucket.
  • Learn how to list the files that exist in a Cloud Storage.

Costs

App Engine has free a level of usage. If your total usage of App Engine is less than the limits specified in the App Engine free quota, there is no charge for doing this tutorial.

Before you begin

  1. Create or select a Cloud Platform project in the Cloud Platform Console and then ensure that project includes an App Engine application:

    Go to App Engine

    The Dashboard opens if an App Engine application already exists in your project. Otherwise, you are prompted to choose the region where you want your App Engine application located.

  2. Downloaded and install the SDK for App Engine.

  3. Set up Cloud Storage:

    1. Activate the default Cloud Storage bucket.

    2. Download and install the Cloud Storage Client Library.

Downloading the sample project

To download the project, choose either of the following methods:

  • Clone the project that contains the sample code:

    git clone https://github.com/GoogleCloudPlatform/google-cloud-go.git
    
  • Download the sample as a .zip file and then extract it to your working directory.

Deploying and running the sample

To deploy the sample app to App Engine and then view the sample in your browser:

  1. Run the following command from within the root directory of your application where the app.yaml file is located, for example google-cloud-go/examples/storage/appengine:

    gcloud app deploy
    

    To learn more about deploying your app from the command line, see Deploying a Go App.

  2. Launch your browser to view the sample at http://[YOUR_PROJECT_ID].appspot.com, by running the following command:

    gcloud app browse
    
  3. Review the app results. Your browser should display details for each of the actions that the script performed on your Cloud Storage bucket.

    In addition to writing, reading, coping, and listing files, the sample app also displays object permissions, manipulates access control lists (ACLs), and then cleans up by deleted all the demo files from your Cloud Storage bucket.

    Example results:

    demo sample

Code walkthrough

This walkthrough only covers how to read and write to a Cloud Storage bucket, and list the bucket contents, even though the sample code contains additional Cloud Storage functionality.

app.yaml walkthrough

The configuration in the app.yaml file is typical for nearly any app. Like any other App Engine app, it allows you to specify application configuration details:

application: your-app-id
version: v1
runtime: go
api_version: go1

handlers:
- url: /.*
  script: _go_app

For more information on configuration options available in this file, see the app.yaml reference.

app.go imports walkthrough

The app.go file contains these imports:

import (
	"bytes"
	"fmt"
	"io"
	"io/ioutil"
	"net/http"
	"strings"

	"cloud.google.com/go/storage"
	"golang.org/x/net/context"
	"google.golang.org/api/iterator"
	"google.golang.org/appengine"
	"google.golang.org/appengine/file"
	"google.golang.org/appengine/log"
)

The imports in the file required for App Engine and for Cloud Storage are:

  • google.golang.org/appengine,
  • google.golang.org/appengine/file
  • cloud.google.com/go/storage

Specifying the Cloud Storage bucket

Before you can execute any Cloud Storage operation, you must supply the bucket name. The easiest way to do this is to use the default bucket for your project, which can be obtained from the App Engine context:

// Use `dev_appserver.py --default_gcs_bucket_name GCS_BUCKET_NAME`
// when running locally.
bucket, err := file.DefaultBucketName(ctx)
if err != nil {
	log.Errorf(ctx, "failed to get default GCS bucket name: %v", err)
}

Writing a file to Cloud Storage

To write a file to Cloud Storage:

// createFile creates a file in Google Cloud Storage.
func (d *demo) createFile(fileName string) {
	fmt.Fprintf(d.w, "Creating file /%v/%v\n", d.bucketName, fileName)

	wc := d.bucket.Object(fileName).NewWriter(d.ctx)
	wc.ContentType = "text/plain"
	wc.Metadata = map[string]string{
		"x-goog-meta-foo": "foo",
		"x-goog-meta-bar": "bar",
	}
	d.cleanUp = append(d.cleanUp, fileName)

	if _, err := wc.Write([]byte("abcde\n")); err != nil {
		d.errorf("createFile: unable to write data to bucket %q, file %q: %v", d.bucketName, fileName, err)
		return
	}
	if _, err := wc.Write([]byte(strings.Repeat("f", 1024*4) + "\n")); err != nil {
		d.errorf("createFile: unable to write data to bucket %q, file %q: %v", d.bucketName, fileName, err)
		return
	}
	if err := wc.Close(); err != nil {
		d.errorf("createFile: unable to close bucket %q, file %q: %v", d.bucketName, fileName, err)
		return
	}
}

When the file is created, the sample specifies Cloud Storage headers (x-goog-meta-foo and x-goog-meta-bar). This optional code introduces the notion of using Cloud Storage headers, which you can use to:

  • Affect request behavior.
  • Specify access to a file in your Cloud Storage bucket that is different from the defaults. See x-goog-acl for more information.
  • Write file metadata.

The x-goog-meta- headers shown above are custom file metadata that you can set; these headers are always returned with the file. Note that the space available for custom headers and their data is limited to a few kilobytes, so use these carefully.

Because the code sample doesn't set x-goog-acl, the default Cloud Storage ACL of public read is applied to the object when it is written to the bucket.

Finally, notice the call to Close the file after you finish the write. If you don't do this, the file is not written to Cloud Storage. Be aware that after you call Close, you cannot append to the file.

Reading a file from Cloud Storage

To read a file from Cloud Storage:

// readFile reads the named file in Google Cloud Storage.
func (d *demo) readFile(fileName string) {
	io.WriteString(d.w, "\nAbbreviated file content (first line and last 1K):\n")

	rc, err := d.bucket.Object(fileName).NewReader(d.ctx)
	if err != nil {
		d.errorf("readFile: unable to open file from bucket %q, file %q: %v", d.bucketName, fileName, err)
		return
	}
	defer rc.Close()
	slurp, err := ioutil.ReadAll(rc)
	if err != nil {
		d.errorf("readFile: unable to read data from bucket %q, file %q: %v", d.bucketName, fileName, err)
		return
	}

	fmt.Fprintf(d.w, "%s\n", bytes.SplitN(slurp, []byte("\n"), 2)[0])
	if len(slurp) > 1024 {
		fmt.Fprintf(d.w, "...%s\n", slurp[len(slurp)-1024:])
	} else {
		fmt.Fprintf(d.w, "%s\n", slurp)
	}
}

Listing bucket contents

This sample code shows how to list the contents of the bucket:

// listBucket lists the contents of a bucket in Google Cloud Storage.
func (d *demo) listBucket() {
	io.WriteString(d.w, "\nListbucket result:\n")

	query := &storage.Query{Prefix: "foo"}
	it := d.bucket.Objects(d.ctx, query)
	for {
		obj, err := it.Next()
		if err == iterator.Done {
			break
		}
		if err != nil {
			d.errorf("listBucket: unable to list bucket %q: %v", d.bucketName, err)
			return
		}
		d.dumpStats(obj)
	}
}

Copying Cloud Storage files

This sample code shows how to list the contents of the bucket:

// copyFile copies a file in Google Cloud Storage.
func (d *demo) copyFile(fileName string) {
	copyName := fileName + "-copy"
	fmt.Fprintf(d.w, "Copying file /%v/%v to /%v/%v:\n", d.bucketName, fileName, d.bucketName, copyName)

	obj, err := d.bucket.Object(copyName).CopierFrom(d.bucket.Object(fileName)).Run(d.ctx)
	if err != nil {
		d.errorf("copyFile: unable to copy /%v/%v to bucket %q, file %q: %v", d.bucketName, fileName, d.bucketName, copyName, err)
		return
	}
	d.cleanUp = append(d.cleanUp, copyName)

	d.dumpStats(obj)
}

What's next

Learn more about the following topics:

Send feedback about...

App Engine standard environment for Go