Cloud Storage Tutorial

This simple tutorial demonstrates writing, deploying, and triggering a Background Cloud Function with a Cloud Storage trigger.

Objectives

• Write and deploy a Background Cloud Function.
• Trigger the function by uploading a file to Cloud Storage.

Costs

This tutorial uses billable components of Cloud Platform, including:

• Google Cloud Functions
• Google Cloud Storage

Use the Pricing Calculator to generate a cost estimate based on your projected usage.

New Cloud Platform users might be eligible for a free trial.

Before you begin

1. Sign in to your Google Account.

   If you don't already have one, sign up for a new account.

2. In the Google Cloud Console, on the project selector page, select or create a Google Cloud project.

   Note: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project.

   Go to the project selector page

3. Make sure that billing is enabled for your Cloud project. Learn how to confirm that billing is enabled for your project.

4. Enable the Cloud Functions, Cloud Build, and Cloud Storage APIs.

   Enable the APIs

5. Install and initialize the Cloud SDK.

If you already have the Cloud SDK installed, update it by running the following command:

gcloud components update

6. Prepare your development environment:

   Node.js

   Go to the Node.js setup guide

   Python

   Go to the Python setup guide

   Go

   Go to the Go setup guide

   Java

   Go to the Java setup guide

   C#

   Go to the .NET setup guide

   Ruby

   Go to the Ruby setup guide

Preparing the application

1. Create a Cloud Storage bucket to upload a test file, where YOUR_TRIGGER_BUCKET_NAME is a globally unique bucket name:

   gsutil mb gs://YOUR_TRIGGER_BUCKET_NAME
   

2. Clone the sample app repository to your local machine:

   Node.js

   git clone https://github.com/GoogleCloudPlatform/nodejs-docs-samples.git

   Alternatively, you can download the sample as a zip file and extract it.

   Python

   git clone https://github.com/GoogleCloudPlatform/python-docs-samples.git

   Alternatively, you can download the sample as a zip file and extract it.

   Go

   git clone https://github.com/GoogleCloudPlatform/golang-samples.git

   Alternatively, you can download the sample as a zip file and extract it.

   Java

   git clone https://github.com/GoogleCloudPlatform/java-docs-samples.git

   Alternatively, you can download the sample as a zip file and extract it.

   C#

   git clone https://github.com/GoogleCloudPlatform/dotnet-docs-samples.git

   Alternatively, you can download the sample as a zip file and extract it.

   Ruby

   git clone https://github.com/GoogleCloudPlatform/ruby-docs-samples.git

   Alternatively, you can download the sample as a zip file and extract it.

3. Change to the directory that contains the Cloud Functions sample code:

   Node.js

   cd nodejs-docs-samples/functions/helloworld/

   Python

   cd python-docs-samples/functions/helloworld/

   Go

   cd golang-samples/functions/helloworld/

   Java

   cd java-docs-samples/functions/helloworld/hello-gcs/

   C#

   cd dotnet-docs-samples/functions/helloworld/HelloGcs/

   Ruby

   cd ruby-docs-samples/functions/helloworld/storage/

Deploying and triggering the function

Currently, Cloud Storage functions are based on Pub/Sub notifications from Cloud Storage and support similar event types:

• finalize

• delete

• archive

• metadata update

The following sections describe how to deploy and trigger a function for each of these event types.

Object Finalize

Object finalize events trigger when a "write" of a Cloud Storage Object is successfully finalized. In particular, this means that creating a new object or overwriting an existing object triggers this event. Archive and metadata update operations are ignored by this trigger.

Object Finalize: deploying the function

Take a look at the sample function, which handles Cloud Storage events:

Node.js

functions/helloworld/index.js

View on GitHub

/**
 * Generic background Cloud Function to be triggered by Cloud Storage.
 *
 * @param {object} file The Cloud Storage file metadata.
 * @param {object} context The event metadata.
 */
exports.helloGCS = (file, context) => {
  console.log(`  Event: ${context.eventId}`);
  console.log(`  Event Type: ${context.eventType}`);
  console.log(`  Bucket: ${file.bucket}`);
  console.log(`  File: ${file.name}`);
  console.log(`  Metageneration: ${file.metageneration}`);
  console.log(`  Created: ${file.timeCreated}`);
  console.log(`  Updated: ${file.updated}`);
};

Python

functions/helloworld/main.py

View on GitHub

def hello_gcs(event, context):
    """Background Cloud Function to be triggered by Cloud Storage.
       This generic function logs relevant data when a file is changed.

    Args:
        event (dict):  The dictionary with data specific to this type of event.
                       The `data` field contains a description of the event in
                       the Cloud Storage `object` format described here:
                       https://cloud.google.com/storage/docs/json_api/v1/objects#resource
        context (google.cloud.functions.Context): Metadata of triggering event.
    Returns:
        None; the output is written to Stackdriver Logging
    """

    print('Event ID: {}'.format(context.event_id))
    print('Event type: {}'.format(context.event_type))
    print('Bucket: {}'.format(event['bucket']))
    print('File: {}'.format(event['name']))
    print('Metageneration: {}'.format(event['metageneration']))
    print('Created: {}'.format(event['timeCreated']))
    print('Updated: {}'.format(event['updated']))

Go

functions/helloworld/hello_cloud_storage.go

View on GitHub

// Package helloworld provides a set of Cloud Functions samples.
package helloworld

import (
	"context"
	"fmt"
	"log"
	"time"

	"cloud.google.com/go/functions/metadata"
)

// GCSEvent is the payload of a GCS event.
type GCSEvent struct {
	Kind                    string                 `json:"kind"`
	ID                      string                 `json:"id"`
	SelfLink                string                 `json:"selfLink"`
	Name                    string                 `json:"name"`
	Bucket                  string                 `json:"bucket"`
	Generation              string                 `json:"generation"`
	Metageneration          string                 `json:"metageneration"`
	ContentType             string                 `json:"contentType"`
	TimeCreated             time.Time              `json:"timeCreated"`
	Updated                 time.Time              `json:"updated"`
	TemporaryHold           bool                   `json:"temporaryHold"`
	EventBasedHold          bool                   `json:"eventBasedHold"`
	RetentionExpirationTime time.Time              `json:"retentionExpirationTime"`
	StorageClass            string                 `json:"storageClass"`
	TimeStorageClassUpdated time.Time              `json:"timeStorageClassUpdated"`
	Size                    string                 `json:"size"`
	MD5Hash                 string                 `json:"md5Hash"`
	MediaLink               string                 `json:"mediaLink"`
	ContentEncoding         string                 `json:"contentEncoding"`
	ContentDisposition      string                 `json:"contentDisposition"`
	CacheControl            string                 `json:"cacheControl"`
	Metadata                map[string]interface{} `json:"metadata"`
	CRC32C                  string                 `json:"crc32c"`
	ComponentCount          int                    `json:"componentCount"`
	Etag                    string                 `json:"etag"`
	CustomerEncryption      struct {
		EncryptionAlgorithm string `json:"encryptionAlgorithm"`
		KeySha256           string `json:"keySha256"`
	}
	KMSKeyName    string `json:"kmsKeyName"`
	ResourceState string `json:"resourceState"`
}

// HelloGCS consumes a GCS event.
func HelloGCS(ctx context.Context, e GCSEvent) error {
	meta, err := metadata.FromContext(ctx)
	if err != nil {
		return fmt.Errorf("metadata.FromContext: %v", err)
	}
	log.Printf("Event ID: %v\n", meta.EventID)
	log.Printf("Event type: %v\n", meta.EventType)
	log.Printf("Bucket: %v\n", e.Bucket)
	log.Printf("File: %v\n", e.Name)
	log.Printf("Metageneration: %v\n", e.Metageneration)
	log.Printf("Created: %v\n", e.TimeCreated)
	log.Printf("Updated: %v\n", e.Updated)
	return nil
}

Java

functions/helloworld/hello-gcs/src/main/java/functions/HelloGcs.java

View on GitHub

import com.google.cloud.functions.BackgroundFunction;
import com.google.cloud.functions.Context;
import functions.eventpojos.GcsEvent;
import java.util.logging.Logger;

public class HelloGcs implements BackgroundFunction<GcsEvent> {
  private static final Logger logger = Logger.getLogger(HelloGcs.class.getName());

  @Override
  public void accept(GcsEvent event, Context context) {
    logger.info("Event: " + context.eventId());
    logger.info("Event Type: " + context.eventType());
    logger.info("Bucket: " + event.getBucket());
    logger.info("File: " + event.getName());
    logger.info("Metageneration: " + event.getMetageneration());
    logger.info("Created: " + event.getTimeCreated());
    logger.info("Updated: " + event.getUpdated());
  }
}

C#

functions/helloworld/HelloGcs/Function.cs

View on GitHub

using CloudNative.CloudEvents;
using Google.Cloud.Functions.Framework;
using Google.Events.Protobuf.Cloud.Storage.V1;
using Microsoft.Extensions.Logging;
using System.Threading;
using System.Threading.Tasks;

namespace HelloGcs
{
    public class Function : ICloudEventFunction<StorageObjectData>
    {
        private readonly ILogger _logger;

        public Function(ILogger<Function> logger) =>
            _logger = logger;

        public Task HandleAsync(CloudEvent cloudEvent, StorageObjectData data, CancellationToken cancellationToken)
        {
            _logger.LogInformation("Event: {event}", cloudEvent.Id);
            _logger.LogInformation("Event Type: {type}", cloudEvent.Type);
            _logger.LogInformation("Bucket: {bucket}", data.Bucket);
            _logger.LogInformation("File: {file}", data.Name);
            _logger.LogInformation("Metageneration: {metageneration}", data.Metageneration);
            _logger.LogInformation("Created: {created:s}", data.TimeCreated?.ToDateTimeOffset());
            _logger.LogInformation("Updated: {updated:s}", data.Updated?.ToDateTimeOffset());
            return Task.CompletedTask;
        }
    }
}

Ruby

functions/helloworld/storage/app.rb

View on GitHub

require "functions_framework"

FunctionsFramework.cloud_event "hello_gcs" do |event|
  # The event parameter is a CloudEvents::Event::V1 object.
  # See https://cloudevents.github.io/sdk-ruby/latest/CloudEvents/Event/V1.html
  payload = event.data

  logger.info "Event: #{event.id}"
  logger.info "Event Type: #{event.type}"
  logger.info "Bucket: #{payload['bucket']}"
  logger.info "File: #{payload['name']}"
  logger.info "Metageneration: #{payload['metageneration']}"
  logger.info "Created: #{payload['timeCreated']}"
  logger.info "Updated: #{payload['updated']}"
end

To deploy the function, run the following command in the directory where the sample code is located:

Node.js

gcloud functions deploy helloGCS \
--runtime nodejs10 \
--trigger-resource YOUR_TRIGGER_BUCKET_NAME \
--trigger-event google.storage.object.finalize

You can use the following values for the --runtime flag to specify your preferred Node.js version:

• nodejs10
• nodejs12
• nodejs14 (public preview)

Python

gcloud functions deploy hello_gcs \
--runtime python37 \
--trigger-resource YOUR_TRIGGER_BUCKET_NAME \
--trigger-event google.storage.object.finalize

You can use the following values for the --runtime flag to specify your preferred Python version:

• python37
• python38
• python39 (public preview)

Go

gcloud functions deploy HelloGCSInfo \
--runtime go111 \
--trigger-resource YOUR_TRIGGER_BUCKET_NAME \
--trigger-event google.storage.object.finalize

You can use the following values for the --runtime flag to specify your preferred Go version:

• go111
• go113

Java

gcloud functions deploy java-gcs-function \
--entry-point functions.HelloGcs \
--runtime java11 \
--memory 512MB \
--trigger-resource YOUR_TRIGGER_BUCKET_NAME \
--trigger-event google.storage.object.finalize

C#

gcloud functions deploy csharp-gcs-function \
--entry-point HelloGcs.Function \
--runtime dotnet3 \
--trigger-resource YOUR_TRIGGER_BUCKET_NAME \
--trigger-event google.storage.object.finalize

Ruby

gcloud functions deploy hello_gcs --runtime ruby26 \
--trigger-resource YOUR_TRIGGER_BUCKET_NAME \
--trigger-event google.storage.object.finalize

You can use the following values for the --runtime flag to specify your preferred Ruby version:

• ruby26
• ruby27

where YOUR_TRIGGER_BUCKET_NAME is the name of the Cloud Storage bucket that triggers the function.

Object Finalize: triggering the function

To trigger the function:

1. Create an empty gcf-test.txt file in the directory where the sample code is located.

2. Upload the file to Cloud Storage in order to trigger the function:

   gsutil cp gcf-test.txt gs://YOUR_TRIGGER_BUCKET_NAME
   

   where YOUR_TRIGGER_BUCKET_NAME is the name of your Cloud Storage bucket where you will upload a test file.

3. Check the logs to make sure the executions have completed:

   gcloud functions logs read --limit 50
   

Object Delete

Object delete events are most useful for non-versioning buckets. They are triggered when an old version of an object is deleted. In addition, they are triggered when an object is overwritten. Object delete triggers can also be used with versioning buckets, triggering when a version of an object is permanently deleted.

Object Delete: deploying the function

Using the same sample code as in the finalize example, deploy the function with object delete as the trigger event. Run the following command in the directory where the sample code is located:

Node.js

gcloud functions deploy helloGCS \
--runtime nodejs10 \
--trigger-resource YOUR_TRIGGER_BUCKET_NAME \
--trigger-event google.storage.object.delete

You can use the following values for the --runtime flag to specify your preferred Node.js version:

• nodejs10
• nodejs12
• nodejs14 (public preview)

Python

gcloud functions deploy hello_gcs \
--runtime python37 \
--trigger-resource YOUR_TRIGGER_BUCKET_NAME \
--trigger-event google.storage.object.delete

You can use the following values for the --runtime flag to specify your preferred Python version:

• python37
• python38
• python39 (public preview)

Go

gcloud functions deploy HelloGCSInfo \
--runtime go111 \
--trigger-resource YOUR_TRIGGER_BUCKET_NAME \
--trigger-event google.storage.object.delete

You can use the following values for the --runtime flag to specify your preferred Go version:

• go111
• go113

Java

gcloud functions deploy java-gcs-function \
--entry-point functions.HelloGcs \
--runtime java11 \
--memory 512MB \
--trigger-resource YOUR_TRIGGER_BUCKET_NAME \
--trigger-event google.storage.object.delete

C#

gcloud functions deploy csharp-gcs-function \
--entry-point HelloGcs.Function \
--runtime dotnet3 \
--trigger-resource YOUR_TRIGGER_BUCKET_NAME \
--trigger-event google.storage.object.delete

Ruby

gcloud functions deploy hello_gcs --runtime ruby26 \
--trigger-resource YOUR_TRIGGER_BUCKET_NAME \
--trigger-event google.storage.object.delete

You can use the following values for the --runtime flag to specify your preferred Ruby version:

• ruby26
• ruby27

where YOUR_TRIGGER_BUCKET_NAME is the name of the Cloud Storage bucket that triggers the function.

Object Delete: triggering the function

To trigger the function:

1. Create an empty gcf-test.txt file in the directory where the sample code is located.

2. Make sure that your bucket is non-versioning:

   gsutil versioning set off gs://YOUR_TRIGGER_BUCKET_NAME
   

3. Upload the file to Cloud Storage:

   gsutil cp gcf-test.txt gs://YOUR_TRIGGER_BUCKET_NAME
   

   where YOUR_TRIGGER_BUCKET_NAME is the name of your Cloud Storage bucket where you will upload a test file. At this point the function should not execute yet.

4. Delete the file to trigger the function:

   gsutil rm gs://YOUR_TRIGGER_BUCKET_NAME/gcf-test.txt
   

5. Check the logs to make sure the executions have completed:

   gcloud functions logs read --limit 50
   

Note that the function may take some time to finish executing.

Object Archive

Object archive events can be used only with versioning buckets. They are triggered when an old version of an object is archived. In particular, this means that when an object is overwritten or deleted, an archive event is triggered.

Object Archive: deploying the function

Using the same sample code as in the finalize example, deploy the function with object archive as the trigger event. Run the following command in the directory where the sample code is located:

Node.js

gcloud functions deploy helloGCS \
--runtime nodejs10 \
--trigger-resource YOUR_TRIGGER_BUCKET_NAME \
--trigger-event google.storage.object.archive

You can use the following values for the --runtime flag to specify your preferred Node.js version:

• nodejs10
• nodejs12
• nodejs14 (public preview)

Python

gcloud functions deploy hello_gcs \
--runtime python37 \
--trigger-resource YOUR_TRIGGER_BUCKET_NAME \
--trigger-event google.storage.object.archive

You can use the following values for the --runtime flag to specify your preferred Python version:

• python37
• python38
• python39 (public preview)

Go

gcloud functions deploy HelloGCSInfo \
--runtime go111 \
--trigger-resource YOUR_TRIGGER_BUCKET_NAME \
--trigger-event google.storage.object.archive

You can use the following values for the --runtime flag to specify your preferred Go version:

• go111
• go113

Java

gcloud functions deploy java-gcs-function \
--entry-point functions.HelloGcs \
--runtime java11 \
--memory 512MB \
--trigger-resource YOUR_TRIGGER_BUCKET_NAME \
--trigger-event google.storage.object.archive

C#

gcloud functions deploy csharp-gcs-function \
--entry-point HelloGcs.Function \
--runtime dotnet3 \
--trigger-resource YOUR_TRIGGER_BUCKET_NAME \
--trigger-event google.storage.object.archive

Ruby

gcloud functions deploy hello_gcs --runtime ruby26 \
--trigger-resource YOUR_TRIGGER_BUCKET_NAME \
--trigger-event google.storage.object.archive

You can use the following values for the --runtime flag to specify your preferred Ruby version:

• ruby26
• ruby27

where YOUR_TRIGGER_BUCKET_NAME is the name of the Cloud Storage bucket that triggers the function.

Object Archive: triggering the function

To trigger the function:

1. Create an empty gcf-test.txt file in the directory where the sample code is located.

2. Make sure that your bucket has versioning enabled:

   gsutil versioning set on gs://YOUR_TRIGGER_BUCKET_NAME
   

3. Upload the file to Cloud Storage:

   gsutil cp gcf-test.txt gs://YOUR_TRIGGER_BUCKET_NAME
   

   where YOUR_TRIGGER_BUCKET_NAME is the name of your Cloud Storage bucket where you will upload a test file. At this point the function should not execute yet.

4. Archive the file to trigger the function:

   gsutil rm gs://YOUR_TRIGGER_BUCKET_NAME/gcf-test.txt
   

5. Watch the logs to make sure the executions have completed:

   gcloud functions logs read --limit 50
   

Object Metadata Update

Metadata update events are triggered when the metadata of existing object is updated.

Object Metadata Update: deploying the function

Using the same sample code as in the finalize example, deploy the function with metadata update as the trigger event. Run the following command in the directory where the sample code is located:

Node.js

gcloud functions deploy helloGCS \
--runtime nodejs10 \
--trigger-resource YOUR_TRIGGER_BUCKET_NAME \
--trigger-event google.storage.object.metadataUpdate

You can use the following values for the --runtime flag to specify your preferred Node.js version:

• nodejs10
• nodejs12
• nodejs14 (public preview)

Python

gcloud functions deploy hello_gcs \
--runtime python37 \
--trigger-resource YOUR_TRIGGER_BUCKET_NAME \
--trigger-event google.storage.object.metadataUpdate

You can use the following values for the --runtime flag to specify your preferred Python version:

• python37
• python38
• python39 (public preview)

Go

gcloud functions deploy HelloGCSInfo \
--runtime go111 \
--trigger-resource YOUR_TRIGGER_BUCKET_NAME \
--trigger-event google.storage.object.metadataUpdate

You can use the following values for the --runtime flag to specify your preferred Go version:

• go111
• go113

Java

gcloud functions deploy java-gcs-function \
--entry-point functions.HelloGcs \
--runtime java11 \
--memory 512MB \
--trigger-resource YOUR_TRIGGER_BUCKET_NAME \
--trigger-event google.storage.object.metadataUpdate

C#

gcloud functions deploy csharp-gcs-function \
--entry-point HelloGcs.Function \
--runtime dotnet3 \
--trigger-resource YOUR_TRIGGER_BUCKET_NAME \
--trigger-event google.storage.object.metadataUpdate

Ruby

gcloud functions deploy hello_gcs --runtime ruby26 \
--trigger-resource YOUR_TRIGGER_BUCKET_NAME \
--trigger-event google.storage.object.metadataUpdate

You can use the following values for the --runtime flag to specify your preferred Ruby version:

• ruby26
• ruby27

where YOUR_TRIGGER_BUCKET_NAME is the name of the Cloud Storage bucket that triggers the function.

Object Metadata Update: triggering the function

To trigger the function:

1. Create an empty gcf-test.txt file in the directory where the sample code is located.

2. Make sure that your bucket is non-versioning:

   gsutil versioning set off gs://YOUR_TRIGGER_BUCKET_NAME
   

3. Upload the file to Cloud Storage:

   gsutil cp gcf-test.txt gs://YOUR_TRIGGER_BUCKET_NAME
   

   where YOUR_TRIGGER_BUCKET_NAME is the name of your Cloud Storage bucket where you will upload a test file. At this point the function should not execute yet.

4. Update the metadata of the file:

   gsutil -m setmeta -h "Content-Type:text/plain" gs://YOUR_TRIGGER_BUCKET_NAME/gcf-test.txt
   

5. Watch the logs to make sure the executions have completed:

   gcloud functions logs read --limit 50
   

Note: The events are delivered with Google Cloud Storage Cloud Pub/Sub Notifications. Therefore, registering too many notifications may exceed the notification limit for a bucket and make further function creations with this bucket impossible. Attempts to do so will fail with the error Cloud Storage bucket ...: Pub/Sub notification limit reached. See the Google Cloud Storage Cloud Pub/Sub Notifications documentation for more information about notification limits.

Cleaning up

To avoid incurring charges to your Google Cloud account for the resources used in this tutorial, either delete the project that contains the resources, or keep the project and delete the individual resources.

Deleting the project

The easiest way to eliminate billing is to delete the project that you created for the tutorial.

To delete the project:

Caution: Deleting a project has the following effects:

• Everything in the project is deleted. If you used an existing project for this tutorial, when you delete it, you also delete any other work you've done in the project.
• Custom project IDs are lost. When you created this project, you might have created a custom project ID that you want to use in the future. To preserve the URLs that use the project ID, such as an appspot.com URL, delete selected resources inside the project instead of deleting the whole project.

If you plan to explore multiple tutorials and quickstarts, reusing projects can help you avoid exceeding project quota limits.

1. In the Cloud Console, go to the Manage resources page.

   Go to Manage resources

2. In the project list, select the project that you want to delete, and then click Delete.
3. In the dialog, type the project ID, and then click Shut down to delete the project.

Deleting the Cloud Function

Deleting Cloud Functions does not remove any resources stored in Cloud Storage.

To delete the Cloud Function you created in this tutorial, run the following command:

Node.js

gcloud functions delete helloGCS 

Python

gcloud functions delete hello_gcs 

Go

gcloud functions delete HelloGCSInfo 

Java

gcloud functions delete java-gcs-function 

C#

gcloud functions delete csharp-gcs-function 

Ruby

gcloud functions delete hello_gcs 

You can also delete Cloud Functions from the Google Cloud Console.