Google Cloud Storage Triggers

Cloud Functions can respond to change notifications emerging from Google Cloud Storage. These notifications can be configured to trigger in response to various events inside a bucket—object creation, deletion, archiving and metadata updates.

Note: Cloud Functions can only be triggered by Cloud Storage buckets in the same Google Cloud Platform project.

Event Types

Cloud Storage events used by Cloud Functions are based on Cloud Pub/Sub Notifications for Google Cloud Storage and can be configured in a similar way.

Note: Since Storage Triggers utilize Cloud Pub/Sub, this means they have at-least-once delivery. Your function could execute more than once for a given event as a result.

Supported trigger type values are:

• google.storage.object.finalize (default)

• google.storage.object.delete

• google.storage.object.archive

• google.storage.object.metadataUpdate

Object Finalize

Trigger type value: google.storage.object.finalize

This event is sent when a new object is created (or an existing object is overwritten, and a new generation of that object is created) in the bucket.

For example, the following function logs relevant data when an event occurs:

Node.js

 

functions/helloworld/index.js

View on GitHub Feedback

/**
 * Generic background Cloud Function to be triggered by Cloud Storage.
 *
 * @param {object} file The Cloud Storage file metadata.
 * @param {object} context The event metadata.
 */
exports.helloGCS = (file, context) => {
  console.log(`  Event: ${context.eventId}`);
  console.log(`  Event Type: ${context.eventType}`);
  console.log(`  Bucket: ${file.bucket}`);
  console.log(`  File: ${file.name}`);
  console.log(`  Metageneration: ${file.metageneration}`);
  console.log(`  Created: ${file.timeCreated}`);
  console.log(`  Updated: ${file.updated}`);
};

Python

 

functions/helloworld/main.py

View on GitHub Feedback

def hello_gcs(event, context):
    """Background Cloud Function to be triggered by Cloud Storage.
       This generic function logs relevant data when a file is changed.

    Args:
        event (dict):  The dictionary with data specific to this type of event.
                       The `data` field contains a description of the event in
                       the Cloud Storage `object` format described here:
                       https://cloud.google.com/storage/docs/json_api/v1/objects#resource
        context (google.cloud.functions.Context): Metadata of triggering event.
    Returns:
        None; the output is written to Stackdriver Logging
    """

    print('Event ID: {}'.format(context.event_id))
    print('Event type: {}'.format(context.event_type))
    print('Bucket: {}'.format(event['bucket']))
    print('File: {}'.format(event['name']))
    print('Metageneration: {}'.format(event['metageneration']))
    print('Created: {}'.format(event['timeCreated']))
    print('Updated: {}'.format(event['updated']))

Go

 

functions/helloworld/hello_cloud_storage.go

View on GitHub Feedback

// Package helloworld provides a set of Cloud Functions samples.
package helloworld

import (
	"context"
	"fmt"
	"log"
	"time"

	"cloud.google.com/go/functions/metadata"
)

// GCSEvent is the payload of a GCS event.
type GCSEvent struct {
	Kind                    string                 `json:"kind"`
	ID                      string                 `json:"id"`
	SelfLink                string                 `json:"selfLink"`
	Name                    string                 `json:"name"`
	Bucket                  string                 `json:"bucket"`
	Generation              string                 `json:"generation"`
	Metageneration          string                 `json:"metageneration"`
	ContentType             string                 `json:"contentType"`
	TimeCreated             time.Time              `json:"timeCreated"`
	Updated                 time.Time              `json:"updated"`
	TemporaryHold           bool                   `json:"temporaryHold"`
	EventBasedHold          bool                   `json:"eventBasedHold"`
	RetentionExpirationTime time.Time              `json:"retentionExpirationTime"`
	StorageClass            string                 `json:"storageClass"`
	TimeStorageClassUpdated time.Time              `json:"timeStorageClassUpdated"`
	Size                    string                 `json:"size"`
	MD5Hash                 string                 `json:"md5Hash"`
	MediaLink               string                 `json:"mediaLink"`
	ContentEncoding         string                 `json:"contentEncoding"`
	ContentDisposition      string                 `json:"contentDisposition"`
	CacheControl            string                 `json:"cacheControl"`
	Metadata                map[string]interface{} `json:"metadata"`
	CRC32C                  string                 `json:"crc32c"`
	ComponentCount          int                    `json:"componentCount"`
	Etag                    string                 `json:"etag"`
	CustomerEncryption      struct {
		EncryptionAlgorithm string `json:"encryptionAlgorithm"`
		KeySha256           string `json:"keySha256"`
	}
	KMSKeyName    string `json:"kmsKeyName"`
	ResourceState string `json:"resourceState"`
}

// HelloGCS consumes a GCS event.
func HelloGCS(ctx context.Context, e GCSEvent) error {
	meta, err := metadata.FromContext(ctx)
	if err != nil {
		return fmt.Errorf("metadata.FromContext: %v", err)
	}
	log.Printf("Event ID: %v\n", meta.EventID)
	log.Printf("Event type: %v\n", meta.EventType)
	log.Printf("Bucket: %v\n", e.Bucket)
	log.Printf("File: %v\n", e.Name)
	log.Printf("Metageneration: %v\n", e.Metageneration)
	log.Printf("Created: %v\n", e.TimeCreated)
	log.Printf("Updated: %v\n", e.Updated)
	return nil
}

Java

functions/helloworld/hello-gcs/src/main/java/functions/HelloGcs.java

View on GitHub Feedback

import com.google.cloud.functions.BackgroundFunction;
import com.google.cloud.functions.Context;
import functions.eventpojos.GcsEvent;
import java.util.logging.Logger;

public class HelloGcs implements BackgroundFunction<GcsEvent> {
  private static final Logger logger = Logger.getLogger(HelloGcs.class.getName());

  @Override
  public void accept(GcsEvent event, Context context) {
    logger.info("Event: " + context.eventId());
    logger.info("Event Type: " + context.eventType());
    logger.info("Bucket: " + event.getBucket());
    logger.info("File: " + event.getName());
    logger.info("Metageneration: " + event.getMetageneration());
    logger.info("Created: " + event.getTimeCreated());
    logger.info("Updated: " + event.getUpdated());
  }
}

C#

functions/helloworld/HelloGcs/Function.cs

View on GitHub Feedback

using CloudNative.CloudEvents;
using Google.Cloud.Functions.Framework;
using Google.Events.Protobuf.Cloud.Storage.V1;
using Microsoft.Extensions.Logging;
using System.Threading;
using System.Threading.Tasks;

namespace HelloGcs
{
    public class Function : ICloudEventFunction<StorageObjectData>
    {
        private readonly ILogger _logger;

        public Function(ILogger<Function> logger) =>
            _logger = logger;

        public Task HandleAsync(CloudEvent cloudEvent, StorageObjectData data, CancellationToken cancellationToken)
        {
            _logger.LogInformation("Event: {event}", cloudEvent.Id);
            _logger.LogInformation("Event Type: {type}", cloudEvent.Type);
            _logger.LogInformation("Bucket: {bucket}", data.Bucket);
            _logger.LogInformation("File: {file}", data.Name);
            _logger.LogInformation("Metageneration: {metageneration}", data.Metageneration);
            _logger.LogInformation("Created: {created:s}", data.TimeCreated?.ToDateTimeOffset());
            _logger.LogInformation("Updated: {updated:s}", data.Updated?.ToDateTimeOffset());
            return Task.CompletedTask;
        }
    }
}

To deploy the function with an object finalize trigger, run the following command in the directory that contains the function code:

Node.js

gcloud functions deploy helloGCS \
--runtime nodejs10 \
--trigger-resource YOUR_TRIGGER_BUCKET_NAME \
--trigger-event google.storage.object.finalize

You can use the following values for the --runtime flag to specify your preferred Node.js version:

• nodejs10
• nodejs12

Python

gcloud functions deploy hello_gcs \
--runtime python37 \
--trigger-resource YOUR_TRIGGER_BUCKET_NAME \
--trigger-event google.storage.object.finalize

You can use the following values for the --runtime flag to specify your preferred Python version:

• python37
• python38

Go

gcloud functions deploy HelloGCS \
--runtime go111 \
--trigger-resource YOUR_TRIGGER_BUCKET_NAME \
--trigger-event google.storage.object.finalize

You can use the following values for the --runtime flag to specify your preferred Go version:

• go111
• go113

Java

gcloud functions deploy java-gcs-function \
--entry-point HelloGcs \
--runtime java11 \
--memory 512MB \
--trigger-resource YOUR_TRIGGER_BUCKET_NAME \
--trigger-event google.storage.object.finalize

C#

gcloud functions deploy csharp-gcs-function \
--entry-point HelloGcs.Function \
--runtime dotnet3 \
--trigger-resource YOUR_TRIGGER_BUCKET_NAME \
--trigger-event google.storage.object.finalize

where YOUR_TRIGGER_BUCKET_NAME is the name of the Cloud Storage Bucket that the function will monitor.

Object Delete

Trigger type value: google.storage.object.delete

This event is sent when an object is permanently deleted. Depending on the object versioning setting for a bucket this means:

• For versioning buckets, this is only sent when a version is permanently deleted (but not when an object is archived).

• For non-versioning buckets, this is sent when an object is deleted or overwritten.

Object Archive

Trigger type value: google.storage.object.archive

This event is sent when a live version of an object is archived or deleted.

This event is only sent for versioning buckets.

Object Metadata Update

Trigger type value: google.storage.object.metadataUpdate

This event is sent when the metadata of an existing object changes.

Event structure

Storage event data is delivered in the Cloud Storage object format.

Event delivery mechanism

The events are delivered with Pub/Sub notifications from Cloud Storage. Setting up too many functions against the same bucket may exhaust the notification limit for the bucket and make it impossible to create the function, as indicated by the error Cloud Storage bucket ...: Pub/Sub notification limit reached. See Google Cloud Storage Cloud Pub/Sub Notifications documentation for more information about notification limits.

Legacy Cloud Storage triggers

The gcloud command below deploys a function that is triggered by legacy object change notifications on a specific bucket. Generally, Cloud Pub/Sub Notifications are easier to use, more flexible, and more powerful than object change notifications. However, these legacy notifications are supported for legacy functions already consuming these events.

gcloud functions deploy YOUR_FUNCTION_NAME \
--trigger-resource YOUR_TRIGGER_BUCKET_NAME \
--trigger-event providers/cloud.storage/eventTypes/object.change \
FLAGS...

Argument	Description
--trigger-resource NAME	The name of the Cloud Storage bucket the function watches for changes.
--trigger-event NAME	The name of the event type that the function wishes to receive. In this case, it is the legacy object.change event.
FLAGS...	Additional flags you must specify during deployment, such as --runtime. For a full reference, see the gcloud functions deploy documentation.

Next steps

See the Cloud Storage Tutorial for an example of how to implement a background function that is triggered by Cloud Storage.