Google Cloud Storage Triggers

Cloud Functions uses event-driven functions to handle events from your Cloud infrastructure. For example, Cloud Functions can respond to change notifications emerging from Google Cloud Storage. These notifications can be configured to trigger in response to various events inside a bucket—object creation, deletion, archiving and metadata updates.

Note: Cloud Functions can only be triggered by Cloud Storage buckets in the same Google Cloud Platform project.

Event types

Cloud Storage events used by Cloud Functions are based on Cloud Pub/Sub Notifications for Google Cloud Storage and are provided in the Cloud Storage JSON API format.

Note: Since Storage triggers use Cloud Pub/Sub, this means they have at-least-once delivery. Your function could execute more than once for a given event as a result.

Storage-triggered functions support four trigger types. These trigger type values are used upon function deployment to specify which Cloud Storage events will trigger your functions:

• google.storage.object.finalize (default)

• google.storage.object.delete

• google.storage.object.archive

• google.storage.object.metadataUpdate

Sample function code

The following sample function logs relevant data when an event occurs.

You specify the trigger type when you deploy the function. For example, the deployment example below uses the function to log every time an object is created by specifying the google.storage.object.finalize trigger type.

For a full tutorial on running this code, see the Cloud storage tutorial:

Node.js

 

functions/helloworld/index.js

View on GitHub Feedback

/**
 * Generic background Cloud Function to be triggered by Cloud Storage.
 *
 * @param {object} file The Cloud Storage file metadata.
 * @param {object} context The event metadata.
 */
exports.helloGCS = (file, context) => {
  console.log(`  Event: ${context.eventId}`);
  console.log(`  Event Type: ${context.eventType}`);
  console.log(`  Bucket: ${file.bucket}`);
  console.log(`  File: ${file.name}`);
  console.log(`  Metageneration: ${file.metageneration}`);
  console.log(`  Created: ${file.timeCreated}`);
  console.log(`  Updated: ${file.updated}`);
};

Python

 

functions/helloworld/main.py

View on GitHub Feedback

def hello_gcs(event, context):
    """Background Cloud Function to be triggered by Cloud Storage.
       This generic function logs relevant data when a file is changed.

    Args:
        event (dict):  The dictionary with data specific to this type of event.
                       The `data` field contains a description of the event in
                       the Cloud Storage `object` format described here:
                       https://cloud.google.com/storage/docs/json_api/v1/objects#resource
        context (google.cloud.functions.Context): Metadata of triggering event.
    Returns:
        None; the output is written to Stackdriver Logging
    """

    print('Event ID: {}'.format(context.event_id))
    print('Event type: {}'.format(context.event_type))
    print('Bucket: {}'.format(event['bucket']))
    print('File: {}'.format(event['name']))
    print('Metageneration: {}'.format(event['metageneration']))
    print('Created: {}'.format(event['timeCreated']))
    print('Updated: {}'.format(event['updated']))

Go

 

functions/helloworld/hello_cloud_storage.go

View on GitHub Feedback

// Package helloworld provides a set of Cloud Functions samples.
package helloworld

import (
	"context"
	"fmt"
	"log"
	"time"

	"cloud.google.com/go/functions/metadata"
)

// GCSEvent is the payload of a GCS event.
type GCSEvent struct {
	Kind                    string                 `json:"kind"`
	ID                      string                 `json:"id"`
	SelfLink                string                 `json:"selfLink"`
	Name                    string                 `json:"name"`
	Bucket                  string                 `json:"bucket"`
	Generation              string                 `json:"generation"`
	Metageneration          string                 `json:"metageneration"`
	ContentType             string                 `json:"contentType"`
	TimeCreated             time.Time              `json:"timeCreated"`
	Updated                 time.Time              `json:"updated"`
	TemporaryHold           bool                   `json:"temporaryHold"`
	EventBasedHold          bool                   `json:"eventBasedHold"`
	RetentionExpirationTime time.Time              `json:"retentionExpirationTime"`
	StorageClass            string                 `json:"storageClass"`
	TimeStorageClassUpdated time.Time              `json:"timeStorageClassUpdated"`
	Size                    string                 `json:"size"`
	MD5Hash                 string                 `json:"md5Hash"`
	MediaLink               string                 `json:"mediaLink"`
	ContentEncoding         string                 `json:"contentEncoding"`
	ContentDisposition      string                 `json:"contentDisposition"`
	CacheControl            string                 `json:"cacheControl"`
	Metadata                map[string]interface{} `json:"metadata"`
	CRC32C                  string                 `json:"crc32c"`
	ComponentCount          int                    `json:"componentCount"`
	Etag                    string                 `json:"etag"`
	CustomerEncryption      struct {
		EncryptionAlgorithm string `json:"encryptionAlgorithm"`
		KeySha256           string `json:"keySha256"`
	}
	KMSKeyName    string `json:"kmsKeyName"`
	ResourceState string `json:"resourceState"`
}

// HelloGCS consumes a GCS event.
func HelloGCS(ctx context.Context, e GCSEvent) error {
	meta, err := metadata.FromContext(ctx)
	if err != nil {
		return fmt.Errorf("metadata.FromContext: %v", err)
	}
	log.Printf("Event ID: %v\n", meta.EventID)
	log.Printf("Event type: %v\n", meta.EventType)
	log.Printf("Bucket: %v\n", e.Bucket)
	log.Printf("File: %v\n", e.Name)
	log.Printf("Metageneration: %v\n", e.Metageneration)
	log.Printf("Created: %v\n", e.TimeCreated)
	log.Printf("Updated: %v\n", e.Updated)
	return nil
}

Java

functions/helloworld/hello-gcs/src/main/java/functions/HelloGcs.java

View on GitHub Feedback

import com.google.cloud.functions.BackgroundFunction;
import com.google.cloud.functions.Context;
import functions.eventpojos.GcsEvent;
import java.util.logging.Logger;

public class HelloGcs implements BackgroundFunction<GcsEvent> {
  private static final Logger logger = Logger.getLogger(HelloGcs.class.getName());

  @Override
  public void accept(GcsEvent event, Context context) {
    logger.info("Event: " + context.eventId());
    logger.info("Event Type: " + context.eventType());
    logger.info("Bucket: " + event.getBucket());
    logger.info("File: " + event.getName());
    logger.info("Metageneration: " + event.getMetageneration());
    logger.info("Created: " + event.getTimeCreated());
    logger.info("Updated: " + event.getUpdated());
  }
}

C#

functions/helloworld/HelloGcs/Function.cs

View on GitHub Feedback

using CloudNative.CloudEvents;
using Google.Cloud.Functions.Framework;
using Google.Events.Protobuf.Cloud.Storage.V1;
using Microsoft.Extensions.Logging;
using System.Threading;
using System.Threading.Tasks;

namespace HelloGcs
{
    public class Function : ICloudEventFunction<StorageObjectData>
    {
        private readonly ILogger _logger;

        public Function(ILogger<Function> logger) =>
            _logger = logger;

        public Task HandleAsync(CloudEvent cloudEvent, StorageObjectData data, CancellationToken cancellationToken)
        {
            _logger.LogInformation("Event: {event}", cloudEvent.Id);
            _logger.LogInformation("Event Type: {type}", cloudEvent.Type);
            _logger.LogInformation("Bucket: {bucket}", data.Bucket);
            _logger.LogInformation("File: {file}", data.Name);
            _logger.LogInformation("Metageneration: {metageneration}", data.Metageneration);
            _logger.LogInformation("Created: {created:s}", data.TimeCreated?.ToDateTimeOffset());
            _logger.LogInformation("Updated: {updated:s}", data.Updated?.ToDateTimeOffset());
            return Task.CompletedTask;
        }
    }
}

Ruby

functions/helloworld/storage/app.rb

View on GitHub Feedback

require "functions_framework"

FunctionsFramework.cloud_event "hello_gcs" do |event|
  # The event parameter is a CloudEvents::Event::V1 object.
  # See https://cloudevents.github.io/sdk-ruby/latest/CloudEvents/Event/V1.html
  payload = event.data

  logger.info "Event: #{event.id}"
  logger.info "Event Type: #{event.type}"
  logger.info "Bucket: #{payload['bucket']}"
  logger.info "File: #{payload['name']}"
  logger.info "Metageneration: #{payload['metageneration']}"
  logger.info "Created: #{payload['timeCreated']}"
  logger.info "Updated: #{payload['updated']}"
end

PHP

functions/helloworld_storage/index.php

View on GitHub Feedback

use Google\CloudFunctions\CloudEvent;

function helloGCS(CloudEvent $cloudevent)
{
    $log = fopen(getenv('LOGGER_OUTPUT') ?: 'php://stderr', 'wb');
    $data = $cloudevent->getData();
    fwrite($log, 'Event: ' . $cloudevent->getId() . PHP_EOL);
    fwrite($log, 'Event Type: ' . $cloudevent->getType() . PHP_EOL);
    fwrite($log, 'Bucket: ' . $data['bucket'] . PHP_EOL);
    fwrite($log, 'File: ' . $data['name'] . PHP_EOL);
    fwrite($log, 'Metageneration: ' . $data['metageneration'] . PHP_EOL);
    fwrite($log, 'Created: ' . $data['timeCreated'] . PHP_EOL);
    fwrite($log, 'Updated: ' . $data['updated'] . PHP_EOL);
}

Specifying the trigger type at deployment

The following gcloud command deploys the function with an object.finalize trigger.

Node.js

gcloud functions deploy helloGCS \
--runtime nodejs16 \
--trigger-resource YOUR_TRIGGER_BUCKET_NAME \
--trigger-event google.storage.object.finalize

You can use the following values for the --runtime flag to specify your preferred Node.js version:

• nodejs16 (recommended)
• nodejs14
• nodejs12
• nodejs10

Python

gcloud functions deploy hello_gcs \
--runtime python39 \
--trigger-resource YOUR_TRIGGER_BUCKET_NAME \
--trigger-event google.storage.object.finalize

You can use the following values for the --runtime flag to specify your preferred Python version:

• python39 (recommended)
• python38
• python37

Go

gcloud functions deploy HelloGCS \
--runtime go116 \
--trigger-resource YOUR_TRIGGER_BUCKET_NAME \
--trigger-event google.storage.object.finalize

You can use the following values for the --runtime flag to specify your preferred Go version:

• go116 (recommended)
• go113
• go111

Java

gcloud functions deploy java-gcs-function \
--entry-point functions.HelloGcs \
--runtime java11 \
--memory 512MB \
--trigger-resource YOUR_TRIGGER_BUCKET_NAME \
--trigger-event google.storage.object.finalize

C#

gcloud functions deploy csharp-gcs-function \
--entry-point HelloGcs.Function \
--runtime dotnet3 \
--trigger-resource YOUR_TRIGGER_BUCKET_NAME \
--trigger-event google.storage.object.finalize

Ruby

gcloud functions deploy hello_gcs --runtime ruby27 \
--trigger-resource YOUR_TRIGGER_BUCKET_NAME \
--trigger-event google.storage.object.finalize

You can use the following values for the --runtime flag to specify your preferred Ruby version:

• ruby27 (recommended)
• ruby26

PHP

gcloud functions deploy helloGCS --runtime php74 \
--trigger-resource YOUR_TRIGGER_BUCKET_NAME \
--trigger-event google.storage.object.finalize

where YOUR_TRIGGER_BUCKET_NAME is the name of the Cloud Storage bucket that the function will monitor.

Object Finalize

Trigger type value: google.storage.object.finalize

This event is sent when a new object is created (or an existing object is overwritten, and a new generation of that object is created) in the bucket.

Object Delete

Trigger type value: google.storage.object.delete

This event is sent when an object is permanently deleted. Depending on the object versioning setting for a bucket this means:

• For versioning buckets, this is only sent when a version is permanently deleted (but not when an object is archived).

• For non-versioning buckets, this is sent when an object is deleted or overwritten.

Object Archive

Trigger type value: google.storage.object.archive

This event is sent when a live version of an object is archived or deleted.

This event is only sent for versioning buckets.

Object Metadata Update

Trigger type value: google.storage.object.metadataUpdate

This event is sent when the metadata of an existing object changes.

Event structure

Storage event data is delivered in the Cloud Storage object format.

Event delivery mechanism

Events are delivered using Pub/Sub notifications from Cloud Storage.

Notifications limit

A bucket can have up to 10 notification configurations set to trigger for a specific event. Setting up too many notifications for the same bucket might exceed the notifications limit for the bucket and make it impossible to create the function, as indicated by the following error:

Cloud Storage bucket ...: Pub/Sub notification limit reached

If you reach the limit, you can't create a function until you take remedial action, such as removing notifications.

Learn more about the quotas and request limits for Cloud Storage.

Permissions

You must have sufficient permissions on the project that will receive notifications. This includes ensuring that you do the following:

1. Get the email address of the service agent associated with the project that contains your Cloud Storage bucket.

2. Use the email address that you obtained in the previous step to give the service agent the IAM role pubsub.publisher for the relevant Pub/Sub topic.

   Note: There might be a delay of several seconds between assigning the role and having it applied to your service agent. If you grant this permission programmatically, wait 30 seconds before configuring Cloud Storage.

Learn more about configuring Pub/Sub notifications for Cloud Storage.

Legacy Cloud Storage triggers

The gcloud command below deploys a function that is triggered by legacy object change notifications on a specific bucket. Generally, Cloud Pub/Sub notifications are easier to use, more flexible, and more powerful than object change notifications. However, these legacy notifications are supported for legacy functions already consuming these events.

gcloud functions deploy YOUR_FUNCTION_NAME \
--trigger-resource YOUR_TRIGGER_BUCKET_NAME \
--trigger-event providers/cloud.storage/eventTypes/object.change \
FLAGS...

Argument	Description
--trigger-resource NAME	The name of the Cloud Storage bucket the function watches for changes.
--trigger-event NAME	The name of the event type that the function wishes to receive. In this case, it is the legacy object.change event.
FLAGS...	Additional flags you must specify during deployment, such as --runtime. For a full reference, see the gcloud functions deploy documentation.

Next steps

See the Cloud Storage Tutorial for an example of how to implement an event-driven function that is triggered by Cloud Storage.