Cloud Pub/Sub Notifications for Google Cloud Storage

This page provides an overview of Google Cloud Pub/Sub Notifications for Google Cloud Storage. To learn how to set up and use Cloud Pub/Sub Notifications, see Registering Object Changes.

Overview

Cloud Pub/Sub Notifications sends information about changes to objects in your buckets to Google Cloud Pub/Sub, where the information is added to a Cloud Pub/Sub topic of your choice in the form of messages. For example, you can track objects that are created and deleted in your bucket. Each notification contains information describing both the event that triggered it and the object that changed.

You can send notifications to any Cloud Pub/Sub topic in any project for which you have sufficient permissions. Once received by the Cloud Pub/Sub topic, the resulting message can be sent to any number of subscribers to the topic. See Prerequisites for information on connecting your Cloud Storage buckets to a Cloud Pub/Sub topic.

Other notification options

Subscribing to Cloud Pub/Sub Notifications is a versatile way to trigger alerts and actions in response to changes in a bucket. The following options are also available:

  • Google Cloud Functions: If you only want to trigger a lightweight, stand-alone function in response to events and don't want to manage a Cloud Pub/Sub topic, use Cloud Functions. Cloud Functions allow you to execute JavaScript functions when an object in your bucket changes. See the associated tutorial for a demonstration of using Cloud Functions with Cloud Storage.

  • Object Change Notification: Object Change Notification is a separate, older feature within Cloud Storage for generating notifications. This feature sends HTTPS messages to a client application that you've set up separately. Generally, Cloud Pub/Sub Notifications are cheaper, easier to use, and more flexible.

Notification configurations

A notification configuration is a rule you attach to a bucket that specifies:

  • The topic in Cloud Pub/Sub that receives notifications.
  • The events that trigger a notification to be sent.
  • The information contained within notifications.

You can attach multiple notification configurations to a bucket, but their triggers cannot overlap such that a single event could send multiple notifications.

For example, if you have one notification configuration that sends all deletion notifications to one Cloud Pub/Sub topic, you cannot add a second notification configuration to the bucket that sends deletion notifications to another topic. If you try to do so, you receive an error. However, you can create a second notification that sends object creation notifications, either to the same Cloud Pub/Sub topic as the first notification configuration, or to a different topic.

Each notification configuration is identified by an integer. This integer is returned:

  • When you create the notification configuration.
  • When you list the notification configurations attached to a bucket.
  • In the notificationConfig attribute of each notification triggered by the notification configuration.

Creating and deleting notification configurations increment a bucket's metageneration number.

Event types

The following is a list of event types currently supported by Cloud Storage:

Event type Description
OBJECT_FINALIZE Sent when a new object (or a new generation of an existing object) is successfully created in the bucket. This includes copying or rewriting an existing object. A failed upload does not trigger this event.
OBJECT_METADATA_UPDATE Sent when the metadata of an existing object changes.
OBJECT_DELETE Sent when an object has been permanently deleted. This includes objects that are overwritten or are deleted as part of the bucket's lifecycle configuration. For buckets with object versioning enabled, this is not sent when an object is archived (see OBJECT_ARCHIVE), even if archival occurs via the storage.objects.delete method.
OBJECT_ARCHIVE Only sent when a bucket has enabled object versioning. This event indicates that the live version of an object has become an archived version, either because it was archived or because it was overwritten by the upload of an object of the same name.

Overwriting objects

Overwriting an existing object with a new one of the same name triggers two separate events: OBJECT_FINALIZE for the new version of the object and either OBJECT_ARCHIVE or OBJECT_DELETE for the overwritten object. The OBJECT_FINALIZE event will contain an additional attribute overwroteGeneration, which provides the generation number of the object that was overwritten. The OBJECT_ARCHIVE or OBJECT_DELETE event will contain an additional attribute overwrittenByGeneration, which provides the generation number of the new object.

Notification format

Notifications sent to the Cloud Pub/Sub topic consist of two parts:

  • Attributes: A set of key:value pairs describing the event.
  • Payload: A string that contains the metadata of the changed object.

Attributes

Attributes are key:value pairs contained in all notifications sent by Cloud Storage to your Cloud Pub/Sub topic. Notifications always contain the following set of key:value pairs, regardless of the notification's payload:

Attribute name Example Description
notificationConfig projects/_/buckets/foo/notificationConfigs/3 An identifier for the notification configuration that triggered this notification.
eventType OBJECT_FINALIZE The type of event that has just occurred. See Event types for a list of possible values.
payloadFormat JSON_API_V1 The format of the object payload. See Payload for a list of possible values.
bucketId foo The name of the bucket that contains the changed object.
objectId bar The name of the changed object.
objectGeneration 123456 The generation number of the changed object.

Notifications sometimes contain the following set of key:value pairs, regardless of the notification's payload:

Attribute name Example Description
overwrittenByGeneration 107458 The generation number of the object that overwrote the object that this notification pertains to. This attribute only appears in OBJECT_ARCHIVE or OBJECT_DELETE events in the case of an overwrite.
overwroteGeneration 352947 The generation number of the object that was overwritten by the object that this notification pertains to. This attribute only appears in OBJECT_FINALIZE events in the case of an overwrite.

The following key:value pairs are deprecated and may not appear for new subscriptions:

Attribute name Example Description
resource projects/_/buckets/foo/objects/bar#123456 The path of the Cloud Storage object that changed.

In addition to the above attributes, notifications can contain custom attributes. These attributes are defined when creating a notification configuration, using the -m flag in a gsutil notification command or the custom_attributes object in the body of a POST notificationConfigs JSON request.

Payload

The payload is a string that contains the metadata of the changed object. When you create a notification configuration, you specify a type of payload to include in notifications triggered by that configuration. You can specify the following types of payload:

Payload type Description
NONE No payload is included with the notification.
JSON_API_V1 The payload will be a UTF-8 string containing the resource representation of the object’s metadata.

For OBJECT_DELETE notifications, the metadata contained in the payload represents the object metadata as it was before the delete, along with an additional timeDeleted property. For all other notifications, the metadata included in the payload represents the object metadata after the change occurs.

For example, say you have a notification configuration that tracks OBJECT_METADATA_CHANGE events. If a user changes the contentType property of an object from binary/octet-stream to video/mp4, an OBJECT_METADATA_CHANGE notification is sent, and the metadata in the payload includes "contentType":"video/mp4".

Delivery guarantees

When you add a notification configuration, Cloud Storage may take up to 30 seconds to begin sending notifications associated with it. Once started, Cloud Storage guarantees at-least-once delivery to Cloud Pub/Sub. Cloud Pub/Sub also offers at-least-once delivery to the recipient, which means that you could receive multiple messages, with multiple IDs, that represent the same Cloud Storage event. There is currently no SLA for delivery time, but notifications are typically delivered within seconds. In rare circumstances notifications may be delayed substantially longer.

Notifications are not guaranteed to be published in the order Cloud Pub/Sub receives them. If you plan to modify the Cloud Storage object based on a notification, it is recommended that you use the object's generation and metageneration numbers as preconditions on your update request.

What's next

Monitor your resources on the go

Get the Google Cloud Console app to help you manage your projects.

Send feedback about...

Cloud Storage