Pub/Sub notifications

After you complete a Storage Transfer Service transfer, you might want to start another task, handle an error encountered during a transfer, or log that a transfer occurred. Pub/Sub provides a queue that your programs subscribe to to receive messages when a transfer completes. This enables you to reuse code and to program next steps based on the state of a transfer.

For more information about Pub/Sub, see What is Cloud Pub/Sub

Prerequisites

Before you can use this feature, do the following:

Enable the Pub/Sub API for the project receiving Pub/Sub notifications.

Enable the API
Have an existing Pub/Sub topic that you want to send notifications to.
Get the email address of the service account associated with the project that contains your Storage Transfer Service bucket.
1. Grant the service account the IAM role roles.pubsub.publisher for the desired Pub/Sub topic.
Note: There might be a delay of several seconds between assigning the role and having it applied to your service account. If you grant this permission programmatically, wait 30 seconds before configuring Storage Transfer Service.

Configuring notification settings

To configure Storage Transfer Service Pub/Sub notification settings, you use the Storage Transfer Service transferJobs API, to create NotificationConfig message. The topic will receive notifications on the configured Pub/Sub channel.

Specify the following in your NotificationConfig message:

The Pub/Sub topic to send notifications to
The format of the message, "JSON" or "NONE"
The desired event types, corresponding to completed TransferOperation.Statuses:
- "TRANSFER_OPERATION_SUCCESS"
- "TRANSFER_OPERATION_FAILED"
- "TRANSFER_OPERATION_ABORTED"

The following is an example NotificationConfig message:

{
   ...
   "notificationConfig": {
     "pubsubTopic": "projects/project-id/topics/topic-id",
     "eventTypes": ["TRANSFER_OPERATION_SUCCESS"],
     "payloadFormat": "JSON"
   },
   ...
}

Replace the following:

project-id: The Google Cloud project ID of the transfer
topic-id: The Pub/Sub topic name

For more information, see Storage Transfer Service Pub Sub REST specification.

Notification format

Notifications sent to the Pub/Sub topic consist of two parts:

Attributes: A set of key:value pairs describing the event.
Payload: A string that contains the metadata of the changed object.

Attributes

Attributes are key:value pairs contained in all notifications sent by Storage Transfer Service to your Pub/Sub topic. Notifications always contain the following set of key:value pairs, regardless of the notification's payload:

See PubsubMessage for more information about the Pub/Sub message format.

Attribute name	Example	Description
`eventType`	`TRANSFER_OPERATION_SUCCESS`	Status of the `TransferOperation`, from `NotificationConfig.EventType`
`payloadFormat`	`"JSON"`	Format of the message, either `"JSON"` or `"NONE"`. From `NotificationConfig.PayloadFormat`.
`projectId`	`project-3`	ID of the transfer host project.
`transferJobName`	`transferJobs/123`	Name of the transfer job.
`transferOperationName`	`transferOperations/456`	Name of the transfer operation.

Payload

The payload contains the TransferOperation metadata. When you create a notification configuration, you specify a type of payload to include in notifications triggered by that configuration. You can specify the following types of payload:

Payload type	Description
NONE	No payload is included with the notification.
JSON	The payload is formatted as a JSON response, in application/json.

Example Pub/Sub notification configurations

Notifying for failed transfers only

To receive messages for failed transfers only, submit a TransferJob with a NotificationConfig that filters for failed transfers only:

// REST JSON format:
//   https://cloud.google.com/storage-transfer/docs/create-manage-transfer-program
{
   ...
   "notificationConfig": {
     "pubsubTopic": "projects/project-id/topics/topic-id",
     "eventTypes": ["TRANSFER_OPERATION_FAILED"],
     "payloadFormat: "JSON"
   },
   ...
}

Replace the following:

project-id: The Google Cloud project ID of the transfer
topic-id: The Pub/Sub topic name

Notify on all completed transfers

To notify on all completed transfers, regardless of status, submit a TransferJob with a NotificationConfig without EventType filters:

// REST JSON format:
//   https://cloud.google.com/storage-transfer/docs/create-manage-transfer-program
{
   ...
   "notificationConfig": {
     "pubsubTopic": "projects/project-id/topics/topic-id",
     "payloadFormat: "JSON"
   },
   ...
}

Replace the following:

project-id: The Google Cloud project ID of the transfer
topic-id: The Pub/Sub topic name

Performing arbitrary actions on a transfer

To perform an arbitrary action, you create a Pub/Sub notification using a Cloud function with a Pub/Sub trigger paired with a background cloud function. For more information, see the Cloud Pub/Sub tutorial.

Examples of arbitrary actions include:

Delivery guarantees

Storage Transfer Service sends notifications for any TransferOperations started after you add a notification configuration, and guarantees at-least-once delivery to Pub/Sub. Pub/Sub also offers at-least-once delivery to the recipient, which means that you could receive multiple messages, with multiple IDs, that represent the same Storage Transfer Service event.

Notifications are not guaranteed to be published in the order Pub/Sub receives them.

If a notification consistently cannot be delivered to a Pub/Sub topic, Storage Transfer Service might delete the notification after 7 days. Delivery failure can occur when the Pub/Sub topic no longer exists, when Storage Transfer Service no longer has permission to publish to the topic, or when the project that owns the topic exceeds its publishing quota.