BigQuery Data Transfer Service Run Notifications

This page provides an overview of run notifications for the BigQuery Data Transfer Service.

Overview

There are two types of run notifications you can configure for the BigQuery Data Transfer Service:

  • Cloud Pub/Sub notifications — machine-readable notifications sent when a transfer run succeeds or fails
  • Email notifications — human-readable notifications sent when a transfer run fails

You can configure each type individually, or you can use both Cloud Pub/Sub and email run notifications.

Cloud Pub/Sub notifications

Cloud Pub/Sub notifications send information about transfer runs to a Cloud Pub/Sub topic. Cloud Pub/Sub notifications are triggered by completed transfer runs in the following states:

  • SUCCEEDED
  • FAILED
  • CANCELLED

You can send notifications to any Cloud Pub/Sub topic in any project for which you have sufficient permissions. Once received by the Cloud Pub/Sub topic, the resulting message can be sent to any number of subscribers to the topic.

Before you begin

Before configuring Cloud Pub/Sub transfer run notifications, you should:

  1. Enable the Cloud Pub/Sub API for the project that will receive notifications.

    Enable the API

  2. Have sufficient permissions on the project that will receive notifications:

    • If you own the project that will receive notifications, you most likely have the necessary permission.

    • If you plan to create topics for receiving notifications, you should have pubsub.topics.create permissions.

    • Whether you plan to use new or existing topics, you should have pubsub.topics.setIamPolicy permissions. If you create a topic, you typically have pubsub.topics.setIamPolicy for it. The following predefined, project-level IAM roles have pubsub.topics.setIamPolicy permissions:

      • The pubsub.admin predefined role
      • The Owner primitive role

      See Cloud Pub/Sub Access Control for more information.

  3. Have an existing Cloud Pub/Sub topic that you wish to send notifications to.

Notification format

Notifications sent to the Cloud Pub/Sub topic consist of two parts:

  • Attributes: A set of key:value pairs describing the event.
  • Payload: A string that contains the metadata of the changed object.

Attributes

Attributes are key:value pairs contained in all notifications sent by BigQuery Data Transfer Service to your Cloud Pub/Sub topic. Notifications always contain the following set of key:value pairs, regardless of the notification's payload:

Attribute name Example Description
eventType TRANSFER_RUN_FINISHED The type of event that has just occurred. TRANSFER_RUN_FINISHED is the only possible value.
payloadFormat JSON_API_V1 The format of the object payload. JSON_API_V1 is the only possible value.

Payload

The payload is a string that contains the metadata of the transfer run. The type of payload is not configurable at this time and is provided to accommodate future API version changes.

Payload type Description
JSON_API_V1 The payload will be a UTF-8 JSON-serialized string containing the resource representation of a TransferRun.

Email notifications

Email notifications send human-readable email messages when a transfer run fails. These messages are sent to the user who set up the transfer. The messages are not configurable.

If other users should receive transfer run email notifications, set up email forwarding rules to distribute the messages. If you are using Gmail, you can Automatically forward Gmail messages to another account.

The email notification is sent by the BigQuery Data Transfer Service and contains details on the transfer configuration, the transfer run, and a link to the run history for the failed run. For example:

From: bigquery-data-transfer-service-noreply@google.com
To: [USER_WHO_SET_UP_TRANSFER]
Title: BigQuery Data Transfer Service - Transfer Run Failure - [DISPLAY_NAME]

Transfer Configuration
Display Name: [DISPLAY_NAME]
Source: [DATA_SOURCE]
Destination: [PROJECT_ID]

Run Summary
Run: [RUN_NAME]
Schedule Time: [SCHEDULE_TIME]
Run Time: [RUN_TIME]
View Run History

Google LLC 1600 Amphitheatre Parkway, Mountain View, CA 94043

This email was sent because you indicated you are willing to receive Run
Notifications from the BigQuery Data Transfer Service. If you do not wish to
receive such emails in the future, click View Transfer Configuration and
un-check the “Send E-mail Notifications” option.

Run notification pricing

During the Alpha stage, the BigQuery Data Transfer Service will not charge for run notifications. If you configure Cloud Pub/Sub run notifications, you will incur Cloud Pub/Sub charges. For more information, see Cloud Pub/Sub Pricing.

Next steps

Cette page vous a-t-elle été utile ? Évaluez-la :

Envoyer des commentaires concernant…