This page provides an overview of run notifications for the BigQuery Data Transfer Service.
There are two types of run notifications you can configure for the BigQuery Data Transfer Service:
- Pub/Sub notifications: machine-readable notifications sent when a transfer run succeeds or fails
- Email notifications: human-readable notifications sent when a transfer run fails
You can configure each type individually, or you can use both Pub/Sub and email run notifications.
You can send notifications to any Pub/Sub topic in any project for which you have sufficient permissions. Once received by the Pub/Sub topic, the resulting message can be sent to any number of subscribers to the topic.
Before you begin
Before configuring Pub/Sub transfer run notifications, you should:
Enable the Pub/Sub API for the project that will receive notifications.
Have sufficient permissions on the project that will receive notifications:
If you own the project that will receive notifications, you most likely have the necessary permission.
If you plan to create topics for receiving notifications, you should have
Whether you plan to use new or existing topics, you should have
pubsub.topics.setIamPolicypermissions. If you create a topic, you typically have
pubsub.topics.setIamPolicyfor it. The following predefined IAM role has
pubsub.admin. See Pub/Sub access control for more information.
Have an existing Pub/Sub topic that you wish to send notifications to.
Notifications sent to the Pub/Sub topic consist of two parts:
- Attributes: A set of key:value pairs describing the event.
- Payload: A string that contains the metadata of the changed object.
Attributes are key:value pairs contained in all notifications sent by BigQuery Data Transfer Service to your Pub/Sub topic. Notifications always contain the following set of key:value pairs, regardless of the notification's payload:
||The type of event that has just occurred.
||The format of the object payload.
The payload is a string that contains the metadata of the transfer run. The type of payload is not configurable at this time and is provided to accommodate future API version changes.
|JSON_API_V1||The payload will be a UTF-8 JSON-serialized string containing the resource representation of a
Email notifications send human-readable email messages when a transfer run fails. These messages are sent to the user who set up the transfer. The messages are not configurable.
If other users should receive transfer run email notifications, set up email forwarding rules to distribute the messages. If you are using Gmail, you can Automatically forward Gmail messages to another account.
The email notification is sent by the BigQuery Data Transfer Service and contains details on the transfer configuration, the transfer run, and a link to the run history for the failed run. For example:
From: email@example.com To: user_who_set_up_transfer Title: BigQuery Data Transfer Service — Transfer Run Failure — display_name Transfer Configuration Display Name: display_name Source: data_source Destination: project_id Run Summary Run: run_name Schedule Time: schedule_time Run Time: run_time View Run History Google LLC 1600 Amphitheatre Parkway, Mountain View, CA 94043 This email was sent because you indicated you are willing to receive Run Notifications from the BigQuery Data Transfer Service. If you do not wish to receive such emails in the future, click View Transfer Configuration and un-check the "Send E-mail Notifications" option.
Turning on notifications
You can turn on notifications by:
- In the Notification options section of the BigQuery Data Transfer Service console:
- Click the toggle to enable email notifications. When you enable this option, the transfer administrator receives an email notification when a transfer run fails.
- For Select a Pub/Sub topic, choose your topic name or click Create a topic. This option configures Pub/Sub run notifications for your transfer.
Run notification pricing
If you configure Pub/Sub run notifications, you will incur Pub/Sub charges. For more information, see the Pub/Sub Pricing page.
- Learn more about Pub/Sub.
- Learn more about creating Pub/Sub topics.
- Learn more about the BigQuery Data Transfer Service.