Monitor job status using Pub/Sub notifications and BigQuery

This document explains how you can monitor job and task state changes by setting up Pub/Sub notifications. You can use the notifications for various use cases—for example, this document explains how stream the notifications to a BigQuery table, which you can use to analyze the data. To learn more about the use cases for Pub/Sub notifications, see Choose a subscription type in the Pub/Sub documentation.

Note: Pub/Sub and BigQuery are separate Google Cloud products that have their own pricing. For more information, see Pub/Sub pricing and BigQuery pricing.

Before you begin

• If you haven't used Batch before, review Get started with Batch and enable Batch by completing the prerequisites for projects and users.
• If you want to set up Pub/Sub notifications for your Batch jobs, do the following:

  1. Enable the Pub/Sub API.

     Enable the API

  2. To get the permissions that you need to configure a Pub/Sub topic and subscription, ask your administrator to grant you the Pub/Sub Editor (roles/pubsub.editor) IAM role on a project. For more information about granting roles, see Manage access.

     You might also be able to get the required permissions through custom roles or other predefined roles.

  Note: After you configure a Pub/Sub topic and subscription, your administrator will need to grant additional permissions to various principals to allow jobs to send notifications to that topic. Specifically, the required roles are listed on the Create and run a job that sends Pub/Sub status notifications page.
• Additional requirements vary based on the type of subscription that you decide to configure. For example, if you want to stream and analyze notifications to a BigQuery table, you must do the following:

  1. Enable the BigQuery API.

     Enable the API

  2. To ensure that the Pub/Sub service account has the necessary permissions to stream Pub/Sub notifications to BigQuery, ask your administrator to grant the Pub/Sub service account the BigQuery Data Editor (roles/bigquery.dataEditor) IAM role on the Pub/Sub topic's project.

     For more information about assigning this role, see Assign BigQuery roles to the Pub/Sub service account.

  3. Ensure that any users who you want to analyze notifications in BigQuery have the required permissions.

     To get the permissions that you need to query Pub/Sub notifications in BigQuery, ask your administrator to grant you the following IAM roles:

     • BigQuery Job User (roles/bigquery.jobUser) on the BigQuery table's project
     • BigQuery Data Viewer (roles/bigquery.dataViewer) on the BigQuery table

Set up Pub/Sub notifications for Batch

Batch supports Pub/Sub notifications for changes to job and task states, which you can use for alerts, observability, or analysis.

To set up Pub/Sub notifications for one or more Batch jobs, complete the steps described in the following sections:

1. Create or identify a Pub/Sub topic for your Batch jobs.
2. Create at least one subscription for your topic to receive and use its notifications.
3. Configure jobs to send notifications to your topic.

After you set up Pub/Sub notifications for Batch, you can optionally configure each job to send specific notifications. You can receive notifications any time job or task states change, or you can filter the notifications for specific job or task states. Notably, Pub/Sub doesn't send notifications for the first state a job or task enters; Pub/Sub only sends notifications when a job moves from one state to another. For more information about the attributes you can configure for Pub/Sub notifications, see Create and run a job that sends Pub/Sub status notifications.

Create or identify a Pub/Sub topic for Batch

Create a new Pub/Sub topic or identify an existing Pub/Sub topic that you want to use for Batch notifications.

As explained later in this document, jobs cannot send notifications to this Pub/Sub topic until your administrator grants additional permissions that allow this.

Create a subscription to receive and use the notifications

To receive and make use of messages published to a Pub/Sub topic, you must create at least one subscription to that topic.

For example, one option is to configure your topic to stream its notifications to BigQuery. Then, when you are running jobs that publish notifications, you can receive and analyze the notifications in a BigQuery table. For more information, see Example: Receive and use Pub/Sub notifications in BigQuery in this document.

Otherwise, to learn more about subscriptions and other subscription options, see Subscriptions overview in the Pub/Sub documentation.

Configure jobs to send notifications to your topic

To configure jobs that send Pub/Sub notifications to your topic, follow the instructions on the Create and run a job that sends Pub/Sub status notifications page. Specifically, do the following:

1. If you haven't already, ask your administrator to grant all the permissions required for jobs to send notifications to your Pub/Sub topic. These permissions are listed in the Required roles section of that page.
2. Create and run a job that enables Pub/Sub notifications by defining the job's notifications field.

Example: Receive and use Pub/Sub notifications in BigQuery

The following sections explain an example use case for receiving and using Pub/Sub notifications in BigQuery. By configuring a subscription that streams your topic's Pub/Sub notifications to a BigQuery table, you can retain the history of job and task state changes and query those changes.

Configure a subscription that streams notifications to BigQuery

Configure a Pub/Sub subscription that streams notifications from a topic to a BigQuery table by doing the following:

1. If you haven't already done so, create or identify a Pub/Sub topic for Batch.

2. If you don't already have one, create a BigQuery dataset.

3. Create a BigQuery table with the following schema definition:

   [
     {
       "name": "data",
       "type": "STRING"
     },
     {
       "name": "subscription_name",
       "type": "STRING"
     },
     {
       "name": "message_id",
       "type": "STRING"
     },
     {
       "name": "attributes",
       "type": "JSON"
     },
     {
       "name": "publish_time",
       "type": "TIMESTAMP",
       "mode": "NULLABLE"
     }
   ]
   

4. Create a Pub/Sub subscription that streams to BigQuery and specify the following options:

   • Topic: select the Pub/Sub topic for Batch.
   • Project: select the project that contains your BigQuery table.
   • Dataset: select the dataset that contains your BigQuery table.
   • Table: select the table you created in the previous step.
   • Select Use topic schema.
   • Select Write metadata.

Now, whenever a Pub/Sub notification is sent to that topic, Pub/Sub streams that notification to the BigQuery table. After you configure jobs to send Pub/Sub notifications, you can analyze notifications in BigQuery.

Analyze notifications in BigQuery

You can analyze notifications in BigQuery after you configure a subscription to a BigQuery table for your topic and run jobs that send notifications to your topic. To analyze the notifications in BigQuery, run a query on the attributes column of the BigQuery table. The attributes that you can query depend on whether you configured notifications for tasks or jobs.

You can query the following attributes of task notifications:

• JobUID
• NewTaskState
• Region
• TaskName
• TaskUID
• Type

You can query the following attributes of job notifications:

• JobUID
• JobName
• NewJobState
• Region
• Type

For example, the following query shows the JobUID associated with each notification:

select attributes.JobUID from `example-table`

What's next

• Learn how to create and run a job that sends Pub/Sub status notifications.
• Learn more about Pub/Sub subscriptions.
• Learn more about BigQuery analytics.