Monitor job status using Pub/Sub notifications and BigQuery

This document explains how you can monitor job state changes using Pub/Sub notifications and how to optionally stream those notifications to BigQuery.

Note: Pub/Sub and BigQuery are separate Google Cloud products that have their own pricing. For more information, see Pub/Sub pricing and BigQuery pricing.

Before you begin

• If you haven't used Batch before, review Get started with Batch and enable Batch by completing the prerequisites for projects and users.

Required roles

• To get the permissions that you need to create a Pub/Sub topic, ask your administrator to grant you the Pub/Sub Editor (roles/pubsub.editor) IAM role on the Pub/Sub topic or project. For more information about granting roles, see Manage access.

  You might also be able to get the required permissions through custom roles or other predefined roles.

• To ensure that the job's service account has the necessary permissions to publish Pub/Sub notifications, ask your administrator to grant the job's service account the Pub/Sub Publisher (roles/pubsub.publisher) IAM role on your Pub/Sub topic.

  By default, the job's service account is the default Compute Engine service account.

• To ensure that the Pub/Sub service account has the necessary permissions to stream Pub/Sub notifications to BigQuery, ask your administrator to grant the Pub/Sub service account the BigQuery Data Editor (roles/bigquery.dataEditor) IAM role on the project.

  For more information about assigning this role, see Assign BigQuery roles to the Pub/Sub service account.

• To get the permissions that you need to query Pub/Sub notifications in BigQuery, ask your administrator to grant you the following IAM roles:

  • BigQuery Job User (roles/bigquery.jobUser) on the project
  • BigQuery Data Viewer (roles/bigquery.dataViewer) on your BigQuery table

Use Pub/Sub notifications with Batch

Batch supports Pub/Sub notifications about job and task states for alerts, observability, or for use with BigQuery.

You can receive notifications any time a job or task state changes, or you can filter based on specified job or task states. Task state notifications attributes are applied for all tasks in a job, not for specific tasks within a job. For more information about job and task states, see Job lifecycle.

Note: Pub/Sub doesn't send notifications for the first state a job or task enters. Pub/Sub only sends notifications when a job moves from one state to another.

Enable notifications for a job

To enable Pub/Sub notifications for a job, do the following:

1. Create a Pub/Sub topic in the same project that you want to run your Batch job in.
2. Optionally, configure Pub/Sub to stream notifications to BigQuery.
3. Create a job that sends Pub/Sub notifications.

Configure Pub/Sub to stream notifications to BigQuery

If you want to retain the history of job and task state changes and query those changes, you can stream Pub/Sub notifications to BigQuery, by doing the following:

1. If you haven't already done so, enable notifications for a job.

2. If you don't already have one, create a BigQuery dataset.

3. Create a BigQuery table with the following schema definition:

   [
     {
       "name": "data",
       "type": "STRING"
     },
     {
       "name": "subscription_name",
       "type": "STRING"
     },
     {
       "name": "message_id",
       "type": "STRING"
     },
     {
       "name": "attributes",
       "type": "JSON"
     },
     {
       "name": "publish_time",
       "type": "TIMESTAMP",
       "mode": "NULLABLE"
     }
   ]
   

4. Create a Pub/Sub subscription that streams to BigQuery and specify the following:

   • Topic: select the Pub/Sub topic you previously created.
   • Project: select the project that contains your BigQuery table.
   • Dataset: select the dataset that contains your BigQuery table.
   • Table: select the table you created in the previous step.
   • Select Use topic schema.
   • Select Write metadata.

As the Batch job runs, Pub/Sub streams notifications to BigQuery.

Analyze notifications in BigQuery

Analyze Pub/Sub notifications in BigQuery, by running a query on the attributes column of the BigQuery table. The attributes that you can query depend on whether you configured notifications for tasks or jobs.

You can query the following attributes of task notifications:

• JobUID
• NewTaskState
• Region
• TaskName
• TaskUID
• Type

You can query the following attributes of job notifications:

• JobUID
• JobName
• NewJobState
• Region
• Type

For example, the following query shows the JobUID associated with each notification:

select attributes.JobUID from `example-table`

What's next

• Learn how to Analyze a job using logs.