Introduction to workflows

To request support or provide feedback for this feature, email bigquery-workflows-preview-feedback@google.com.

This document provides an introduction to workflows in BigQuery. You can use workflows to execute code assets in sequence, on a schedule.

Workflows are powered by Dataform.

A workflow consists of one or more of the following code assets:

You can use workflows to schedule the execution of code assets. For example, you can schedule a SQL query to run daily and update a table with the most recent source data, which can then power a dashboard.

In a workflow with multiple code assets, you define the execution sequence. For example, to train a machine learning model, you can create a workflow in which a SQL query prepares data, and then a subsequent notebook trains the model using that data.

You can schedule a workflow to automatically run at a specified time and frequency.

Limitations

Workflows are subject to the following limitations:

  • You can't add an existing notebook or SQL query to a workflow. To add a workflow task, you need to create a new notebook or SQL query in the workflow.
  • You can't grant access to a selected workflow to other users.
  • Workflows are available only in the Google Cloud console.
  • You can't change the region for storing a workflow after it is created. For more information, see Set the default region for code assets.

Supported regions

All code assets are stored in your default region for code assets. Updating the default region changes the region for all code assets created after that point.

The following table lists the regions where workflows are available:

Region description Region name Details
Africa
Johannesburg africa-south1
Americas
Columbus us-east5
Dallas us-south1 leaf icon Low CO2
Iowa us-central1 leaf icon Low CO2
Los Angeles us-west2
Las Vegas us-west4
Montréal northamerica-northeast1 leaf icon Low CO2
N. Virginia us-east4
Oregon us-west1 leaf icon Low CO2
São Paulo southamerica-east1 leaf icon Low CO2
South Carolina us-east1
Asia Pacific
Hong Kong asia-east2
Jakarta asia-southeast2
Mumbai asia-south1
Seoul asia-northeast3
Singapore asia-southeast1
Sydney australia-southeast1
Taiwan asia-east1
Tokyo asia-northeast1
Europe
Belgium europe-west1 leaf icon Low CO2
Frankfurt europe-west3 leaf icon Low CO2
London europe-west2 leaf icon Low CO2
Madrid europe-southwest1 leaf icon Low CO2
Netherlands europe-west4 leaf icon Low CO2
Turin europe-west12
Zürich europe-west6 leaf icon Low CO2
Middle East
Doha me-central1
Dammam me-central2

Quotas and limits

Workflows are subject to Dataform quotas and limits.

Pricing

Execution of workflow tasks incurs compute and storage charges in BigQuery. For more information, see BigQuery pricing.

Workflows containing notebooks incur Colab Enterprise runtime charges based on the default machine type. For pricing details, see Colab Enterprise pricing.

Each BigQuery workflow run is logged using Cloud Logging. Logging is automatically enabled for BigQuery workflow runs, which can incur Cloud Logging billing charges. For more information, see Cloud Logging pricing.

What's next