Schedule production executions
This quickstart walks you through the following steps to schedule production executions in Dataform:
- Create a Dataform repository.
- Grant Dataform access to BigQuery.
Create a release configuration to configure the production environment.
Create a
production
release configuration at set the frequency of creatingproduction
compilation results.Create a workflow configuration to schedule executions in the production environment.
Create a
production
workflow configuration, select theproduction
release configuration, and set a schedule for executingproduction
compilation results.
Before you begin
- Sign in to your Google Cloud account. If you're new to Google Cloud, create an account to evaluate how our products perform in real-world scenarios. New customers also get $300 in free credits to run, test, and deploy workloads.
-
In the Google Cloud console, on the project selector page, select or create a Google Cloud project.
-
Make sure that billing is enabled for your Google Cloud project.
-
Enable the BigQuery and Dataform APIs.
-
In the Google Cloud console, on the project selector page, select or create a Google Cloud project.
-
Make sure that billing is enabled for your Google Cloud project.
-
Enable the BigQuery and Dataform APIs.
Required roles
To get the permissions that you need to perform all tasks in this tutorial, ask your administrator to grant you the following IAM roles:
-
Dataform Admin (
roles/dataform.admin
) on repositories -
Dataform Editor (
roles/dataform.editor
) on workspaces and workflowInvocations
For more information about granting roles, see Manage access to projects, folders, and organizations.
You might also be able to get the required permissions through custom roles or other predefined roles.
Create a Dataform repository
In the Google Cloud console, go to the Dataform page.
Click
Create repository.On the Create repository page, do the following:
In the Repository ID field, enter
quickstart-production
.In the Region list, select
europe-west4
.Click Create.
Click Done.
Grant Dataform access to BigQuery
To execute workflows in BigQuery, the Dataform service account must have the following required roles:
- BigQuery Data Editor on projects to which Dataform needs both read and write access. They usually include the project hosting your Dataform repository.
- BigQuery Data Viewer on projects to which Dataform needs read-only access.
- BigQuery Job User on the project hosting your Dataform repository.
To grant these roles, follow these steps:
In the Google Cloud console, go to the IAM page.
Click Add.
In the New principals field, enter your Dataform service account ID.
In the Select a role drop-down list, select the BigQuery Job User role.
Click Add another role, and then in the Select a role drop-down list, select the BigQuery Data Editor role.
Click Add another role, and then in the Select a role drop-down list, select the BigQuery Data Viewer role.
Click Save.
Create a release configuration
To create production compilation results of the quickstart-production
repository, follow these steps:
In the Google Cloud console, go to the Dataform page.
Click
quickstart-production
.Click Release configurations, and then click New release configuration.
In the Create release configuration pane, configure the following settings:
In the Release ID field, enter
production
.In the Git commitish field, leave the default value
main
.In the Frequency drop-down, select Custom.
In the Custom schedule field, enter
0 16 * * *
.In the Timezone drop-down, select a UTC+1 timezone, for example, Greenwich Mean Time (GMT).
Every day at 4 PM UTC+1, Dataform will compile the
quickstart-production
repository and apply compilation settings configured in this release configuration to createproduction
compilation results.Click Create.
The production
release configuration creates a compilation result of the
entire quickstart-production
repository every day at 4PM UTC+1.
Create a workflow configuration
To schedule execution of production
tables, follow these steps:
In the Google Cloud console, go to the Dataform page.
Click
quickstart-production
.Click Workflow configurations, and then click New workflow configuration.
In the Create workflow configuration pane, in the Configuration ID field, enter
production
.In the Release configuration drop-down, select
production
.In the Frequency field, enter
0 17 * * *
.In the Timezone drop-down, select a UTC+1 timezone, for example, Greenwich Mean Time (GMT).
Every day, at 5PM UTC+1, Dataform will execute the latest
production
compilation result of thequickstart-production
repository.To ensure that Dataform executes the latest compilation result in the corresponding release configuration, keep a minimum 1 hour break between the time of compilation result creation and the time of scheduled execution.
Click All actions.
Dataform will execute all SQL workflow actions in the
production
compilation result.Click Create.
The production
workflow configuration executes the entire latest
compilation result created by the production
release configuration
every day at 4PM UTC+1.
View past production
compilation results
To view past scheduled production
compilation results, follow these steps:
In the Google Cloud console, go to the Dataform page.
Click
quickstart-production
.Click Release configurations.
Click
production
.
View past production
workflow executions
To view past production
workflow executions, follow these steps:
In the Google Cloud console, go to the Dataform page.
Click
quickstart-production
.Click Workflow configurations.
Clean up
To avoid incurring charges to your Google Cloud account for the resources used on this page, follow these steps.
Delete the dataset created in BigQuery
To avoid incurring charges for BigQuery assets, follow these
steps to delete the dataset called dataform_production
:
In the Google Cloud console, go to the BigQuery page.
In the Explorer panel, expand your project and select
dataform_production
.Click the
Actions menu, and then select Delete.In the Delete dataset dialog, enter delete, and then click Delete.
Delete the Dataform release configuration
There are no costs associated with creating Dataform release
configurations. However, if you want to delete the production
release
configuration, follow these steps:
In the Google Cloud console, go to the Dataform page.
Click
quickstart-production
.Click Release configurations.
By the
production
release configuration, click the More menu, and then click Delete.In the Delete release configuration dialog, click Delete.
Delete the Dataform workflow configuration
To avoid incurring charges for BigQuery assets, follow these steps to delete the Dataform production
workflow configuration:
In the Google Cloud console, go to the Dataform page.
Click
quickstart-production
.Click Workflow configurations.
By the
production
workflow configuration, click the More menu, and then click Delete.In the Delete release configuration dialog, click Delete.
Delete the Dataform repository
There are no costs associated with creating Dataform repositories. However, if you want to delete a repository and all its contents, follow these steps:
In the Google Cloud console, go to the Dataform page.
By
quickstart-production
, click the More menu, and then select Delete.In the Delete repository window, enter the name of the repository to confirm deletion.
To confirm, click Delete.
What's next
To learn more about code lifecycle in Dataform, see Introduction to code lifecycle in Dataform.
To learn more about best practices for code lifecycle in Dataform, see Managing code lifecycle.
To learn more about release configurations in Dataform, see Create a release configuration.
To learn more about workflow configurations in Dataform, see Schedule executions with workflow configurations.