Schedule production executions

Schedule production executions

This quickstart walks you through the following steps to schedule production executions in Dataform:

• Create a Dataform repository.
• Grant the required roles.

• Create a release configuration to configure the production environment.

  Create a production release configuration at set the frequency of creating production compilation results.

• Create a workflow configuration to schedule executions in the production environment.

  Create a production workflow configuration, select the production release configuration, and set a schedule for executing production compilation results.

• View past compilation results.

• View past executions.

• Clean up created assets.

Before you begin

Sign in to your Google Cloud account. If you're new to Google Cloud, create an account to evaluate how our products perform in real-world scenarios. New customers also get $300 in free credits to run, test, and deploy workloads.

In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

Roles required to select or create a project

• Select a project: Selecting a project doesn't require a specific IAM role—you can select any project that you've been granted a role on.
• Create a project: To create a project, you need the Project Creator (roles/resourcemanager.projectCreator), which contains the resourcemanager.projects.create permission. Learn how to grant roles.

Note: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project.

Go to project selector

Verify that billing is enabled for your Google Cloud project.

Enable the BigQuery and Dataform APIs.

Roles required to enable APIs

To enable APIs, you need the Service Usage Admin IAM role (roles/serviceusage.serviceUsageAdmin), which contains the serviceusage.services.enable permission. Learn how to grant roles.

Enable the APIs

In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

Roles required to select or create a project

• Select a project: Selecting a project doesn't require a specific IAM role—you can select any project that you've been granted a role on.
• Create a project: To create a project, you need the Project Creator (roles/resourcemanager.projectCreator), which contains the resourcemanager.projects.create permission. Learn how to grant roles.

Note: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project.

Go to project selector

Verify that billing is enabled for your Google Cloud project.

Enable the BigQuery and Dataform APIs.

Roles required to enable APIs

To enable APIs, you need the Service Usage Admin IAM role (roles/serviceusage.serviceUsageAdmin), which contains the serviceusage.services.enable permission. Learn how to grant roles.

Enable the APIs

Required roles

To get the permissions that you need to perform all tasks in this tutorial, ask your administrator to grant you the following IAM roles:

• Dataform Admin (roles/dataform.admin) on repositories
• Dataform Editor (roles/dataform.editor) on workspaces and workflowInvocations

For more information about granting roles, see Manage access to projects, folders, and organizations.

You might also be able to get the required permissions through custom roles or other predefined roles.

Create a Dataform repository

1. In the Google Cloud console, go to the Dataform page.

   Go to Dataform

2. Click Create repository.

3. On the Create repository page, do the following:

   1. In the Repository ID field, enter quickstart-production.

   2. In the Region list, select europe-west4.

   3. Click Create.

4. Click Done.

Grant required roles

To run workflows in BigQuery, the Dataform service agent or custom service account must have the following required roles:

• BigQuery Data Editor on projects to which Dataform needs both read and write access. They usually include the project hosting your Dataform repository.
• BigQuery Data Viewer on projects to which Dataform needs read-only access.
• BigQuery Job User on the project hosting your Dataform repository.

Additionally, you need to grant the Service Account Token Creator role (roles/iam.serviceAccountTokenCreator) and the Service Account User role (roles/iam.serviceAccountUser) to the default Dataform service agent on the custom service account resource.

To grant these roles, follow these steps:

1. In the Google Cloud console, go to the IAM page.

   Go to the IAM page

2. Click Add.

3. In the New principals field, enter your Dataform service agent or service account ID.

   Note: Service agent or service account credentials are the default option for scheduled runs. Using Google Account user account credentials (Preview) is discouraged for scheduled runs.

4. In the Select a role list, select the BigQuery Job User role.

5. Click Add another role, and then in the Select a role list, select the BigQuery Data Editor role.

6. Click Add another role, and then in the Select a role list, select the BigQuery Data Viewer role.

7. If you're adding roles to the default Dataform service agent, click Add another role, and then in the Select a role list, select the Service Account Token Creator role.

8. Click Save.

Create a release configuration

To create production compilation results of the quickstart-production repository, follow these steps:

1. In the Google Cloud console, go to the Dataform page.

   Go to Dataform

2. Click quickstart-production.

3. Click Release configurations, and then click New release configuration.

4. In the Create release configuration pane, configure the following settings:

   1. In the Release ID field, enter production.

   2. In the Git commitish field, leave the default value main .

   3. In the Timezone menu, select a UTC+1 timezone, for example, Greenwich Mean Time (GMT).

   Every day at 4 PM UTC+1, Dataform will compile the quickstart-production repository and apply compilation settings configured in this release configuration to create production compilation results.

5. Click Create.

The production release configuration creates a compilation result of the entire quickstart-production repository every day at 4PM UTC+1.

Create a workflow configuration

To schedule execution of production tables, follow these steps:

1. In the Google Cloud console, go to the Dataform page.

   Go to Dataform

2. Click quickstart-production.

3. Click Workflow configurations, and then click New workflow configuration.

4. In the Create workflow configuration pane, in the Configuration ID field, enter production.

5. In the Release configuration menu, select production.

6. In the Frequency field, enter 0 17 * * *.

7. In the Timezone menu, select a UTC+1 timezone, for example, Greenwich Mean Time (GMT).

   Every day at 5PM UTC+1, Dataform runs the latest production compilation result of the quickstart-production repository.

   To ensure that Dataform runs the latest compilation result in the corresponding release configuration, keep a minimum 1 hour break between the time of compilation result creation and the time of scheduled execution.

8. Click All actions.

   Dataform run all the workflow actions in the production compilation result.

9. Click Create.

The production workflow configuration runs the entire latest compilation result created by the production release configuration every day at 4PM UTC+1.

View past production compilation results

To view past scheduled production compilation results, follow these steps:

1. In the Google Cloud console, go to the Dataform page.

   Go to Dataform

2. Click quickstart-production.

3. Click Release configurations.

4. Click production.

View past production workflow executions

To view past production workflow executions, follow these steps:

1. In the Google Cloud console, go to the Dataform page.

   Go to Dataform

2. Click quickstart-production.

3. Click Workflow configurations.

Clean up

To avoid incurring charges to your Google Cloud account for the resources used on this page, follow these steps.

Delete the dataset created in BigQuery

To avoid incurring charges for BigQuery assets, follow these steps to delete the dataset called dataform_production:

1. In the Google Cloud console, go to the BigQuery page.

   Go to BigQuery

2. In the Explorer panel, expand your project and select dataform_production.

3. Click the Actions menu, and then select Delete.

4. In the Delete dataset dialog, enter delete, and then click Delete.

Delete the Dataform release configuration

There are no costs associated with creating Dataform release configurations. However, if you want to delete the production release configuration, follow these steps:

1. In the Google Cloud console, go to the Dataform page.

   Go to Dataform

2. Click quickstart-production.

3. Click Release configurations.

4. By the production release configuration, click the More menu, and then click Delete.

5. In the Delete release configuration dialog, click Delete.

Delete the Dataform workflow configuration

To avoid incurring charges for BigQuery assets, follow these steps to delete the Dataform production workflow configuration:

1. In the Google Cloud console, go to the Dataform page.

   Go to Dataform

2. Click quickstart-production.

3. Click Workflow configurations.

4. By the production workflow configuration, click the More menu, and then click Delete.

5. In the Delete release configuration dialog, click Delete.

Delete the Dataform repository

There are no costs associated with creating Dataform repositories. However, if you want to delete a repository and all its contents, follow these steps:

1. In the Google Cloud console, go to the Dataform page.

   Go to Dataform

2. By quickstart-production, click the More menu, and then select Delete.

3. In the Delete repository window, enter the name of the repository to confirm deletion.

4. To confirm, click Delete.

What's next

• To learn more about code lifecycle in Dataform, see Introduction to code lifecycle in Dataform.

• To learn more about best practices for the workflow lifecycle in Dataform, see Best practices for the workflow lifecycle.

• To learn more about release configurations in Dataform, see Create a release configuration.

• To learn more about workflow configurations in Dataform, see Schedule runs with workflow configurations.