Create development environments

Create development environments

This quickstart walks you through the steps to create individual development environments in Dataform, so that each developer can work in their own development environment:

• Create a Dataform repository.

• Grant Dataform access to BigQuery.

  • Configure workspace compilation overrides.

  Configure compilation overrides that Dataform will apply to all workspaces in your repository, turning workspaces into isolated development environments.

• Clean up created assets.

Before you begin

1. Sign in to your Google Cloud account. If you're new to Google Cloud, create an account to evaluate how our products perform in real-world scenarios. New customers also get $300 in free credits to run, test, and deploy workloads.

2. In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

   Note: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project.

   Go to project selector

3. Make sure that billing is enabled for your Google Cloud project.

4. Enable the BigQuery and Dataform APIs.

   Enable the APIs

5. In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

   Note: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project.

   Go to project selector

6. Make sure that billing is enabled for your Google Cloud project.

7. Enable the BigQuery and Dataform APIs.

   Enable the APIs

Required roles

To get the permissions that you need to perform all tasks in this tutorial, ask your administrator to grant you the following IAM roles on your project:

• Dataform Admin (roles/dataform.admin) - repositories
• Dataform Editor (roles/dataform.editor) - workspaces and workflow invocations

For more information about granting roles, see Manage access to projects, folders, and organizations.

You might also be able to get the required permissions through custom roles or other predefined roles.

Create a Dataform repository

1. In the Google Cloud console, go to the Dataform page.

   Go to Dataform

2. Click Create repository.

3. On the Create repository page, do the following:

   1. In the Repository ID field, enter quickstart-dev.

   2. In the Region list, select europe-west4.

   3. Click Create.

4. Click Done.

Grant Dataform access to BigQuery

To execute workflows in BigQuery, the Dataform service account must have the following required roles:

• BigQuery Data Editor on projects to which Dataform needs both read and write access. They usually include the project hosting your Dataform repository.
• BigQuery Data Viewer on projects to which Dataform needs read-only access.
• BigQuery Job User on the project hosting your Dataform repository.

To grant these roles, follow these steps:

1. In the Google Cloud console, go to the IAM page.

   Go to the IAM page

2. Click Add.

3. In the New principals field, enter your Dataform service account ID.

4. In the Select a role drop-down list, select the BigQuery Job User role.

5. Click Add another role, and then in the Select a role drop-down list, select the BigQuery Data Editor role.

6. Click Add another role, and then in the Select a role drop-down list, select the BigQuery Data Viewer role.

7. Click Save.

Configure workspace compilation overrides

To create isolated development environments with Dataform workspace compilation overrides, follow these steps:

1. In the Google Cloud console, go to the Dataform page.

   Go to Dataform

2. Click quickstart-dev.

3. Click Settings, and then click Edit.

4. In the Workspace compilation overrides pane, in the Schema suffix field, enter ${workspaceName}.

   When you trigger execution in a workspace, Dataform will append the name of the workspace as a suffix to the default schema set in workflow settings. This quickstart uses the default Dataform settings with the dataform default schema.

5. Click Save.

Dataform applies workspace compilation overrides to all workspaces in your repository.

With this configuration, when you manually execute a SQL workflow in a workspace, Dataform executes it with workspace compilation overrides. As a result, Dataform executes the SQL workflow in the dataform_workspaceName schema in BigQuery, turning the workspace into an isolated development environment.

For example, when you manually trigger execution in a sasha workspace, Dataform executes your SQL workflow in the dataform_sasha schema in BigQuery.

To execute SQL workflows without workspace compilation overrides, but keep workspace compilation overrides for development environments, you can create release configurations to configure other execution environments, for example, staging and production.

Clean up

To avoid incurring charges to your Google Cloud account for the resources used on this page, follow these steps.

Delete datasets created in BigQuery

To avoid incurring charges for BigQuery assets, delete datasets with names that begin in dataform, for example, dataform-sasha.

1. In the Google Cloud console, go to the BigQuery page.

   Go to BigQuery

2. In the Explorer panel, expand your project and select a dataset.

3. Click the Actions menu, and then select Delete.

4. In the Delete dataset dialog, enter delete into the field, and then click Delete.

Delete the Dataform repository

There are no costs associated with creating Dataform repositories. However, if you want to delete a repository and all its contents, follow these steps:

1. In the Google Cloud console, go to the Dataform page.

   Go to Dataform

2. By quickstart-dev, click the More menu, and then select Delete.

   1. In the Delete repository window, enter the name of the repository to confirm deletion.

3. To confirm, click Delete.

What's next

• To learn more about code lifecycle in Dataform, see Introduction to code lifecycle in Dataform.

• To learn more about best practices for code lifecycle in Dataform, see Managing code lifecycle.

• To learn more about workspace compilation overrides in Dataform, see Create workspace compilation overrides.

• To learn how to schedule production executions in Dataform, see Schedule production executions.