Create workspace compilation overrides

This document shows you how to create workspace compilation overrides to isolate tables and views created from your Dataform workspaces in BigQuery. You can use workspace compilation overrides to create isolated Dataform development environments.

About workspace compilation overrides

When you develop SQL workflow code in a Dataform workspace, Dataform compiles the code in your workspace in real-time to create a compilation result of the workspace. Dataform uses settings defined in the workflow settings file to create the workspace compilation result. Then, when you trigger execution in a workspace, Dataform executes the workspace compilation result in BigQuery.

To override the default settings set in workflow settings for all workspaces in your repository, you can create workspace compilation overrides.

With workspace compilation overrides, you can override the following settings for all workspaces in your repository:

Project
Google Cloud project in which Dataform executes the workspace compilation result, set in workflow_settings.yaml as defaultProject or in dataform.json as defaultDatabase.
Table prefix
Custom prefix added to all table names in all workspaces in the repository.
Schema suffix

Custom suffix appended to the schema of tables defined in

Set as defaultDataset in workflow_settings.yaml, defaultSchema in dataform.json, or in the schema parameter in the config block of a table.

To create isolated development environments, you can isolate workspaces with unique compilation overrides. You can dynamically modify the table prefix and schema suffix compilation overrides with the ${workspaceName} variable.

When you manually trigger execution in a workspace, the ${workspaceName} variable injects the name of the workspace into the workspace compilation overrides.

When you set ${workspaceName} as the table prefix, Dataform adds the name of the workspace to names of all tables in the workspace. After execution, in BigQuery, you can identify which workspace a table originates from.

When you set ${workspaceName} as the schema suffix, Dataform appends the name of the workspace to defaultSchema, creating a custom schema dedicated to the workspace. After execution, in BigQuery, you can locate all tables executed from a specific workspace in the dedicated schema.

Example of dynamic workspace compilation overrides

The following example shows dynamic workspace compilation overrides applied to a repository that contains workspaces named after developers who work on the repository: Sasha and Kai.

The goal of workspace compilation overrides in this example is to create isolated development environments for Sasha and Kai.

The following default settings are set in workflow_settings.yaml:

  • defaultProject : analytics
  • defaultDataset: dataform

The following workspace compilation overrides create a dynamic table prefix and schema suffix for each workspace in the repository:

  • Google Cloud Project ID : analytics_dev
  • Table prefix: ${workspaceName}
  • Schema suffix: ${workspaceName}

When Sasha manually triggers execution in the Sasha workspace, Dataform executes its content with the following settings:

  • Google Cloud project: analytics_dev
  • schema: dataform_sasha
  • table names: sasha_name, for example, sasha_orders.

When Kai manually triggers execution in the Kai workspace, Dataform executes its content with the following settings:

  • Google Cloud project: analytics_dev
  • schema: dataform_kai
  • tables names: kai_name, for example, kai_orders

Before you begin

  1. In the Google Cloud console, go to the Dataform page.

    Go to the Dataform page

  2. Select or create a repository.

  3. Optional: To override the default Google Cloud project, grant your Dataform service account access to the Google Cloud project you plan to use.

Required roles

To get the permissions that you need to create workspace compilation overrides, ask your administrator to grant you the Dataform Admin (roles/dataform.admin) IAM role on repositories. For more information about granting roles, see Manage access to projects, folders, and organizations.

You might also be able to get the required permissions through custom roles or other predefined roles.

Create workspace compilation overrides

To create Dataform workspace compilation overrides, follow these steps:

  1. In your repository, go to Settings.
  2. Click Edit.
  3. In the Workspace compilation overrides pane, in the Google Cloud Project ID field, enter the ID of the Google Cloud project.
  4. In the Table prefix field, enter a prefix for all table names.
    1. Optional: To create a dynamic table prefix that is unique for each workspace, enter ${workspaceName} as the table prefix.
  5. In the Schema suffix field, enter a suffix to append to the schema of the created table or view.
    1. Optional: To create a dynamic schema suffix that is unique for each workspace, enter ${workspaceName} as the table suffix.
  6. Click Save.

Dataform applies workspace compilation overrides to all workspaces in your repository.

Edit workspace compilation overrides

To edit Dataform workspace compilation overrides, follow these steps:

  1. In your repository, go to Settings.
  2. Click Edit.
  3. Edit workspace compilation overrides, and then click Save.

Delete workspace compilation overrides

To delete Dataform workspace compilation overrides, follow these steps:

  1. In your repository, go to Settings.
  2. Click Edit.
  3. In the Workspace compilation overrides pane, click Clear all, and then click Save.

What's next