This document shows you how to create workspace compilation overrides to isolate tables and views created from your Dataform workspaces in BigQuery. You can use workspace compilation overrides to create isolated Dataform development environments.
About workspace compilation overrides
When you develop SQL workflow code in a Dataform workspace, Dataform compiles the code in your workspace in real-time to create a compilation result of the workspace. Dataform uses settings defined in the workflow settings file to create the workspace compilation result. Then, when you trigger execution in a workspace, Dataform executes the workspace compilation result in BigQuery.
To override the default settings set in workflow settings for all workspaces in your repository, you can create workspace compilation overrides.
With workspace compilation overrides, you can override the following settings for all workspaces in your repository:
- Project
- Google Cloud project in which Dataform executes the workspace
compilation result, set in
workflow_settings.yaml
asdefaultProject
or indataform.json
asdefaultDatabase
. - Table prefix
- Custom prefix added to all table names in all workspaces in the repository.
- Schema suffix
Custom suffix appended to the schema of tables defined in
Set as
defaultDataset
inworkflow_settings.yaml
,defaultSchema
indataform.json
, or in theschema
parameter in theconfig
block of a table.
To create isolated development environments, you can isolate workspaces with
unique compilation overrides. You can dynamically modify the table prefix and
schema suffix compilation overrides with the ${workspaceName}
variable.
When you manually trigger execution in a workspace, the ${workspaceName}
variable injects the name of the workspace into the workspace compilation
overrides.
When you set ${workspaceName}
as the table prefix, Dataform adds the
name of the workspace to names of all tables in the workspace. After execution,
in BigQuery, you can identify which workspace a table originates from.
When you set ${workspaceName}
as the schema suffix, Dataform appends
the name of the workspace to defaultSchema
, creating a custom schema dedicated
to the workspace. After execution, in BigQuery, you can locate all
tables executed from a specific workspace in the dedicated schema.
Example of dynamic workspace compilation overrides
The following example shows dynamic workspace compilation overrides applied to a
repository that contains workspaces named after developers who work on the
repository: Sasha
and Kai
.
The goal of workspace compilation overrides in this example is to create isolated development environments for Sasha and Kai.
The following default settings are set in workflow_settings.yaml
:
defaultProject
:analytics
defaultDataset
:dataform
The following workspace compilation overrides create a dynamic table prefix and schema suffix for each workspace in the repository:
- Google Cloud Project ID :
analytics_dev
- Table prefix:
${workspaceName}
- Schema suffix:
${workspaceName}
When Sasha manually triggers execution in the Sasha
workspace,
Dataform executes its content with the following settings:
- Google Cloud project:
analytics_dev
- schema:
dataform_sasha
- table names:
sasha_name
, for example,sasha_orders
.
When Kai manually triggers execution in the Kai
workspace, Dataform
executes its content with the following settings:
- Google Cloud project:
analytics_dev
- schema:
dataform_kai
- tables names:
kai_name
, for example,kai_orders
Before you begin
In the Google Cloud console, go to the Dataform page.
Select or create a repository.
Optional: To override the default Google Cloud project, grant your Dataform service account access to the Google Cloud project you plan to use.
Required roles
To get the permissions that you need to create workspace compilation overrides,
ask your administrator to grant you the
Dataform Admin (roles/dataform.admin
) IAM role on repositories.
For more information about granting roles, see Manage access to projects, folders, and organizations.
You might also be able to get the required permissions through custom roles or other predefined roles.
Create workspace compilation overrides
To create Dataform workspace compilation overrides, follow these steps:
- In your repository, go to Settings.
- Click Edit.
- In the Workspace compilation overrides pane, in the Google Cloud Project ID field, enter the ID of the Google Cloud project.
- In the Table prefix field, enter a prefix for all table names.
- Optional: To create a dynamic table prefix that is unique for each
workspace, enter
${workspaceName}
as the table prefix.
- Optional: To create a dynamic table prefix that is unique for each
workspace, enter
- In the Schema suffix field, enter a suffix to append to the schema of
the created table or view.
- Optional: To create a dynamic schema suffix that is unique for each
workspace, enter
${workspaceName}
as the table suffix.
- Optional: To create a dynamic schema suffix that is unique for each
workspace, enter
- Click Save.
Dataform applies workspace compilation overrides to all workspaces in your repository.
Edit workspace compilation overrides
To edit Dataform workspace compilation overrides, follow these steps:
- In your repository, go to Settings.
- Click Edit.
- Edit workspace compilation overrides, and then click Save.
Delete workspace compilation overrides
To delete Dataform workspace compilation overrides, follow these steps:
- In your repository, go to Settings.
- Click Edit.
- In the Workspace compilation overrides pane, click Clear all, and then click Save.
What's next
- To learn how to manually trigger execution, see Trigger execution.