This document shows you how to create workspace compilation overrides to isolate tables and views created from your Dataform workspaces in BigQuery. You can use workspace compilation overrides to create isolated Dataform development environments.
About workspace compilation overrides
When you develop SQL workflow code in a Dataform workspace,
Dataform compiles the code in your workspace in real-time to create a
compilation result
of the workspace. Dataform uses settings defined in the dataform.json
file to create the workspace compilation result. Then, when you
trigger execution in a workspace,
Dataform executes the workspace compilation result in BigQuery.
To override the default settings set in dataform.json
for all workspaces in
your repository, you can create workspace compilation overrides.
With workspace compilation overrides, you can override the following settings for all workspaces in your repository:
- Project
- Google Cloud project in which Dataform executes the workspace
compilation result, set in
dataform.json
asdefaultDatabase
. - Table prefix
- Custom prefix added to all table names in all workspaces in the repository.
- Schema suffix
- Custom suffix appended to the schema of tables defined in
defaultSchema
indataform.json
or in theschema
parameter in theconfig
block of a table.
To create isolated development environments, you can isolate workspaces
with unique compilation overrides. You can dynamically modify the table prefix
and schema suffix compilation overrides with the ${workspaceName}
variable.
When you manually trigger execution in a workspace, the ${workspaceName}
variable injects the name of the workspace into the workspace compilation
overrides.
When you set ${workspaceName}
as the table prefix,
Dataform adds the name of the workspace to names of all tables
in the workspace. After execution, in BigQuery, you can identify which
workspace a table originates from.
When you set ${workspaceName}
as the schema suffix,
Dataform appends the name of the workspace to defaultSchema
,
creating a custom schema dedicated to the workspace. After execution, in
BigQuery, you can locate all tables executed from a specific workspace
in the dedicated schema.
Example of dynamic workspace compilation overrides
The following example shows dynamic workspace compilation overrides applied to a
repository that contains workspaces named after developers who work on the
repository: Sasha
and Kai
.
The goal of workspace compilation overrides in this example is to create isolated development environments for Sasha and Kai.
The following default settings are set in dataform.json
:
defaultDatabase
:analytics
defaultSchema
:dataform
The following workspace compilation overrides create a dynamic table prefix and schema suffix for each workspace in the repository:
- Google Cloud Project ID :
analytics_dev
- Table prefix:
${workspaceName}
- Schema suffix:
${workspaceName}
When Sasha manually triggers execution in the Sasha
workspace,
Dataform executes its content with the following settings:
- Google Cloud project:
analytics_dev
- schema:
dataform_sasha
- table names:
sasha_name
, for example,sasha_orders
.
When Kai manually triggers execution in the Kai
workspace,
Dataform executes its content with the following settings:
- Google Cloud project:
analytics_dev
- schema:
dataform_kai
- tables names:
kai_name
, for example,kai_orders
Before you begin
In the Google Cloud console, go to the Dataform page.
Select or create a repository.
Optional: To override the default Google Cloud project, grant your Dataform service account access to the Google Cloud project you plan to use.
Required roles
To get the permissions that you need to create workspace compilation overrides,
ask your administrator to grant you the
Dataform Admin (roles/dataform.admin
) IAM role on repositories.
For more information about granting roles, see
Manage access.
You might also be able to get the required permissions through custom roles or other predefined roles.
Create workspace compilation overrides
To create Dataform workspace compilation overrides, follow these steps:
- In your repository, go to Settings.
- Click Edit.
- In the Workspace compilation overrides pane, in the Google Cloud Project ID field, enter the ID of the Google Cloud project.
- In the Table prefix field, enter a prefix for all table names.
- Optional: To create a dynamic table prefix that is unique for each
workspace, enter
${workspaceName}
as the table prefix.
- Optional: To create a dynamic table prefix that is unique for each
workspace, enter
- In the Schema suffix field, enter a suffix to append to the
schema of the created table or view.
- Optional: To create a dynamic schema suffix that is unique for each
workspace, enter
${workspaceName}
as the table suffix.
- Optional: To create a dynamic schema suffix that is unique for each
workspace, enter
- Click Save.
Dataform applies workspace compilation overrides to all workspaces in your repository.
Edit workspace compilation overrides
To edit Dataform workspace compilation overrides, follow these steps:
- In your repository, go to Settings.
- Click Edit.
- Edit workspace compilation overrides, and then click Save.
Delete workspace compilation overrides
To delete Dataform workspace compilation overrides, follow these steps:
- In your repository, go to Settings.
- Click Edit.
- In the Workspace compilation overrides pane, click Clear all, and then click Save.
What's next
- To learn how to manually trigger execution, see Trigger execution.