Create a release configuration

This document shows you how to create a release configuration in Dataform to configure templates of compilation settings, which you can execute on a schedule. You can use release configurations to configure execution environments, for example, staging and production.

About Dataform release configurations

To execute SQL workflows in BigQuery, Dataform compiles SQL workflow code into a compilation result. This process happens automatically when you develop a SQL workflow in a Dataform workspace.

Release configurations let you customize compilation results. You can use them to create execution environments, for example, staging.

Release configuration settings

In a release configuration, you can configure compilation overrides of workflow_settings.yaml settings, set compilation variables, and set the frequency of creating compilation results.

A Dataform release configuration contains the following compilation settings:

Release settings
ID of the release, git commitish for compilation results, and frequency of creating compilation results. Frequency is an optional setting. If set, the minimum frequency is 1 hour.
Compilation overrides
Overrides of the Google Cloud project, table prefix, schema suffix, and compilation variables defined in workflow_settings.yaml.

How a release configuration works

Dataform creates compilation results from a release configuration at the specified frequency, or when you trigger compilation. Frequency is an optional setting and it is not required to create a release configuration. If set, the minimum frequency is 1 hour. You can manually trigger compilation on the Release configuration details page, or trigger compilation with the Dataform API releaseConfigs.

During compilation, Dataform pulls code from the specified Git commitish of your repository. Then, Dataform compiles the code with the applied compilation overrides (if there are any), and creates the compilation result. The latest compilation result created for the release configuration is the live compilation result.

You can schedule executions of compilation results from release configurations in workflow configurations. You can also execute a selected release configuration outside of any schedule. During a workflow configuration execution, Dataform executes the live compilation result from the selected release configuration.

Before you begin

  1. In the Google Cloud console, go to the Dataform page.

    Go to the Dataform page

  2. Select or create a repository.

  3. Optional: To override the default Google Cloud project in the release configuration, grant your Dataform service account access to the Google Cloud project you plan to use.

Required roles

To get the permissions that you need to create a release configuration, ask your administrator to grant you the Dataform Editor (roles/dataform.editor) IAM role on repositories. For more information about granting roles, see Manage access.

You might also be able to get the required permissions through custom roles or other predefined roles.

Create a release configuration

To create a Dataform release configuration, follow these steps:

  1. In your repository, go to Releases & Scheduling.
  2. In the Release configurations section, click Create.

  3. In the Create release configuration pane, configure release settings.

    1. In the Release ID field, enter a unique ID for the release configuration.

      IDs can only include numbers, letters, hyphens, and underscores.

    2. In the Git commitish field, enter the Git branch or Git commit SHA for the release.

    In a Dataform repository not connected to a remote repository, the value is always main.

    1. Optional: In the Frequency drop-down, select the frequency of creating compilation results.

      If set, the minimum frequency is 1 hour.

  4. Optional: In the Compilation overrides section, configure compilation settings.

    1. In the Google Cloud Project ID field, enter the ID of the Google Cloud project in which you want to store the compilation result.
    2. In the Schema suffix field, enter a suffix to append to the schema configured in workflow_settings.yaml.
    3. In the Table prefix field, enter a prefix for all table names.

  5. Optional: In the Compilation variables section, set compilation variables.

    1. Click Add variable.
    2. In the Key field, enter the compilation variable.
    3. In the Value field, enter the value for the compilation variable.
    4. To add another compilation variable, click Add variable.

  6. Click Create.

If you override the default Google Cloud project, make sure your Dataform service account has access to the Google Cloud project set in the release configuration.



For example, the following release configuration creates a production compilation result from the main branch every hour, without any compilation overrides:

  • Release ID: production
  • Git commitish: main
  • Frequency: hourly
  • no compilation overrides

View details of a release configuration

You can view the following details of a release configuration:

  • Release settings
    • Git commitish
    • Last compilation result timestamp
    • Cron schedule
    • Compilation overrides
    • Compilation variables
  • Live compilation result
    • Creation timestamp
    • Git commitish
    • Commit SHA
  • Past scheduled compilation results

To view details of a release configuration, follow these steps:

  1. In your repository, go to Releases & Scheduling.
  2. Select a release configuration.
  3. On the Release configuration details page, inspect details of the release configuration.

The Scheduled compilation results table displays past compilation results created automatically by the release configuration. Compilation results created manually or through calls to the Dataform API are not displayed in the table.

Manually trigger compilation

To manually create a compilation result from a selected release configuration, follow these steps:

  1. In your repository, go to Releases & Scheduling.
  2. Select a release configuration.
  3. On the Release configuration details page, click New compilation.

The newly created compilation result becomes the live result for that release configuration.

Manual compilation results are not displayed in the Scheduled compilation results table on the Release configuration details page.

Trigger execution of a release configuration

To trigger execution of the live compilation result in a selected release configuration to BigQuery, follow these steps:

  1. In your repository, go to Releases & Scheduling.
  2. In the Release configurations section, click Start execution.
  3. In the Execute manual workflow pane, in the Release configuration drop-down, select a release configuration.
  4. Select SQL workflow actions to be executed:
    1. To execute the entire SQL workflow, click All actions.
    2. To execute selected actions in the SQL workflow, click Selection of actions, and then select actions.
    3. To execute actions with selected tags, click Selection of tags, and then select tags.
  5. Optional: To execute selected actions or tags and their dependencies, select the Include dependencies option.
  6. Optional: To execute selected actions or tags and their dependents, select the Include dependents option.

  7. Optional: To rebuild all tables from scratch, select the Run with full refresh option.

    Without this option, Dataform updates incremental tables without rebuilding them from scratch.

  8. Click Start execution.

Edit a release configuration

To edit a release configuration, follow these steps:

  1. In your repository, go to Releases & Scheduling.
  2. By the release configuration that you want to edit, click the More menu, and then click Edit.
  3. In the Edit release configuration pane, edit release configuration settings, and then click Save.

Delete a release configuration

To delete a release configuration, follow these steps:

  1. In your repository, go to Releases & Scheduling.
  2. By the release configuration that you want to delete, click the More menu, and then click Delete.
  3. In the Delete release configuration dialog, click Delete.

What's next