Manage the Dataform core package

This document shows you how to manage the Dataform core framework dependency package and update it to the latest version.

Dataform core is the open-source Dataform framework for developing SQL workflows with SQL, SQLX, and JavaScript. As a best practice, always use the latest available version of the Dataform core framework. For information about releases of the Dataform core framework, see Dataform releases on GitHub.

Before you begin

  1. Create a repository.
  2. Create and initialize a development workspace.

Required roles

To get the permissions that you need to manage and update Dataform core, ask your administrator to grant you the following IAM roles:

  • Manage the location of the Dataform core package: Dataform Admin (roles/dataform.admin) on repositories
  • Update the Dataform core package: Dataform Editor (roles/dataform.editor) on workspaces

For more information about granting roles, see Manage access to projects, folders, and organizations.

You might also be able to get the required permissions through custom roles or other predefined roles.

Manage Dataform core package location

When you initialize the first workspace in a repository, Dataform automatically sets Dataform core as a dependency package. As of Dataform core 3.0.0, Dataform installs the Dataform core package in the workflow_settings.yaml file by default. In earlier versions of Dataform core, Dataform core was set in the package.json file.

In Dataform core 3.0.0 and later, if Dataform core is the only package in your repository, then you should set it in the workflow_settings.yaml file. For repositories created with earlier versions of Dataform core, move the Dataform core package to workflow_settings.yaml.

The package.json file is required to install additional packages in Dataform. If your repository uses additional packages, set the Dataform core package in package.json so all packages are set in one place. If your repository doesn't have package.json, create package.json and move the Dataform core package to install additional packages.

Move Dataform core to workflow_settings.yaml

For repositories created with versions of Dataform core earlier than 3.0.0, if you don't have dependency packages other than Dataform core, then you should move the Dataform core package from package.json to workflow_settings.yaml, and delete the redundant package.json file.

To migrate the Dataform core package from package.json to workflow_settings.yaml, do the following:

  1. In the Google Cloud console, go to the Dataform page.

    Go to the Dataform page

  2. Select a repository, and then select a workspace.

  3. In the Files pane, select the workflow_settings.yaml file.

  4. In the workflow_settings.yaml file, add the Dataform core package in the following format:

    dataformCoreVersion: "VERSION"

    Replace VERSION with the latest version of Dataform, for example, 3.0.0.

  5. In the Files pane, click the More menu next to package.json, and then click Delete.

  6. To confirm deletion of dataform.json, click Delete.

  7. Click Install packages.

Move Dataform core to package.json

The package.json file is required to install additional packages in a repository. If your repository uses additional packages, then you should store all packages, including the Dataform core package, in package.json.

If your repository doesn't contain the package.json file because the Dataform core package is set in the workflow_settings.yaml file, you must create the package.json file to install additional packages, and then move the Dataform core package from workflow_settings.yaml to the newly created package.json.

To create the package.json file and move the Dataform core package, follow these steps:

  1. In the Google Cloud console, go to the Dataform page.

    Go to the Dataform page

  2. Select a repository, and then select a workspace.

  3. In the Files pane, click addAdd, and then click Create file.

  4. In the Add a file path field, enter package.json.

  5. Click Create file.

  6. In the package.json file, add the Dataform core package in the following format:

    {
    "dependencies": {
        "@dataform/core": "VERSION"
    }
}

    Replace VERSION with the latest version of Dataform, for example, 3.0.0.

  7. Click Install packages.

  8. In the Files pane, select workflow_settings.yaml.

  9. In the workflow_settings.yaml file, delete the dataformCoreVersion property.

Update Dataform core

Always test new package versions in a non-production environment before deploying in your production environment.

To update the Dataform core dependency package, follow these steps:

  1. Look up the latest version of @dataform/core on the Dataform releases page on GitHub.

  2. In the Google Cloud console, go to the Dataform page.

    Go to the Dataform page

  3. Select a repository, and then select a workspace.

  4. In the Files pane, select the package.json file or the workflow_settings.yaml file.

    The location where the Dataform core dependency package is set depends on your version of Dataform core and your use of packages. For more information, see Manage Dataform core package location.

  5. Update the Dataform core dependency package with the latest version:

    package.json

    {
    "dependencies": {
        "@dataform/core": "VERSION"
    }
}

    Replace VERSION with the latest version of Dataform, for example, 3.0.0. To prevent issues with package installation, explicitly specify the Dataform core package version. Don't use other dependencies options of package.json, for example, >version.

    1. Click Install packages.

    workflow_settings.yaml

    dataformCoreVersion: "VERSION"

    Replace VERSION with the latest version of Dataform, for example, 3.0.0.

    1. Click Install packages.

  6. Commit changes.

  7. Push changes to your repository.

The following code sample shows the @dataform/core dependency updated to the 3.0.0 version in the package.json file:

{
    "dependencies": {
        "@dataform/core": "3.0.0"
    }
}

What's next