Dataform core is the open-source Dataform framework for developing SQL workflows with SQL, SQLX, and JavaScript. As a best practice, always use the latest available version of the Dataform core framework. For information about releases of the Dataform core framework, see Dataform releases on GitHub.
Before you begin
Required roles
To get the permissions that you need to manage and update Dataform core, ask your administrator to grant you the following IAM roles:
-
Manage the location of the Dataform core package:
Dataform Admin (
roles/dataform.admin
) on repositories -
Update the Dataform core package:
Dataform Editor (
roles/dataform.editor
) on workspaces
For more information about granting roles, see Manage access to projects, folders, and organizations.
You might also be able to get the required permissions through custom roles or other predefined roles.
Manage Dataform core package location
When you initialize the first workspace in a repository, Dataform
automatically sets Dataform core as a dependency package.
As of Dataform core 3.0.0
,
Dataform installs the Dataform core package in the
workflow_settings.yaml
file by default. In earlier versions of
Dataform core, Dataform core was set in
the package.json
file.
In Dataform core 3.0.0
and later, if Dataform core is the
only package in your repository, then you should set it in
the workflow_settings.yaml
file. For repositories created with earlier
versions of Dataform core,
move the Dataform core package to workflow_settings.yaml
.
The package.json
file is
required to install additional packages in Dataform.
If your repository uses additional packages, set the
Dataform core package in package.json
so all packages
are set in one place. If your repository doesn't have package.json
,
create package.json
and move the Dataform core package
to install additional packages.
Move Dataform core to workflow_settings.yaml
For repositories created with versions of Dataform core earlier than
3.0.0
, if you don't have dependency packages other than Dataform core,
then you should move the Dataform core package
from package.json
to workflow_settings.yaml
, and delete the redundant
package.json
file.
To migrate the Dataform core package from package.json
to
workflow_settings.yaml
, do the following:
In the Google Cloud console, go to the Dataform page.
Select a repository, and then select a workspace.
In the Files pane, select the
workflow_settings.yaml
file.In the
workflow_settings.yaml
file, add the Dataform core package in the following format:dataformCoreVersion: "VERSION"
Replace VERSION with the latest version of Dataform, for example,
3.0.0
.In the Files pane, click the More menu next to
package.json
, and then click Delete.To confirm deletion of
dataform.json
, click Delete.Click Install packages.
Move Dataform core to package.json
The package.json
file is required to
install additional packages in a repository.
If your repository uses additional packages, then you should store all packages,
including the Dataform core package, in package.json
.
If your repository doesn't contain the package.json
file because the
Dataform core package is set in the
workflow_settings.yaml
file, you must create the package.json
file to install additional packages, and then move the Dataform core
package from workflow_settings.yaml
to the newly created package.json
.
To create the package.json
file and move the
Dataform core package, follow these steps:
In the Google Cloud console, go to the Dataform page.
Select a repository, and then select a workspace.
In the Files pane, click addAdd, and then click Create file.
In the Add a file path field, enter
package.json
.Click Create file.
In the
package.json
file, add the Dataform core package in the following format:{ "dependencies": { "@dataform/core": "VERSION" } }
Replace VERSION with the latest version of Dataform, for example,
3.0.0
.Click Install packages.
In the Files pane, select
workflow_settings.yaml
.In the
workflow_settings.yaml
file, delete thedataformCoreVersion
property.
Update Dataform core
Always test new package versions in a non-production environment before deploying in your production environment.
To update the Dataform core dependency package, follow these steps:
Look up the latest version of
@dataform/core
on the Dataform releases page on GitHub.In the Google Cloud console, go to the Dataform page.
Select a repository, and then select a workspace.
In the Files pane, select the
package.json
file or theworkflow_settings.yaml
file.The location where the Dataform core dependency package is set depends on your version of Dataform core and your use of packages. For more information, see Manage Dataform core package location.
Update the Dataform core dependency package with the latest version:
package.json
{ "dependencies": { "@dataform/core": "VERSION" } }
Replace VERSION with the latest version of Dataform, for example,
3.0.0
. To prevent issues with package installation, explicitly specify the Dataform core package version. Don't use otherdependencies
options ofpackage.json
, for example,>version
.- Click Install packages.
workflow_settings.yaml
dataformCoreVersion: "VERSION"
Replace VERSION with the latest version of Dataform, for example,
3.0.0
.- Click Install packages.
The following code sample shows the @dataform/core
dependency updated to
the 3.0.0
version in the package.json
file:
{
"dependencies": {
"@dataform/core": "3.0.0"
}
}
What's next
- To learn more about the
workflow_settings.yaml
file, see Configure Dataform settings. - To learn how to install additional packages, see Install a package in Dataform.