To use a package in Dataform, you need to install it in your repository.
You can install the following types of packages in Dataform:
- Published public NPM packages.
- Non-published public NPM packages.
- Authenticated private NPM packages.
Then, to use the package in a JavaScript or SQLX file, you need to import selected contents of the package to the file. You can also import a whole package to a JavaScript or SQLX file instead of its selected contents.
Before you begin
In the Google Cloud console, go to the Dataform page.
Select or create a repository.
Select or create a development workspace.
[Optional] To install a private package, authenticate the private package.
Required roles
To get the permissions that you need to import a package,
ask your administrator to grant you the
Dataform Editor (roles/dataform.editor
) IAM role on workspaces.
For more information about granting roles, see Manage access.
You might also be able to get the required permissions through custom roles or other predefined roles.
Install a package in a Dataform repository
To install a package inside a Dataform repository, you need to
add it as a dependency in the package.json
file. The format of the dependency
definition in the package.json
file depends on the type of the package. If you
don't have a package.json
file because your workflow_settings.yaml
file
specifies your dataformCoreVersion
, remove the dataformCoreVersion
from
workflow_settings.yaml
, and then create a package.json
file and add
@dataform/core
as a dependency.
To install a package in a Dataform repository, follow these steps:
- In your workspace, in the Files pane, select
package.json
. Add the package to the
dependencies
block:Add a published public NPM package in the following format:
"PACKAGE-NAME": "PACKAGE-VERSION"
Replace the following:
- PACKAGE-NAME with the name of the package.
- PACKAGE-VERSION with the latest version of the published public NPM package.
Add a non-published public NPM package in the following format:
"PACKAGE-NAME": "PACKAGE-URL"
Replace the following:
- PACKAGE-NAME with the name of the package.
- PACKAGE-URL with the
tar.gz
URL of the third-party package repository, for examplehttps://github.com/user/sample-package-repository/archive/master.tar.gz
.
Add an authenticated private NPM package in the following format:
"REGISTRY-SCOPE/PACKAGE-NAME": "PACKAGE-URL"
Replace the following:
- REGISTRY-SCOPE with the name of the package.
REGISTRY-SCOPE must match the registry scope
defined in the
.nmprc
file in your repository. - PACKAGE-NAME with the name of the package.
- PACKAGE-URL with the
tar.gz
URL of the package repository, for examplehttps://github.com/user/sample-package-repository/archive/master.tar.gz
.
- REGISTRY-SCOPE with the name of the package.
REGISTRY-SCOPE must match the registry scope
defined in the
Click Install packages.
The following code sample shows the public open-source
Slowly changing dimensions package package added to the .package.json
file:
```
{
"name": "repository-name",
"dependencies": {
"@dataform/core": "2.0.3",
"dataform-scd": "https://github.com/dataform-co/dataform-scd/archive/0.3.tar.gz"
}
}
```
Import a package function or constant to a JavaScript file in Dataform
To use a function or a constant from a package inside a JavaScript file in Dataform, you need to first import it to the file.
To import a function or a constant from a package to a JavaScript file, follow these steps:
- In your workspace, in the Files pane, select a
.js
file in which you want to use the package. In the file, import a function or a constant in the following format:
const { EXPORT-NAME } = require("PACKAGE-NAME");
- Replace EXPORT-NAME with the name of the function or
constant that you want to use, declared in
module.exports
in the packageindex.js
file. - Replace PACKAGE-NAME with the name of the package that you want to use.
- Replace EXPORT-NAME with the name of the function or
constant that you want to use, declared in
The following code sample shows the getDomain
function from the
postoffice
package imported and used in a JavaScript file:
/*
* Contents of postoffice index.js:
* module.exports = { getDomain };
*/
const { getDomain } = require("postoffice");
getDomain();
Import a whole package to a JavaScript file in Dataform
To import the whole package to a JavaScript file instead of importing selected functions or constants to a JavaScript file, follow these steps:
- In your workspace, in the Files pane, select a
.js
file in which you want to use the package. In the file, import the package in the following format:
const CONSTANT-NAME = require("PACKAGE-NAME");
- Replace CONSTANT-NAME with a name for the constant.
- Replace PACKAGE-NAME with the name of the package that you want to use.
The following code sample shows the getDomain
function from the
imported postoffice
package used in a JavaScript file:
/*
* Contents of postoffice index.js:
* module.exports = { getDomain };
*/
const postoffice = require("postoffice");
postoffice.getDomain();
Import a package function or constant to a SQLX file in Dataform
To use a function or a constant from a package inside a SQLX file, you need to first import it to the file.
To import a function or a constant from a package to a SQLX file, follow these steps:
- In your workspace, in the Files pane, select a
.sqlx
file in which you want to use the package. In the file, enter the following
js
block:js { const { EXPORT-NAME } = require("PACKAGE-NAME"); }
- Replace EXPORT-NAME with the name of the function
or constant that you want to use, declared in
module.exports
in the packageindex.js
file. - Replace PACKAGE-NAME with the name of the package that you want to use.
- Replace EXPORT-NAME with the name of the function
or constant that you want to use, declared in
The following code sample shows the getDomain
function from the
postoffice
package imported in a js
block and used in a
SELECT
statement in a SQLX file:
/*
* Contents of postoffice index.js:
* module.exports = { getDomain };
*/
config {
type: "table",
}
js {
const { getDomain } = require("postoffice");
}
SELECT ${getDomain("email")} as test
Import a whole package to a SQLX file in Dataform
To import the whole package to a SQLX file instead of importing selected functions or constants to a JavaScript file, follow these steps:
- In your workspace, in the Files pane, select a
.sqlx
file in which you want to use the package. In the file, import the package in the following format:
js { const CONSTANT-NAME = require("PACKAGE-NAME"); }
- Replace CONSTANT-NAME with a name for the constant.
- Replace PACKAGE-NAME with the name of the package that you want to use.
The following code sample shows the postoffice
package imported in
a js
block and its getDomain
function used in a
SELECT
statement in a SQLX file:
/*
* Contents of postoffice index.js:
* module.exports = { getDomain };
*/
config {
type: "table",
}
js {
const postoffice = require("postoffice");
}
SELECT ${postoffice.getDomain("email")} as test
What's next
- To learn how to authenticate a private NPM package in Dataform, see Authenticate a private package in Dataform.
- To learn more about packages in Dataform, see Reuse code across multiple repositories with packages.
- To learn how to create your own package in Dataform, see Create a package in Dataform.
- To learn how to use an open-source package in Dataform, see Use Slowly changing dimensions in Dataform.