Install a package in Dataform

This document shows you how to install a JavaScript package in a Dataform repository, and import it to a JavaScript file and a SQLX file to use the package to develop SQL workflows in Dataform.

To use a package in Dataform, you need to install it in your repository.

You can install the following types of packages in Dataform:

Published public NPM packages.
Non-published public NPM packages.
Authenticated private NPM packages.

Then, to use the package in a JavaScript or SQLX file, you need to import selected contents of the package to the file. You can also import a whole package to a JavaScript or SQLX file instead of its selected contents.

To prevent issues with package installation in your production environment, we recommend that you do the following:

Explicitly specify the package version in package.json, for example, 3.0.0. Don't use other dependencies options of package.json, for example, >version.
Test new package versions in a non-production environment. For more information about configuring different code lifecycle environments, see Managing code lifecycle.

Before you begin

In the Google Cloud console, go to the Dataform page.

Go to the Dataform page
Select or create a repository.
Select or create a development workspace.
If your repository doesn't contain a package.json file, create package.json and move the Dataform core package.
Optional: To install a private package, authenticate the private package.

Required roles

To get the permissions that you need to install a package, ask your administrator to grant you the Dataform Editor (roles/dataform.editor) IAM role on workspaces. For more information about granting roles, see Manage access to projects, folders, and organizations.

You might also be able to get the required permissions through custom roles or other predefined roles.

Install a package

To install a package inside a Dataform repository, you need to add it as a dependency in the package.json file.

To add a package as a dependency in the package.json file, do the following:

In your workspace, in the Files pane, select package.json.
Add the package to the dependencies block:
1. Add a published public NPM package in the following format:
```
"PACKAGE-NAME": "PACKAGE-VERSION"
```
  Replace the following:
  - PACKAGE-NAME with the name of the package.
  - PACKAGE-VERSION with the latest version of the published public NPM package. To prevent issues with package installation, explicitly specify the version, for example, 3.0.0.
2. Add a non-published public NPM package in the following format:
```
"PACKAGE-NAME": "PACKAGE-URL"
```
  Replace the following:
  - PACKAGE-NAME with the name of the package.
  - PACKAGE-URL with the tar.gz URL of the third-party package repository, for example https://github.com/user/sample-package-repository/archive/master.tar.gz.
3. Add an authenticated private NPM package in the following format:
```
"REGISTRY-SCOPE/PACKAGE-NAME": "PACKAGE-URL"
```
  Replace the following:
  - REGISTRY-SCOPE with the name of the package. REGISTRY-SCOPE must match the registry scope defined in the .nmprc file in your repository.
  - PACKAGE-NAME with the name of the package.
  - PACKAGE-URL with the tar.gz URL of the package repository, for example https://github.com/user/sample-package-repository/archive/master.tar.gz.
Click Install packages.
Commit and push your changes.

The following code sample shows the public open-source Slowly changing dimensions package package added to the .package.json file:

 ```json
 {
   "name": "repository-name",
   "dependencies": {
     "@dataform/core": "2.0.3",
     "dataform-scd": "https://github.com/dataform-co/dataform-scd/archive/0.3.tar.gz"
   }
 }
 ```

Import a package function or constant to a JavaScript file in Dataform

To use a function or a constant from a package inside a JavaScript file in Dataform, you need to first import it to the file.

To import a function or a constant from a package to a JavaScript file, follow these steps:

In your workspace, in the Files pane, select a .js file in which you want to use the package.
In the file, import a function or a constant in the following format:
```
const { EXPORT-NAME } = require("PACKAGE-NAME");
```
1. Replace EXPORT-NAME with the name of the function or constant that you want to use, declared in module.exports in the package index.js file.
2. Replace PACKAGE-NAME with the name of the package that you want to use.
Commit and push your changes.

The following code sample shows the getDomain function from the postoffice package imported and used in a JavaScript file:

/*
 * Contents of postoffice index.js:
 * module.exports = { getDomain };
 */

const { getDomain } = require("postoffice");
getDomain();

Import a whole package to a JavaScript file in Dataform

To import the whole package to a JavaScript file instead of importing selected functions or constants to a JavaScript file, follow these steps:

In your workspace, in the Files pane, select a .js file in which you want to use the package.
In the file, import the package in the following format:
```
const CONSTANT-NAME = require("PACKAGE-NAME");
```
1. Replace CONSTANT-NAME with a name for the constant.
2. Replace PACKAGE-NAME with the name of the package that you want to use.
Commit and push your changes.

The following code sample shows the getDomain function from the imported postoffice package used in a JavaScript file:

/*
 * Contents of postoffice index.js:
 * module.exports = { getDomain };
 */

const postoffice = require("postoffice");
postoffice.getDomain();

Import a package function or constant to a SQLX file in Dataform

To use a function or a constant from a package inside a SQLX file, you need to first import it to the file.

To import a function or a constant from a package to a SQLX file, follow these steps:

In your workspace, in the Files pane, select a .sqlx file in which you want to use the package.
In the file, enter the following js block:
```
js {
  const { EXPORT-NAME } = require("PACKAGE-NAME");
}
```
1. Replace EXPORT-NAME with the name of the function or constant that you want to use, declared in module.exports in the package index.js file.
2. Replace PACKAGE-NAME with the name of the package that you want to use.
Commit and push your changes.

The following code sample shows the getDomain function from the postoffice package imported in a js block and used in a SELECT statement in a SQLX file:

/*
 * Contents of postoffice index.js:
 * module.exports = { getDomain };
 */

config {
    type: "table",
}

js {
  const { getDomain } = require("postoffice");
}

SELECT ${getDomain("email")} as test

Import a whole package to a SQLX file in Dataform

To import the whole package to a SQLX file instead of importing selected functions or constants to a JavaScript file, follow these steps:

In your workspace, in the Files pane, select a .sqlx file in which you want to use the package.
In the file, import the package in the following format:
```
js {
  const CONSTANT-NAME = require("PACKAGE-NAME");
}
```
1. Replace CONSTANT-NAME with a name for the constant.
2. Replace PACKAGE-NAME with the name of the package that you want to use.
Commit and push your changes.

The following code sample shows the postoffice package imported in a js block and its getDomain function used in a SELECT statement in a SQLX file:

/*
 * Contents of postoffice index.js:
 * module.exports = { getDomain };
 */

config {
    type: "table",
}

js {
  const postoffice = require("postoffice");
}

SELECT ${postoffice.getDomain("email")} as test

What's next

To learn how to authenticate a private NPM package in Dataform, see Authenticate a private package in Dataform.
To learn how to manage the required Dataform core package, see Manage the Dataform core package.
To learn how to create your own package in Dataform, see Create a package in Dataform.
To learn how to use an open-source package in Dataform, see Use Slowly changing dimensions in Dataform.