Create a package in Dataform

This document shows you how to create a custom JavaScript package that you can use to develop SQL workflows in Dataform.

To create a package that you can reuse in multiple Dataform repositories, you need to create a Dataform repository dedicated to the package and connect it to a third-party Git repository to make it available to other Dataform repositories.

Then, you need to create a top-level index.js file and add your exportable package contents, such as functions and constants, to the file. For an example of a package created in Dataform, see dataform-package-base on GitHub.

Once you create the package, you can install the package to in different Dataform repository and use the exportable contents on the package, such as constants and functions, to develop SQL workflows.

As an alternative to creating a package, you can reuse JavaScript functions and constants across a single Dataform repository with includes. For more information, see Reuse variables and functions with includes in Dataform.

Before you begin

  1. In the Google Cloud console, go to the Dataform page.

    Go to the Dataform page

  2. Create a Dataform repository dedicated to your package. Match the repository name to the name of your package.

  3. Connect the repository to a third-party Git repository that will host your package.

  4. Create and initialize a workspace in the Dataform repository.

Required roles

To get the permissions that you need to create a package, ask your administrator to grant you the Dataform Admin (roles/dataform.admin) IAM role on repositories. For more information about granting roles, see Manage access.

You might also be able to get the required permissions through custom roles or other predefined roles.

Create a package in Dataform

To create your own package with JavaScript code that you can reuse in Dataform, follow these steps in your workspace:

  1. In the Files pane, click the More menu.

  2. Click Create file.

    1. In the Create new file pane, do the following:

    2. In the Add a file path field, enter index.js.

    3. Click Create file.

  3. In the index.js file, enter the JavaScript code that you want your package to export.

    1. Create constants in the following format:

      const CONSTANT_NAME = CONSTANT_VALUE;
module.exports = { CONSTANT_NAME };

      Replace the following:

      • CONSTANT_NAME: the name of your constant
      • CONSTANT_VALUE: the value of your constant

    2. Create functions in the following format:

    function FUNCTION_NAME(PARAMETERS) { FUNCTION_BODY }

module.exports = { FUNCTION_NAME }

    Replace the following:

    • FUNCTION_NAME: the name of your function
    • PARAMETERS: the parameters of your function
    • FUNCTION_BODY: the code that you want the function to execute

  4. Optional: Click Format.

  5. Optional: In the definitions directory, add code of your package that will not be exported.

  6. Commit and push your changes.

The following package code sample shows the index.js file of the postoffice package that exports the getDomain function:

// filename index.js
// package name postoffice

const GENERIC_DOMAINS = "('samplemail.com','samplemail.co.uk','examplemailbox.com'";

function getDomain(email) {
  let cleanEmail = `trim(${email})`
  const domain = `substr(${cleanEmail}, strpos(${cleanEmail}, '@') + 1)`;
  return `case
            when ${domain} in ${common.GENERIC_DOMAINS} then ${cleanEmail}
            when ${domain} = "othermailbox.com" then "other.com"
            when ${domain} = "mailbox.com" then "mailbox.global"
            when ${domain} = "support.postman.com" then "postman.com"
            else ${domain}
          end`;
}

module.exports = { getDomain }

What's next