Authenticate a private package in Dataform

This document shows you how to authenticate a private NPM package in Dataform to enable its installation in a Dataform repository.

To install a private NPM package in a Dataform repository and use it to develop your SQL workflow, you need to first authenticate the package in Dataform. The authentication process is different for the first private package in a repository and a subsequent private package in a repository.

Before you begin

In the Google Cloud console, go to the Dataform page.

Go to the Dataform page
Select or create a repository.
Select or create a development workspace.

Required roles

To get the permissions that you need to authenticate private NPM packages in Dataform, ask your administrator to grant you the Dataform Editor (roles/dataform.editor) IAM role on repositories and workspaces. For more information about granting roles, see Manage access to projects, folders, and organizations.

You might also be able to get the required permissions through custom roles or other predefined roles.

Authenticate the first private package in a Dataform repository

To authenticate private NPM packages in Dataform, you need to do the following before you install the first private NPM package in a Dataform repository:

Create a Secret Manager secret dedicated to storing authentication tokens of private NPM packages in the Dataform repository.
1. Add the authentication token of the package, obtained from your NPM registry, to the secret.
You need to store all authentication tokens of the private NPM packages in your repository in a single secret. You need to create one dedicated secret per a Dataform repository. The secret must be in the JSON format.
Upload the secret to the Dataform repository.
Create an .npmrc file and add the authentication token of the package to the file.

The authentication token in the .npmrc file must match the authentication token in the uploaded secret.

After you authenticate the private NPM package, you can install the package in the Dataform repository.

Create a secret for authentication of private packages

To authenticate private NPM packages in a Dataform repository, you need to create a Secret Manager secret and define authentication tokens for all private packages that you want to install in the Dataform repository inside the secret. Define one authentication token per one private NPM package, and store all authentication tokens in a single secret per repository. The secret must be in the JSON format.

To create a secret with authentication tokens for private NPM packages, follow these steps:

In Secret Manager, create a secret.
1. In the Secret value field, enter one or multiple authentication tokens in the following format:
```
{
  "AUTHENTICATION_TOKEN_NAME": "TOKEN_VALUE"
}
```
Replace the following:
- AUTHENTICATION_TOKEN_NAME: a unique name for the token that identifies the package it authenticates
- TOKEN_VALUE: the value of the authentication token, obtained from your NPM registry
Grant access to the secret to your Dataform service account.

Your Dataform service account is in the following format:
```
service-PROJECT_NUMBER@gcp-sa-dataform.iam.gserviceaccount.com
```
1. When granting access, make sure to grant the roles/secretmanager.secretAccessor role to your Dataform service account.

Upload the secret for authentication of private packages to a Dataform repository

Before you install a private NPM package in a Dataform repository for the first time, upload your secret containing the authentication token of the package to the repository.

To upload the secret with private NPM packages authentication tokens to a Dataform repository, follow these steps:

In the Google Cloud console, go to the Dataform page.

Go to Dataform
Select the repository in which you want to install private NPM packages.
On the repository page, click Settings > Configure private NPM packages.
In the Add NPM package secret token pane, in the Secret drop-down, select your secret containing authentication tokens for private NPM packages.
Click Save.

Create an `.npmrc` file for authentication of private packages

To authenticate private NPM packages in a Dataform repository, you need to create a top-level .npmrc file in the repository. You need to store authentication tokens for all private NPM packages to be installed in the repository inside the .npmrc file. The authentication tokens in the .npmrc file must match the authentication tokens in the secret uploaded to the repository. For more information about .npmrc files, see npmrc documentation.

To create a top-level .npmrc file in your repository, follow these steps:

In the Google Cloud console, go to the Dataform page.

Go to Dataform
Select the repository in which you want to install private NPM packages, and then select a workspace.
In the Files pane, click the More menu, and then click Create file.
In the Create new file pane, do the following:
1. In the Add a file path field, enter .npmrc.
2. Click Create file.

Add an authentication token to the `.npmrc` file in a Dataform repository

To authenticate a private NPM package in a Dataform repository that already contains a secret with package authentication tokens and a .npmrc file, you need to add the authentication token for the private package to the .npmrc file in the repository.

In the .npmrc file, you need to define the scope of your NPM registry and add the authentication token for the private package accessed in that scope. For more information about .npmrc files, see npmrc documentation.

The authentication token in the .npmrc file must match the authentication token in the secret uploaded to the repository.

To add an authentication token to the .npmrc file in a Dataform repository, follow these steps:

In the Google Cloud console, go to the Dataform page.

Go to Dataform
Select the repository in which you want to install private NPM packages, and then select a workspace.
In the Files pane, select the .npmrc file.
In the .npmrc file, define the NPM registry scope and the authentication token for the private package in following format:

   @REGISTRY-SCOPE:registry=NPM-REGISTRY-URL
   NPM-REGISTRY-URL:_authToken=$AUTHENTICATION-TOKEN

Replace the following:

REGISTRY-SCOPE: the NPM registry scope to which you want to apply the authentication token
NPM-REGISTRY-URL: the URL of your NPM registry, for example, https://npm.pkg.github.com
AUTHENTICATION-TOKEN: the authentication token for the private NPM package. The authentication token in the .npmrc file must match the authentication token in the uploaded secret. The authentication token is provided as an environment variable in the .npmrc file, so make sure you add the opening ${ and } closing brackets.

You can enter multiple authentication tokens.

The following code sample shows an authentication token for a private NPM package added to the .npmrc file in a Dataform repository:

@company:registry=https://npm.pkg.github.com
//npm.pkg.github.com/:_authToken=${AUTHENTICATION_TOKEN}

Authenticate a subsequent private package in a Dataform repository

To authenticate a private NPM package in a Dataform repository that already contains a secret with package authentication tokens and an .npmrc file, follow these steps:

In Secret Manager, list secrets and select the secret that stores authentication tokens of private NPM packages of your repository.
Add a new version to the secret.

Dataform uses the latest version of the secret by default.
1. Add the authentication token for the private package to the secret value in the following format:
```
{
  "AUTHENTICATION_TOKEN_NAME": "TOKEN_VALUE"
}
```
Replace the following:
- AUTHENTICATION_TOKEN_NAME: a unique name for the token that identifies the package it authenticates
- TOKEN_VALUE: the value of the authentication token, obtained from your NPM registry
You can add multiple authentication tokens at once.
In Dataform, add the authentication token to the .npmrc file in your repository.