Setting Environment Variables for a Pipeline in Cloud Data Fusion

Problem

Some plugins might require Environment Variables, however there isn't a straightforward way to do this in the Cloud Data Fusion web UI.

Sample error with Database plugin.

Environment

  • CDAP version 6.4.0 and earlier

Solution

Using the Compute Profile below, set Initialization Actions and export the environment variable(s) in the shell script.

  1. In Data Fusion, navigate to: System Admin > Configuration > System Compute Profiles
  2. Click on Create new > Advanced Settings > Initialization Actions.
  3. Otherwise, on a deployed pipeline, go to Configure > Compute Config
  4. Select a profile > Customize > Advanced Settings > Initialization Actions.

Cause

Missing product feature.