This page describes how to create and use a trigger for your Cloud Data Fusion pipeline. You create a trigger on a data pipeline, called the downstream pipeline, to have it run at the completion of another pipeline, called the upstream pipeline. When creating a trigger, you choose when the downstream pipeline runs - when the upstream pipeline run succeeds, fails, stops, or any combination thereof.
Triggers are useful for:
- Cleansing your data once and making it available to multiple downstream pipelines.
- Sharing information, such as runtime arguments and plugin configurations, between pipelines. This is called payload configuration.
- Having a set of dynamic pipelines that can run using the data of the hour/day/week/month, instead of a static pipeline that needs to be updated for every run.
Before you begin
In the Pipeline Studio, deploy the pipelines that are your upstream and downstream pipelines.
Optional: Set runtime arguments for your upstream pipeline
If you want to pass payload configuration as runtime arguments, set the runtime arguments for your upstream pipeline:
Go to the List page. In the Deployed tab, click the name of the upstream pipeline. The Deploy view for that pipeline appears.
Click the arrow to the right of the Run button.
Click the + button and fill in the Key and Value for your runtime argument.
Click Save.
Create an inbound trigger on a downstream pipeline
Create a trigger on the downstream pipeline to have it run at the completion of the upstream pipeline.
Go to the List page. In the Deployed tab, click the name of the downstream pipeline. The Deploy view for that pipeline appears.
On the middle left side of the page, click Inbound triggers. A list of available pipelines appears.
Click the upstream pipeline. Select one or more of the upstream pipeline completion states (Succeeds, Fails, or Stops) as the condition for when the downstream pipeline should run.
If you want the upstream pipeline to share information (called payload configuration) with the downstream pipeline, click Trigger config and then follow the steps for Passing payload configuration as runtime arguments. Otherwise, click Enable trigger.
Pass payload configuration as runtime arguments
Payload configuration allows sharing of information from the upstream pipeline to the downstream pipeline. This information can be, for example, the output directory, the data format, or the day the pipeline was run. This information is then used by the downstream pipeline for decisions such as determining the right dataset to read from.
To pass information from the upstream pipeline to the downstream pipeline, you set the runtime arguments of the downstream pipeline with the values of either the runtime arguments or the configuration of any plugin in the upstream pipeline.
Whenever the downstream pipeline runs (is triggered), its payload configuration is set using the runtime arguments of the particular run of the upstream pipeline that triggered the downstream pipeline.
To pass payload configuration as runtime arguments:
Picking up where you left off in the Creating an inbound trigger section, after clicking Trigger Config, any runtime arguments you previously set for your upstream pipeline will appear. Select the runtime arguments that you want to pass from the upstream pipeline to the downstream pipeline when this trigger executes.
Click the Plugin config tab to see a list of what will be passed from your upstream pipeline to your downstream pipeline when it is triggered.
Click Configure and Enable Trigger.
Test your downstream pipeline trigger
To test your trigger, follow these steps:
Go to the List page. In the Deployed tab, click the name of the upstream pipeline. The Deploy view for that pipeline appears.
Click Run.
Wait for the upstream pipeline run to complete. From the List view, navigate to the downstream pipeline. You should see it running, as a result of the trigger you created.