Cloud Data Fusion Plugins

A plugin is a customizable module that can be used to extend the capabilities of Cloud Data Fusion. The different plugin types are categorized as sources, transforms, analytics, actions, sinks, error handlers, alert publishers, and conditions. The plugin reference table below is a complete list of all plugins available in Cloud Data Fusion. If a plugin you need does not exist, you can build your own plugin by using Cloud Data Fusion's plugin APIs.

This table describes the different types of plugins available in Cloud Data Fusion.

Type Description
Sources Sources are connectors to databases, files, or real-time streams from which you obtain your data. They enable you to ingest data, using a simple UI, so you don't have to worry about coding low-level connections.
Transforms Transforms allow you to manipulate data after the data is ingested. For example, you may clone a record. You can format JSON. You can even write custom transforms using the JavaScript plugin.
Analytics Analytics plugins are used to perform aggregations such as grouping and joining data from different sources, as well as running analytics and machine learning operations. Data Fusion provides built-in plugins for a variety of such use cases.
Actions Action plugins define a custom action that is scheduled to take place during a workflow but doesn't directly manipulate data in the workflow. For example, using the database custom action, you can run an arbitrary database command at the end of your pipeline. Alternatively, you can trigger an action to move files within Cloud Storage.
Sinks Data must be written to a sink. Cloud Data Fusion contains various sinks, such as Cloud Storage, BigQuery, Cloud Spanner, relational databases, file systems, and mainframes.
Error handlers When nodes encounter null values, logical errors, or other sources of errors, you may use an error handler plugin to catch errors. You can connect this plugin to the output of any transform or analytics plugin, which catches errors. You can then process these errors in a separate error-processing flow in your pipeline.
Alert publishers Another type of plugin is an alert publisher, which allows you to publish notifications when uncommon events occur. Downstream processes can then subscribe to these notifications to trigger custom processing for these alerts.
Conditions Pipelines also offer control flow plugins in the form of conditions. Condition plugins allow you to branch your pipeline into two separate paths, depending on whether the specified condition predicate evaluates to true or false.