Jump to
Cloud Data Fusion Plugins

Cloud Data Fusion Plugins

Plugins are customizable modules that extend the capabilities of Cloud Data Fusion. The following is a complete list of plugins available in Cloud Data Fusion, but you can also build your own plugin with plugin APIs.

Contact us Contact us

Most popular

Quickly discover and access the most popular Cloud Data Fusion plugins.
BigQuery
This source reads the entire contents of a BigQuery table.
BigQuery Multi Table
This sink writes to multiple BigQuery tables. Data is first written to a temporary location on Cloud Storage, then loaded into BigQuery from there.
BigQuery Table
This sink writes to a BigQuery table. Data is first written to a temporary location on Cloud Storage, then loaded into BigQuery from there.
Cloud Storage
This source plugin reads objects from a path in a Cloud Storage bucket.
Cloud Storage
This sink plugin writes records to one or more files in a directory on Cloud Storage in various formats such as CSV, Avro, Parquet, and JSON.
Multi Database Table
Reads from multiple tables within a database. Used with multi sink plugins like Cloud Storage multi sink and BigQuery multi sink to load data from entire database in a single pipeline.
MySQL
Reads from a MySQL instance using a configurable SQL query. Outputs one record for each row returned by the query.
Oracle
Reads from an Oracle table using a configurable SQL query. Outputs one record for each row returned by the query.
Salesforce
This source reads sObjects such as opportunities, contacts, accounts, leads, any custom object, etc. from Salesforce in batch.
SAP ODP
Connects to an SAP source for batch-based and delta-based data extraction through Operational Data Provisioning (ODP).
SQL Server
Reads from a SQL Server using a configurable SQL query. Outputs one record for each row returned by the query.

Google Cloud

See all of the plugins available for Google Cloud products including Cloud Storage, BigQuery, Cloud Bigtable, Datastore, Pub/Sub, and Google Workspace.
BigQuery
This source reads the entire contents of a BigQuery table.
BigQuery Argument Setter
Reads from a BigQuery table to fetch arguments to set in the pipeline.
BigQuery Execute
This plugin executes a BigQuery SQL Query.
BigQuery Multi Table
This sink writes to multiple BigQuery tables. Data is first written to a temporary location on Cloud Storage, then loaded into BigQuery from there.
BigQuery Table
This sink writes to a BigQuery table. Data is first written to a temporary location on Cloud Storage, then loaded into BigQuery from there.
Cloud Bigtable
Batch sink that writes data to Cloud Bigtable.
Cloud Bigtable
This source reads data from Cloud Bigtable.
Dataplex Sink
Ingests data to any of the Dataplex supported assets.
Dataplex Source
Reads data from Dataplex entities (tables) residing on Cloud Storage or BigQuery assets.
Datastore
This source reads data from Datastore. Datastore is a NoSQL document database built for automatic scaling and high performance.
Datastore
This sink writes data to Cloud Datastore. Datastore is a NoSQL document database built for automatic scaling and high performance.
Firestore
Reads from Cloud Firestore in batch pipelines.
Firestore
Plugin to write to Cloud Firestore in batch and streaming pipelines.
Cloud Storage
This sink plugin writes records to one or more files in a directory on Cloud Storage in various formats such as CSV, Avro, Parquet, and JSON.
Cloud Storage
This source plugin reads objects from a path in a Cloud Storage bucket.
Cloud Storage Argument Setter
Reads from a Cloud Storage file to fetch arguments to set in the pipeline.
Cloud Storage Delete
This plugin deletes objects in a Cloud Storage bucket.
Cloud Storage Copy
This plugin copies objects from one Cloud Storage bucket to another. A single object can be copied, or a directory of objects can be copied.
Cloud Storage Create
This plugin creates objects in a Cloud Storage bucket.
Cloud Storage Done File Marker
Creates an empty DONE (or SUCCESS) file in the given Cloud Storage bucket upon pipeline completion, success, or failure, to orchestrate dependent processes.
Cloud Storage Move
This plugin moves objects from one Cloud Storage bucket to another. A single object can be moved, or a directory of objects can be moved.
Cloud Storage Multi FIle
This plugin is used in conjunction with the MultiTableDatabase source to write records from multiple databases into separate directories.
Cloud DLP Decrypt
Decrypts sensitive data that was earlier encrypted by Cloud Data Loss Prevention (DLP) using a reversible encryption transform, such as Format Preserving Encryption.
Cloud DLP PII Filter
Separates sensitive records based on whether they match a user-defined template using Cloud Data Loss Prevention (DLP) APIs.
Cloud DLP Redact
Redacts sensitive records using Cloud Data Loss Prevention (DLP) by matching some pre-defined DLP filters or a custom user-defined template.
Google Drive
Reads files from specified Google Drive directory via Google Drive API.
Google Drive
Sink plugin saves files from the pipeline to Google Drive directory via Google Drive API.
Google Sheets
Writes spreadsheets to specified Google Drive directory.
Google Sheets
Reads spreadsheets from specified Google Drive directory.
Pub/Sub
This sink writes to a Pub/Sub topic in streaming pipelines.
Pub/Sub
This sources reads from a Pub/Sub subscription in realtime.
Spanner
This sink writes to a Cloud Spanner table.
Spanner
This source reads from a Cloud Spanner table.
Speech To Text
This plugin converts audio files to text by using Speech-to-Text.

Aggregate

Plugins to aggregate functions like as avg, count, count(*), first, last, max, min, sum, collect including table joins, etc.
Group By
Groups by one or more fields, then performs aggregate functions such as avg, count, count(*), first, last, max, min, sum, collect on each group.
Joiner
Joins records from one or more input based on join keys. Supports inner and outer joins, selection, and renaming of output fields.
Top N
Top N returns the top "n" records from the input set, based on the criteria specified in the plugin configuration.
Window Aggregation
Applies aggregation functions over a window of data.

Alerting

Plugins for alerting.
Email
Sends an email at the end of a pipeline run.
Email by SendGrid
Action to send an email at the end of a pipeline run.
Cloud Storage Done File Marker
Creates an empty DONE (or SUCCESS) file in the given Cloud Storage bucket upon pipeline completion, success, or failure, to orchestrate dependent processes.
HTTP Callback
Performs an HTTP request at the end of a pipeline run.
Kafka Alert Publisher
Kafka Alert Publisher that allows you to publish alerts to Kafka as JSON objects.
MariaDB
Runs a MariaDB query at the end of the pipeline run. Can be configured to run only on success, only on failure, or always at the end of the run.
Netezza
Runs a Netezza query at the end of the pipeline run. Can be configured to run only on success, only on failure, or always at the end of the run.
Oracle
Runs an Oracle query at the end of the pipeline run. Can be configured to run only on success, only on failure, or always at the end of the run.
PostgreSQL
Runs a PostgreSQL query at the end of the pipeline run. Can be configured to run only on success, only on failure, or always at the end of the run.
SQL Server
Runs a SQL Server query at the end of the pipeline run. Can be configured to run only on success, only on failure, or always at the end of the run.
Teradata
Runs a Teradata query at the end of the pipeline run. Can be configured to run only on success, only on failure, or always at the end of the run.
TMS
Publishes alerts to the CDAP Transactional Messaging System (TMS) as JSON objects.

Apache

Plugins for Apache.
Cassandra
Batch sink that writes data to Apache Cassandra.
Cassandra
Batch source that reads from Apache Cassandra.
HBase
Writes records to an HBase table with one field mapping to the row key, and all other record fields mapping to column qualifiers.
HBase
Batch source that reads from a column family in an HBase table.
Hive Bulk Export
Hive Export will take select query as input to run that query on hive table and store results in the specified location.
Hive Bulk Import
Imports data from a directory/file into a Hive table using Hive Load Statement.
HTTP to HDFS
Action to fetch data from an external http endpoint and create a file in HDFS.
Kafka Alert Publisher
Kafka Alert Publisher that allows you to publish alerts to Kafka as JSON objects.
Kafka Consumer
Kafka batch source that reads events from Apache Kafka in batch pipelines.
Kafka Consumer
Kafka streaming source that reads events from Apache Kafka in streaming pipelines.
Kafka Producer
Kafka sink that allows you to write events into CSV or JSON to Kafka.
Kudu
CDAP plugin for ingesting data into Apache Kudu. Plugin can be configured for both batch and real-time pipelines.
Kudu
CDAP plugin for reading data from Apache Kudu table.
PySpark Program
Executes user-provided Spark code in Python.
Repartitioner
This plugin re-partitions a Spark RDD.
Scala Spark Compute
Executes user-provided Spark code in Scala that transforms RDD to RDD with full access to all Spark features.
Scala Spark Program
Executes user-provided Spark code in Scala.
Spark
Executes user-provided Spark code in Scala that operates on an input RDD or Dataframe with full access to all Spark features.
Webserver Log Parser
Parses logs from any input source for relevant information such as URI, IP, browser, device, HTTP status code, and timestamp.

APIs

Plugins for various APIs.
Anaplan
Source plugin for loading data from an Anaplan model into a data pipeline.
Anaplan
Sink plugin for supporting data import from CDAP to Anaplan server.
Anaplan Function
Performs an Action/Process that is predefined in the given Anaplan model. This is most commonly used to prepare for the data import and export.
Facebook Ads
Reads Ads data from Facebook Insights API in batch pipelines.
GitHub
Reads from GitHub in batch pipelines.
Google Ads Multiple Reports
Reads all of the Google Ads reports from a Google Ads account in batch pipelines.
Google Ads Single Report
Reads the specified report type from Google Ads accounts in batch pipelines.
Google Analytics Batch
Reads from Google Analytics 360 in batch pipelines.
Google Analytics Streaming
Reads from Google Analytics 360 in streaming pipelines.
Google Drive
Reads files from specified Google Drive directory via Google Drive API.
Google Drive
Sink plugin saves files from the pipeline to Google Drive directory via Google Drive API.
Google Search Console
Reads from Google Search Console in batch pipelines.
Google Sheets
Writes spreadsheets to specified Google Drive directory.
Google Sheets
Reads spreadsheets from specified Google Drive directory.
HTTP
Reads data in JSON, XML, CSV, TSV, TEXT and BLOB formats from HTTP/HTTPS APIs in batch pipelines.
HTTP
Sink plugin to send the messages from the pipeline to an external http endpoint.
HTTP Callback
Performs an HTTP request at the end of a pipeline run.
HTTP Poller
This is a streaming source that will fetch data from a specified URL at a given interval and pass the results to the next plugin.
HTTP Streaming
Reads data in JSON, XML, CSV, TSV, TEXT and BLOB formats from HTTP/HTTPS APIs in streaming pipelines.
HTTP to HDFS
Action to fetch data from an external http endpoint and create a file in HDFS.
HubSpot
Plugin to write data to HubSpot CRM in batch and streaming pipelines.
HubSpot
Reads data from HubSpot CRM in streaming pipelines.
HubSpot
Reads data from HubSpot CRM in batch pipelines.
Salesforce
This source reads sObjects such as opportunities, contacts, accounts, leads, any custom object, etc. from Salesforce in batch.
Salesforce
This source tracks updates in Salesforce sObjects. Examples of sObjects are opportunities, contacts, accounts, leads, any custom object, etc.
Salesforce
A batch sink that inserts sObjects such as opportunities, contacts, accounts, leads, any custom objects, etc. into Salesforce.
Salesforce Marketing Cloud
This source tracks updates in Salesforce sObjects such as opportunities, contacts, accounts, leads, any custom object, etc.
Salesforce Marketing Cloud
This sink inserts records into a Salesforce Marketing Cloud (SFMC) Data Extension via server-to-server integration with the SFMC API.
Salesforce Multi Objects
This source reads multiple sObjects from Salesforce. It should be used in conjunction with multi-sinks.
ServiceNow
Reads from one or multiple tables within ServiceNow.
Speech To Text
This plugin converts audio files to text by using Speech-to-Text.
Splunk
Reads from Splunk in streaming pipelines.
Splunk
Reads from Splunk in batch pipelines.
Zendesk
Reads objects such as Article Comments, Post Comments, Requests Comments, Ticket Comments, Groups, Organizations, Satisfaction Ratings, Tags, Ticket Fields, Ticket Metrics, Ticket Metric Events, Tickets, Users from Zendesk.
Zendesk Multi Objects
Multi-source that reads multiple objects from Zendesk and routes them to specific destinations. Should be used in conjunction with a multi-sink.
Zuora
Plugin to write data to Zuora in batch and streaming pipelines.
Zuora
Reads data from Zuora in batch pipelines.

Avro

Plugins for Avro.
File
Writes to a filesystem in various formats.
File
This source is used whenever you need to read from a distributed file system.
Cloud Storage
Writes records to one or more files in a directory on Cloud Storage in various formats such as CSV, Avro, Parquet, and JSON.

AWS

Available plugins for AWS.
Amazon S3
This source is used whenever you need to read from Amazon S3.
Amazon S3
This sink is used to write to Amazon S3 in various formats.
Amazon S3 Client
The Amazon S3 Client Action is used to work with S3 buckets and objects before or after the execution of a pipeline.
Kinesis Sink
Kinesis sink that outputs to a specified Amazon Kinesis Stream.
Kinesis Source
Spark streaming source that reads from AWS Kinesis streams.
Redshift to S3
The Redshift To S3 Action runs the UNLOAD command on AWS to save the results of a query from Redshift to one or more files on Amazon S3.
S3 to Redshift
S3 To Redshift Action that will load the data from AWS S3 bucket into the AWS Redshift table.

Cleansing

Plugins for cleansing.
Deduplicate
De-duplicates records either using one or more fields or by using the record as a whole. Additionally, it supports logically choosing a record out of the duplicate records based on a filter field. Supported logical functions are max, min, first, and last.
Distinct
De-duplicates input records so that all output records are distinct.
Fast Filter
Filters out messages based on a specified criteria.
Cloud DLP PII Filter
Separates sensitive records based on whether they match a user-defined template using Cloud Data Loss Prevention (DLP) APIs.
Wrangler
This plugin applies data transformation directives generated either through an interactive UI or manually on your data records.

Cloud Storage

Available plugins for cloud storage.
Amazon S3
This source is used whenever you need to read from Amazon S3.
Azure Blob Store
Batch source to use Microsoft Azure Blob Storage as a source.
Azure Decompress
Azure Decompress Action plugin decompress gz files from a container on Azure Storage Blob service into another container.
Cloud Storage to Snowflake
Loads data from external or internal staged files on Cloud Storage to an existing Snowflake table.
Cloud Storage
This sink plugin writes records to one or more files in a directory on Cloud Storage in various formats such as CSV, Avro, Parquet, and JSON.
Cloud Storage
This plugin reads objects from a path in a Cloud Storage bucket.
Cloud Storage Argument Setter
Reads from a Cloud Storage file to fetch arguments to set in the pipeline.
Cloud Storage Delete
This plugin deletes objects in a Cloud Storage bucket.
Cloud Storage Copy
This plugin copies objects from one Cloud Storage bucket to another. A single object can be copied, or a directory of objects can be copied.
Cloud Storage Create
This plugin creates objects in a Cloud Storage bucket.
Cloud Storage Move
This plugin moves objects from one Cloud Storage bucket to another. A single object can be moved, or a directory of objects can be moved.
Cloud Storage Multi FIle
This plugin is used in conjunction with the MultiTableDatabase source to write records from multiple databases into separate directories.
Cloud Storage Done File Marker
Creates an empty DONE (or SUCCESS) file in the given Cloud Storage bucket upon pipeline completion, success, or failure, to orchestrate dependent processes.
Microsoft Azure Delete
Azure Delete Action plugin deletes a container on Azure Storage Blob service.
Snowflake to Cloud Storage
Unloads data from a Snowflake table to an internal or external Cloud Storage location (stage).

Code

Plugins to execute your own code. 
JavaScript
Executes user-provided JavaScript that transforms one record into zero or more records.
PySpark Program
Executes user-provided Spark code in Python.
Python Evaluator
Executes user-provided Python code for transforming input records, which are available as Python dictionary.
Scala Spark Compute
Executes user-provided Spark code in Scala that transforms RDD to RDD with full access to all Spark features.
Scala Spark Program
Executes user-provided Spark code in Scala.
Spark
Executes user-provided Spark code in Scala that operates on an input RDD or Dataframe with full access to all Spark features.

Compression

Plugins for compression.
Compressor
Compresses configured fields using different compression algorithms such as SNAPPY, ZIP, and GZIP.
Azure Decompress
The Decompress Action is used to expand archived or compressed (.zip, .gz, .tar, .tar.gz, etc) files before further processing in a pipeline.
Field Decompresser
Decompresses configured fields using compression algorithms such as SNAPPY, ZIP, and GZIP.

Conditional

Pipelines also offer control flow plugins in the form of conditions. Condition plugins allow you to branch your pipeline into two separate paths, depending on whether the specified condition predicate evaluates to true or false.
Conditional
A control flow plugin that allows conditional execution within pipelines based on expressions.
Error Collector
Collects errors emitted from the previous stage and adds an error message, code, and stage to the record.
Fail Pipeline
Batch Sink that causes a pipeline to fail immediately after receiving the first record.
Null Field Splitter
This plugin sends records with a null value for a given field to the null port and those with a non-null value to the nonnull port.
Decompress
Routes a record to an appropriate port based on the evaluation of an expression on the value of one or more of its fields.
Union Splitter
The union splitter is used to split data by a union schema, so that type-specific logic can be done downstream.

Control Flow

Plugins for control flow.
Anaplan Function
Performs an Action/Process that is predefined in the given Anaplan model. This is most commonly used to prepare for the data import and export.
BigQuery Execute
Execute a SQL query at the beginning or end of a pipeline.
Database
Action that runs a database command.
Database Query
Runs a database query at the end of the pipeline run. Can be configured to run only on success, only on failure, or always at the end of the run.
DB2 Execute
Action that runs a DB2 command.
Azure Decompress
The Decompress Action is used to expand archived or compressed (.zip, .gz, .tar, .tar.gz, etc) files before further processing in a pipeline.
File Contents Checketer
This action plugin can be used to check if a file is empty or if the contents of a file match a given pattern.
File Delete
Deletes a file or files.
File Move
Moves a file or files.
FTP Copy
Copy files from FTP server to the specified destination.
FTP Put
Copy files to FTP server from a specified destination.
Cloud Storage Delete
This plugin deletes objects in a Cloud Storage bucket.
Cloud Storage Copy
This plugin copies objects from one Cloud Storage bucket to another. A single object can be copied, or a directory of objects can be copied.
Cloud Storage Create
This plugin creates objects in a Cloud Storage bucket.
Cloud Storage Move
This plugin moves objects from one Cloud Storage bucket to another. A single object can be moved, or a directory of objects can be moved.
HTTP to HDFS
Action to fetch data from an external http endpoint and create a file in HDFS.
MariaDB Execute
Action that runs a MariaDB command.
MySQL Execute
Action that runs a MySQL command.
Netezza Execute
Action that runs a Netezza command.
Oracle Execute
Action that runs an Oracle command.
Oracle Export
An Action plugin to efficiently export data from Oracle to a file system using Oracle's command-line tools.
PostgreSQL Execute
Action that runs a PostgreSQL command.
Redshift to S3
The Redshift To S3 Action runs the UNLOAD command on AWS to save the results of a query from Redshift to one or more files on Amazon S3.
Remote Program Executor
Establishes an SSH connection with remote machine to execute command on that machine.
S3 to Redshift
S3 To Redshift Action that will load the data from AWS S3 bucket into the AWS Redshift table.
SFTP Copy
SFTP copy allows copying of the files from the specified directory on SFTP servers to the specified destination.
SFTP Delete
CDAP Action plugin to delete the specified files from an SFTP server.
SFTP Put
SFTP put allows uploading of file(s) from filesystem (local/HDFS) to FTP server. File Regex can also be used to filter only the files that are of interest.
Snowflake Run SQL
Runs a specified SQL query in Snowflake.
SQL Server Execute
Action that runs a SQL Server command.
Teradata Execute
Action that runs a Teradata command.
Windows Share Copy
Copies a file or files on a Microsoft Windows share to a filesystem.

CRM

Plugins for CRM platforms like Salesforce.
HubSpot
Plugin to write data to HubSpot CRM in batch and streaming pipelines.
HubSpot
Reads data from HubSpot CRM in streaming pipelines.
HubSpot
Reads data from HubSpot CRM in batch pipelines.
Salesforce
This source reads sObjects such as opportunities, contacts, accounts, leads, any custom object, etc. from Salesforce in batch.
Salesforce
A batch sink that inserts sObjects such as opportunities, contacts, accounts, leads, any custom objects, etc. into Salesforce.
Salesforce
This source tracks updates in Salesforce sObjects. Examples of sObjects are opportunities, contacts, accounts, leads, any custom object, etc.
Salesforce Marketing Cloud
This source tracks updates in Salesforce sObjects such as opportunities, contacts, accounts, leads, any custom object, etc.
Salesforce Marketing Cloud
This sink inserts records into a Salesforce Marketing Cloud (SFMC) Data Extension via server-to-server integration with the SFMC API.
Salesforce Multi Objects
This source reads multiple sObjects from Salesforce. It should be used in conjunction with multi-sinks.

CSV

Plugins for CSV.
CSV Formatter
Formats input into a CSV Record using formats such as DELIMITED, EXCEL, MYSQL, RFC4180, and TDF.
CSV Parser
Parses a CSV Record formatted as DEFAULT, EXCEL, MYSQL, RFC4180, Tab Delimited, Pipe Delimited into a Structured Record.
File
Writes to a filesystem in various formats.
File
This source is used whenever you need to read from a distributed file system.
Cloud Storage
This sink plugin writes records to one or more files in a directory on Cloud Storage in various formats such as CSV, Avro, Parquet, and JSON.
Kafka Producer
Kafka sink that allows you to write events into CSV or JSON to Kafka.

Database

Plugins for your databases.
Cloud Bigtable
Batch sink that writes data to Cloud Bigtable.
Cloud Bigtable
This source reads data from Cloud Bigtable.
Database
Action that runs a database command.
Database
Reads from a database using a configurable SQL query. Outputs one record for each row returned by the query.
Database
Writes records to a database table. Each record will be written to a row in the table.
Database Argument Setter
Reads from a relational database table to fetch arguments to set in the pipeline.
Database Query
Runs a database query at the end of the pipeline run. Can be configured to run only on success, only on failure, or always at the end of the run.
DB2
Reads from a DB2 using a configurable SQL query. Outputs one record for each row returned by the query.
DB2
Runs a DB2 query at the end of the pipeline run. Can be configured to run only on success, only on failure, or always at the end of the run.
DB2 Execute
Action that runs a DB2 command.
IBM DB2
Writes records to a DB2 table. Each record will be written to a row in the table.
MariaDB
Writes records to a MariaDB table. Each record will be written to a row in the table.
MariaDB
Reads from a MariaDB using a configurable SQL query. Outputs one record for each row returned by the query.
MariaDB
Runs a MariaDB query at the end of the pipeline run. Can be configured to run only on success, only on failure, or always at the end of the run.
MariaDB Execute
Action that runs a MariaDB command.
Multi Database Table
Reads from multiple tables within a database. Used with multi sink plugins like, Cloud Storage multi sink and BigQuery multi sink to load data from entire database in a single pipeline.
MySQL Query
Runs a MySQL query at the end of the pipeline run. Can be configured to run only on success, only on failure, or always at the end of the run.
MySQL
Writes records to a MySQL table. Each record will be written to a row in the table.
MySQL
Reads from a MySQL instance using a configurable SQL query. Outputs one record for each row returned by the query.
MySQL Execute
Action that runs a MySQL command.
Oracle
Reads from an Oracle table using a configurable SQL query. Outputs one record for each row returned by the query.
Oracle
Writes records to an Oracle table. Each record will be written to a row in the table.
Oracle
Runs an Oracle query at the end of the pipeline run. Can be configured to run only on success, only on failure, or always at the end of the run.
Oracle Execute
Action that runs an Oracle command.
Oracle Export
An Action plugin to efficiently export data from Oracle to a file system using Oracle's command-line tools.
PostgreSQL
Reads from a PostgreSQL using a configurable SQL query. Outputs one record for each row returned by the query.
PostgreSQL
Writes records to a PostgreSQL table. Each record will be written to a row in the table.
PostgreSQL
Runs a PostgreSQL query at the end of the pipeline run. Can be configured to run only on success, only on failure, or always at the end of the run.
PostgreSQL Execute
Action that runs a PostgreSQL command.
Spanner
This sink writes to a Cloud Spanner table.
Spanner
This source reads from a Cloud Spanner table.
SQL Server
Writes records to a SQL Server table. Each record will be written to a row in the table.
SQL Server
Runs a SQL Server query at the end of the pipeline run. Can be configured to run only on success, only on failure, or always at the end of the run.
SQL Server
Reads from a SQL Server using a configurable SQL query. Outputs one record for each row returned by the query.
SQL Server Execute
Action that runs a SQL Server command.
IBM DB2
DB2 Runs a DB2 query at the end of the pipeline run. Can be configured to run only on success, only on failure, or always at the end of the run.

Data Warehouse

Plugins for your data warehouse.
BigQuery
This source reads the entire contents of a BigQuery table.
BigQuery Argument Setter
Reads from a BigQuery table to fetch arguments to set in the pipeline.
BigQuery Execute
This plugin executes a BigQuery SQL query.
BigQuery Multi Table
This sink writes to multiple BigQuery tables. Data is first written to a temporary location on Cloud Storage, then loaded into BigQuery from there.
BigQuery Table
This sink writes to a BigQuery table. Data is first written to a temporary location on Cloud Storage, then loaded into BigQuery from there.
Cloud Storage to Snowflake
Loads data from external or internal staged files on Cloud Storage to an existing Snowflake table.
Hive Bulk Export
Hive Export will take select query as input to run that query on hive table and store results in the specified location.
Hive Bulk Import
Imports data from a directory/file into a Hive table using Hive Load Statement.
Netezza
Writes records to a Netezza table. Each record will be written to a row in the table.
Netezza
Reads from a Netezza using a configurable SQL query. Outputs one record for each row returned by the query.
Netezza
Runs a Netezza query at the end of the pipeline run. Can be configured to run only on success, only on failure, or always at the end of the run.
Netezza Execute
Action that runs a Netezza command.
Redshift to S3
The Redshift To S3 Action runs the UNLOAD command on AWS to save the results of a query from Redshift to one or more files on Amazon S3.
S3 to Redshift
S3 To Redshift Action that will load the data from AWS S3 bucket into the AWS Redshift table.
SCD2 Spark Compute
Creates an SCD2 representation of a record by grouping source records by the specified key and generating the start and end (expiry) dates for it.
Snowflake
Writes data to a Snowflake table in batch pipelines.
Snowflake
Reads data from Snowflake tables in batch pipelines.
Snowflake Run SQL
Runs a specified SQL query in Snowflake.
Snowflake to Cloud Storage
Unloads data from a Snowflake table to an internal or external Cloud Storage location (stage).
Teradata
Writes records to a Teradata table. Each record will be written to a row in the table.
Teradata
Reads from a Teradata using a configurable SQL query. Outputs one record for each row returned by the query.
Teradata
Runs a Teradata query at the end of the pipeline run. Can be configured to run only on success, only on failure, or always at the end of the run.
Teradata
Action that runs a Teradata command.
Vertica Bulk Export Action
Bulk exports data in a Vertica table into a file.
Vertica Bulk Import Action
Vertica Bulk Import Action plugin reads all the files in a given directory and bulk imports the contents into a Vertica table.

DLP

Plugins for Cloud Data Loss Prevention (DLP).
Cloud DLP Decrypt
Decrypts sensitive data that was earlier encrypted by Cloud Data Loss Prevention (DLP) using a reversible encryption transform, such as Format Preserving Encryption.
Cloud DLP PII Filter
Separates sensitive records based on whether they match a user-defined template using Cloud Data Loss Prevention (DLP) APIs.
Cloud DLP Redact
Redacts sensitive records using Cloud Data Loss Prevention (DLP) by matching some pre-defined DLP filters or a custom user-defined template.

Dynamic

Dynamic plugins.
Argument Setter
Performs an HTTP request to fetch arguments to set in the pipeline.
Database Argument Setter
Reads from a relational database table to fetch arguments to set in the pipeline.
Cloud Storage Argument Setter
Reads from a Cloud Storage file to fetch arguments to set in the pipeline.

Email

Plugins for email.
Email
Sends an email at the end of a pipeline run.
Email by SendGrid
Action to send an email at the end of a pipeline run.
SendGrid
Reads data about existing marketing campaigns, email analytics, bounces, spam reports from SendGrid.
SendGrid
Writes data to SendGrid in batch and streaming pipelines.

File

File plugins.
CSV Formatter
Formats input into a CSV Record using formats such as DELIMITED, EXCEL, MYSQL, RFC4180, and TDF.
CSV Parser
Parses a CSV Record formatted as DEFAULT, EXCEL, MYSQL, RFC4180, Tab Delimited, Pipe Delimited into a Structured Record.
File
Writes to a filesystem in various formats.
File
This source is used whenever you need to read from a distributed file system.
File
File streaming source that watches a directory and streams file contents of any new files added to the directory.
File Appender
This sink plugin appends to an existing file until it hits either a size or an age based threshold.
File Contents Checketer
This action plugin can be used to check if a file is empty or if the contents of a file match a given pattern.
File Delete
Deletes a file or files.
File Move
Moves a file or files.
FTP
Batch source to read from FTP (ftp://..) or SFTP (sftp://...).
FTP Copy
Copy files from FTP server to the specified destination.
FTP Put
Copy files to FTP server from a specified destination.
PDF Extractor
This transform leverages the Apache PDF Box library to extract text and metadata from a PDF file.
Redshift to S3
The Redshift To S3 Action runs the UNLOAD command on AWS to save the results of a query from Redshift to one or more files on Amazon S3.
Amazon S3
This sink is used to write to Amazon S3 in various formats.
S3 to Redshift
S3 To Redshift Action that will load the data from AWS S3 bucket into the AWS Redshift table.
SFTP Copy
SFTP copy allows copying of the files from the specified directory on SFTP servers to the specified destination.
SFTP Delete
CDAP Action plugin to delete the specified files from an SFTP server.
SFTP Put
SFTP put allows uploading of file(s) from filesystem (local/HDFS) to FTP server. File Regex can also be used to filter only the files that are of interest.

Filter

Filter plugins.
Deduplicate
De-duplicates records either using one or more fields or by using the record as a whole. Additionally, it supports logically choosing a record out of the duplicate records based on a filter field. Supported logical functions are max, min, first, and last.
Distinct
De-duplicates input records so that all output records are distinct.
Fast Filter
Filters out messages based on a specified criteria.
Cloud DLP PII Filter
Separates sensitive records based on whether they match a user-defined template using Cloud Data Loss Prevention (DLP) APIs.
Null Field Splitter
This plugin sends records with a null value for a given field to the null port and those with a non-null value to the nonnull port.
Sampling
Samples large dataset flowing through this plugin to pull random records using systematic or reservoir sampling.
Top N
Top N returns the top "n" records from the input set, based on the criteria specified in the plugin configuration.

FTP

FTP plugins.
FTP
Batch source to read from FTP (ftp://..) or SFTP (sftp://...).
FTP Copy
Copy files from FTP server to the specified destination.
FTP Put
Copy files to FTP server from a specified destination.
SFTP Copy
SFTP copy allows copying of the files from the specified directory on SFTP servers to the specified destination.
SFTP Delete
CDAP Action plugin to delete the specified files from an SFTP server.
SFTP Put
SFTP put allows uploading of file(s) from filesystem (local/HDFS) to FTP server. File Regex can also be used to filter only the files that are of interest.
Plugins for Google Ads.
Google Ads Multiple Reports
Reads all of the Google Ads reports from a Google Ads account in batch pipelines.
Google Ads Single Report
Reads the specified report type from Google Ads accounts in batch pipelines.
Google Search Console
Reads from Google Search Console in batch pipelines.

Google Analytics 360

Plugins for Google Analytics 360.
Google Analytics Batch
Reads from Google Analytics 360 in batch pipelines.
Google Analytics Streaming
Reads from Google Analytics 360 in streaming pipelines.

Google Workspace

Plugins for Google Workspace including Google Sheets and Google Drive.
Google Drive
Reads files from specified Google Drive directory via Google Drive API.
Google Drive
Sink plugin saves files from the pipeline to Google Drive directory via Google Drive API.
Google Sheets
Reads spreadsheets from specified Google Drive directory.
Google Sheets
Writes spreadsheets to specified Google Drive directory.

Hadoop

Plugins for Hadoop.
HBase
Writes records to an HBase table with one field mapping to the row key, and all other record fields mapping to column qualifiers.
HBase
Batch source that reads from a column family in an HBase table.
Hive Bulk Export
Hive Export will take select query as input to run that query on hive table and store results in the specified location.
Hive Bulk Import
Imports data from a directory/file into a Hive table using Hive Load Statement.
HTTP to HDFS
Action to fetch data from an external http endpoint and create a file in HDFS.

HDFS

Plugins for HDFS.
XML Reader
The XML Reader plugin is a source plugin that allows users to read XML files stored on HDFS.
HTTP to HDFS
Action to fetch data from an external http endpoint and create a file in HDFS.

HP

Plugins for HP.
Vertica Bulk Export Action
Bulk exports data in a Vertica table into a file.
Vertica Bulk Import Action
Vertica Bulk Import Action plugin reads all the files in a given directory and bulk imports the contents into a Vertica table.

HTTP

Plugins for HTTP.
Argument Setter
Performs an HTTP request to fetch arguments to set in the pipeline.

IBM

Plugins for IBM.
DB2
Reads from a DB2 using a configurable SQL query. Outputs one record for each row returned by the query.
DB2
Runs a DB2 query at the end of the pipeline run. Can be configured to run only on success, only on failure, or always at the end of the run.
DB2 Execute
Action that runs a DB2 command.
IBM DB2
Writes records to a DB2 table. Each record will be written to a row in the table.
Netezza
Writes records to a Netezza table. Each record will be written to a row in the table.
Netezza
Reads from a Netezza using a configurable SQL query. Outputs one record for each row returned by the query.
Netezza
Runs a Netezza query at the end of the pipeline run. Can be configured to run only on success, only on failure, or always at the end of the run.
Netezza Execute
Action that runs a Netezza command.

JDBC

Plugins for JDBC.
Database
Reads from a database using a configurable SQL query. Outputs one record for each row returned by the query.
Database
Writes records to a database table. Each record will be written to a row in the table.
MongoDB
Reads documents from a MongoDB collection and converts each document into a StructuredRecord with the help of a specified schema.
MongoDB
This sink writes to a MongoDB collection.
MySQL Query
Runs a MySQL query at the end of the pipeline run. Can be configured to run only on success, only on failure, or always at the end of the run.
MySQL
Writes records to a MySQL table. Each record will be written to a row in the table.
MySQL
Reads from a MySQL instance using a configurable SQL query. Outputs one record for each row returned by the query.
MySQL Execute
Action that runs a MySQL command.
Netezza
Writes records to a Netezza table. Each record will be written to a row in the table.
Netezza
Reads from a Netezza using a configurable SQL query. Outputs one record for each row returned by the query.
Netezza
Runs a Netezza query at the end of the pipeline run. Can be configured to run only on success, only on failure, or always at the end of the run.
Netezza Execute
Action that runs a Netezza command.
Oracle
Reads from an Oracle table using a configurable SQL query. Outputs one record for each row returned by the query.
Oracle
Writes records to an Oracle table. Each record will be written to a row in the table.