Overview of Operationalization

Operationalization refers to a general class of platform features that enable repeated application of Cloud Dataprep by TRIFACTA® INC. on production data. Whether deployed in a single flow or across all flows in your environment, operationalization features broaden the scope of wrangled data, simplify job execution, and enable these processes on a repeated or scheduled basis.

In the following sections, you can review short summaries of specific features and explore more detailed information on them.

Single-Flow Operations

These features can be applied to individual flows to simplify job execution.

Parameterization

Parameterization enables you to specify parameters that capture variability in your data source paths or names. For example, you can parameterize the names of folders in your filepaths to capture files within multiple folders. Or, you can parameterize your inputs to capture datasets named within a specific time range. Nested folders of data can be parameterized, too.

Parameter types:

  • dataset parameters: Parameterize the input paths to your data, allowing you to process data in parallel files and tables through the same flow.
  • output parameters: Parameterize the output paths for your results.
  • flow parameters: Define parameters that can be applied in your flows, including recipe steps.

    Tip: You can apply overrides to any parameter at the flow level. These parameter override values are applied to any parameter that is referenced within the flow for any supported parameter type.

Parameter formats:

NOTE: Some of the following may not be available in your product edition.

Parameter TypeDescription
Pattern

Use regular expressions or Cloud Dataprep patterns in your paths or queries to sources to capture a broader set of inputs.

WildcardReplace parts of your paths or queries with wildcards.
DatetimeYou can specify parameterized Datetime values in one of the supported formats.
VariableVariable values can be specified as overrides during import, job execution, and output.

Parameterization is available for the following:

File systems

Input

Output

Date/time

Pattern

Variable

Timestamp

Variable

For more information, see Overview of Parameterization.

Scheduling

The scheduling feature, also known as Automator, enables you to schedule the execution of individual flows on a specified frequency. Frequencies can be specified through the Cloud Dataprep application through a simple interface or, if needed, in a modified form of cron syntax.

Tip: Automator is often used with parameterization to fully automate data preparation processes in Cloud Dataprep by TRIFACTA INC..

For more information, see Overview of Automator.

Job monitoring

After a job has been launched, detailed monitoring permits you to track the progress of your job during all phases of execution. Status, job stats, inputs, outputs and a flow snapshot are available through the Cloud Dataprep application.For more information, see Overview of Job Monitoring.