After you have finished developing the recipes in your flow, you can define scheduled executions of the recipe or recipes within the flow to deliver outputs to known locations. Using the Automator, you can automate execution of jobs on source data, which can be replenished with fresh data asynchronously.
NOTE: Before you begin, you should verify that your data management pipeline into and out of the platform has been appropriately defined. This pipeline includes how data is written to the output location. For more information, see Overview of Automator.
When you schedule a job, you create two objects in Flow View:
NOTE: You must create both of these objects to schedule a job execution.
|Schedule||A schedule applies to the entire flow. It contains one or more intervals at which the recipes of the flow are executed. Recipes are executed if their outputs include a scheduled destination.|
|Scheduled destination||A scheduled destination is an output location, format, and other settings that is populated with the results of executing the related recipes.|
Tip: You can create schedules for datasets with parameters. Any overrides specified through Flow View are automatically applied at runtime for the scheduled job. See Flow View Page.
Notes on scheduled Cloud Dataflow jobs:
Add a Schedule
- Open the flow in Flow View.
- From the context menu, select Add Schedule....
- In the Add Schedule dialog, select your scheduling options:
- Timezone: Select the timezone to use to determine when to execute the specified schedule.
Frequency: Select the time and frequency of execution: Hourly, Daily, Weekly, Monthly, or cron.
NOTE: Scheduling supports a modified version of cron scheduling syntax. For more information, see cron Schedule Syntax Reference.
To add another scheduled time, click Add.
- To save your schedule, click Save.
- A Calendar icon appears in Flow View to indicate that the flow has a schedule associated with it.
For more information, see Add Schedule Dialog.
Schedule a Destination
- To specify a destination for your schedule, click the recipe you wish to execute at the scheduled time.
- If you have not done so already, click the Output icon next to the recipe to create an output for it.
- In the right panel, locate the Scheduled destinations header. Click Add.
Specify an output location, format, and updating method.
For more information, see Run Job Page.
- To edit the scheduled times, click the Calendar icon. Then, click Edit. Make changes as needed and save.
- To edit a scheduled destination, select the output in Flow View. In the right panel, click Edit next to the appropriate scheduled destination.
Delete a schedule
In Flow View, click the Calendar icon. Then, click Delete Schedule.
Delete a destination
Tip: If you have deleted the schedule for the flow, you do not need to delete the scheduled destination. It cannot run without a schedule.
- In Flow View, select the output.
- In the right panel, select Delete Output from the context menu.