Schedule a Job

After you have finished developing the recipes in your flow, you can define scheduled executions of the recipe or recipes within the flow to deliver outputs to known locations. Using the Automator, you can automate execution of jobs on source data, which can be replenished with fresh data asynchronously.

NOTE: Before you begin, you should verify that your data management pipeline into and out of the platform has been appropriately defined. This pipeline includes how data is written to the output location. For more information, see Overview of Automator.

When you schedule a job, you create two objects in Flow View:

NOTE: You must create both of these objects to schedule a job execution.

ObjectDescription
ScheduleA schedule applies to the entire flow. It contains one or more intervals at which the recipes of the flow are executed. Recipes are executed if their outputs include a scheduled destination.
Scheduled destinationA scheduled destination is an output location, format, and other settings that is populated with the results of executing the related recipes.

Tip: You can create schedules for datasets with parameters. Any overrides specified through Flow View are automatically applied at runtime for the scheduled job. See Flow View Page.

NOTE: When a scheduled job is executed, no Cloud Dataflow template is created.

Add a Schedule

Steps:

  1. Open the flow in Flow View.
  2. From the context menu, select Add Schedule....
  3. In the Add Schedule dialog, select your scheduling options:
    1. Timezone: Select the timezone to use to determine when to execute the specified schedule.
    2. Frequency: Select the time and frequency of execution: Hourly, Daily, Weekly, Monthly, or cron.

      NOTE: Scheduling supports a modified version of cron scheduling syntax. For more information, see cron Schedule Syntax Reference.

    3. To add another scheduled time, click Add.

  4. To save your schedule, click Save.
  5. A Calendar icon appears in Flow View to indicate that the flow has a schedule associated with it.

For more information, see Add Schedule Dialog.

Schedule a Destination

Steps:

  1. To specify a destination for your schedule, click the recipe you wish to execute at the scheduled time.
  2. If you have not done so already, click the Output icon next to the recipe to create an output for it.
  3. In the right panel, locate the Scheduled destinations header. Click Add.
  4. Specify an output location, format, and updating method.

  5. Click Save.

For more information, see Run Job Page.

Edit Schedule

  • To edit the scheduled times, click the Calendar icon. Then, click Edit. Make changes as needed and save.
  • To edit a scheduled destination, select the output in Flow View. In the right panel, click Edit next to the appropriate scheduled destination.

Delete

Delete a schedule

In Flow View, click the Calendar icon. Then, click Delete Schedule.

Delete a destination

Tip: If you have deleted the schedule for the flow, you do not need to delete the scheduled destination. It cannot run without a schedule.

  1. In Flow View, select the output.
  2. In the right panel, select Delete Output from the context menu.
Was this page helpful? Let us know how we did:

Send feedback about...

Google Cloud Dataprep Documentation
Need help? Visit our support page.