Preview

This product or feature is covered by the Pre-GA Offerings Terms of the Google Cloud Terms of Service. Pre-GA products and features might have limited support, and changes to pre-GA products and features might not be compatible with other pre-GA versions. For more information, see the launch stage descriptions.

See the supported connectors for Application Integration.

Dataflow - Create Job task

The Dataflow - Create Job task lets you create a job in Cloud Dataflow to run a data pipeline built using one of the Apache Beam SDKs.

Cloud Dataflow is a fully managed Google Cloud service for running stream and Batch data processing pipelines.

Before you begin

Ensure that you perform the following tasks in your Google Cloud project before configuring the Dataflow - Create Job task:

1. Enable the Dataflow API (dataflow.googleapis.com).

   Enable the Dataflow API

2. Create an authentication profile. Application Integration uses an authentication profile to connect to an authentication endpoint for the Dataflow - Create Job task.

   For information about granting additional roles or permissions to a service account, see Granting, changing, and revoking access.

Configure the Dataflow - Create Job task

1. In the Google Cloud console, go to the Application Integration page.

   Go to Application Integration

2. In the navigation menu, click Integrations.

   The Integrations List page appears.

3. Select an existing integration or create a new integration by clicking Create integration.

   If you are creating a new integration:

   1. Enter a name and description in the Create Integration dialog.
   2. Select a Region for the integration from the list of supported regions.
   3. Click Create.

   This opens the integration in the integration designer.

4. In the integration designer navigation bar, click +Add a task/trigger > Tasks to view the list of available tasks.
5. Click and place the Dataflow - Create Job element in the integration designer.
6. Click the Dataflow - Create Job element on the designer to view the Dataflow - Create Job task configuration pane.
7. Go to Authentication, and select an existing authentication profile that you want to use.

   Optional. If you have not created an authentication profile prior to configuring the task, Click + New authentication profile and follow the steps as mentioned in Create a new authentication profile.

8. Go to Task Input, and configure the displayed inputs fields using the following Task input parameters table.

   Changes to the inputs fields are saved automatically.

Task input parameters

The following table describes the input parameters of the Dataflow - Create Job task:

Property	Data type	Description
Region	String	Cloud Dataflow location for the job.
ProjectsId	String	Your Google Cloud project ID.
Location	String	The regional endpoint that contains this job.
Request	JSON	See request JSON structure.

Task output

The Dataflow - Create Job task returns the newly created instance of the Job.

Retry on failure

You can configure various retry strategies to handle errors in a task. The retry strategies allow you to specify how to rerun the task or integration in case of an error. For more information, see Error handling strategies.

What's next

1. Add edges and edge conditions.
2. Test and publish your integration.
3. Configure a trigger.
4. Add a Data Mapping task.
5. See all tasks for Google Cloud services.