Join the Apache Beam community on July 18th-20th for the Beam Summit 2022 to learn more about Beam and share your expertise.

Method: projects.locations.pipelines.run

Creates a job for the specified pipeline directly. You can use this method when the internal scheduler is not configured and you want to trigger the job directly or through an external system.

Returns a "NOT_FOUND" error if the pipeline doesn't exist. Returns a "FORBIDDEN" error if the user doesn't have permission to access the pipeline or run jobs for the pipeline.

HTTP request

POST https://datapipelines.googleapis.com/v1/{name=projects/*/locations/*/pipelines/*}:run

The URL uses gRPC Transcoding syntax.

Path parameters

Parameters
name

string

Required. The pipeline name. For example: projects/PROJECT_ID/locations/LOCATION_ID/pipelines/PIPELINE_ID.

Request body

The request body must be empty.

Response body

If successful, the response body contains data with the following structure:

Response message for pipelines.run

JSON representation
{
  "job": {
    object (Job)
  }
}
Fields
job

object (Job)

Job that was created as part of pipelines.run operation.

Authorization Scopes

Requires the following OAuth scope:

  • https://www.googleapis.com/auth/cloud-platform

For more information, see the Authentication Overview.

IAM Permissions

Requires the following IAM permission on the name resource:

  • datapipelines.pipelines.run

For more information, see the IAM documentation.

Job

Definition of the job information maintained by the pipeline. Fields in this entity are retrieved from the executor API (e.g. Dataflow API).

JSON representation
{
  "name": string,
  "id": string,
  "createTime": string,
  "endTime": string,
  "state": enum (State),
  "status": {
    object (Status)
  },
  "dataflowJobDetails": {
    object (DataflowJobDetails)
  }
}
Fields
name

string

Required. The fully qualified resource name for the job.

id

string

Output only. The internal ID for the job.

createTime

string (Timestamp format)

Output only. The time of job creation.

A timestamp in RFC3339 UTC "Zulu" format, with nanosecond resolution and up to nine fractional digits. Examples: "2014-10-02T15:01:23Z" and "2014-10-02T15:01:23.045123456Z".

endTime

string (Timestamp format)

Output only. The time of job termination. This is absent if the job is still running.

A timestamp in RFC3339 UTC "Zulu" format, with nanosecond resolution and up to nine fractional digits. Examples: "2014-10-02T15:01:23Z" and "2014-10-02T15:01:23.045123456Z".

state

enum (State)

The current state of the job.

status

object (Status)

Status capturing any error code or message related to job creation or execution.

dataflowJobDetails

object (DataflowJobDetails)

All the details that are specific to a Dataflow job.

State

Enum lisitng all the job execution states.

Enums
STATE_UNSPECIFIED The job state isn't specified.
STATE_PENDING The job is waiting to start execution.
STATE_RUNNING The job is executing.
STATE_DONE The job has finished execution successfully.
STATE_FAILED The job has finished execution with a failure.
STATE_CANCELLED The job has been terminated upon user request.

Status

The Status type defines a logical error model that is suitable for different programming environments, including REST APIs and RPC APIs. It is used by gRPC. Each Status message contains three pieces of data: error code, error message, and error details.

You can find out more about this error model and how to work with it in the API Design Guide.

JSON representation
{
  "code": integer,
  "message": string,
  "details": [
    {
      "@type": string,
      field1: ...,
      ...
    }
  ]
}
Fields
code

integer

The status code, which should be an enum value of google.rpc.Code.

message

string

A developer-facing error message, which should be in English. Any user-facing error message should be localized and sent in the google.rpc.Status.details field, or localized by the client.

details[]

object

A list of messages that carry the error details. There is a common set of message types for APIs to use.

An object containing fields of an arbitrary type. An additional field "@type" contains a URI identifying the type. Example: { "id": 1234, "@type": "types.example.com/standard/id" }.

DataflowJobDetails

Pipeline job details specific to the Dataflow API. This is encapsulated here to allow for more executors to store their specific details separately.

JSON representation
{
  "sdkVersion": {
    object (SdkVersion)
  },
  "currentWorkers": integer,
  "resourceInfo": {
    string: number,
    ...
  }
}
Fields
sdkVersion

object (SdkVersion)

Output only. The SDK version used to run the job.

currentWorkers

integer

Output only. The current number of workers used to run the jobs. Only set to a value if the job is still running.

resourceInfo

map (key: string, value: number)

Cached version of all the metrics of interest for the job. This value gets stored here when the job is terminated. As long as the job is running, this field is populated from the Dataflow API.

An object containing a list of "key": value pairs. Example: { "name": "wrench", "mass": "1.3kg", "count": "3" }.

SdkVersion

The version of the SDK used to run the job.

JSON representation
{
  "version": string,
  "versionDisplayName": string,
  "sdkSupportStatus": enum (SdkSupportStatus)
}
Fields
version

string

The version of the SDK used to run the job.

versionDisplayName

string

A readable string describing the version of the SDK.

sdkSupportStatus

enum (SdkSupportStatus)

The support status for this SDK version.

SdkSupportStatus

The support status of the SDK used to run the job.

Enums
UNKNOWN Dataflow is unaware of this version.
SUPPORTED This is a known version of an SDK, and is supported.
STALE A newer version of the SDK exists, and an update is recommended.
DEPRECATED This version of the SDK is deprecated and will eventually be unsupported.
UNSUPPORTED Support for this SDK version has ended and it should no longer be used.