Description of the composing transforms, names/ids, and input/outputs of a stage of execution. Some composing transforms and sources may have been generated by the Dataflow service during execution planning.

ExecutionStageSummary.Types

Container for nested types declared in the ExecutionStageSummary message type.

ExecutionStageSummary.Types.ComponentSource

Description of an interstitial value between transforms in an execution stage.

ExecutionStageSummary.Types.ComponentTransform

Description of a transform executed as part of an execution stage.

ExecutionStageSummary.Types.StageSource

Description of an input or output of an execution stage.

FailedLocation

Indicates which regional endpoint failed to respond to a request for data.

FileIODetails

Metadata for a File connector used by the job.

FlexTemplateRuntimeEnvironment

The environment values to be set at runtime for flex template.

FlexTemplatesService

Provides a service for Flex templates. This feature is not ready yet.

FlexTemplatesService.FlexTemplatesServiceBase

Base class for server-side implementations of FlexTemplatesService

FlexTemplatesService.FlexTemplatesServiceClient

Client for FlexTemplatesService

FlexTemplatesServiceClient

FlexTemplatesService client wrapper, for convenient use.

FlexTemplatesServiceClientBuilder

Builder class for FlexTemplatesServiceClient to provide simple configuration of credentials, endpoint etc.

FlexTemplatesServiceClientImpl

FlexTemplatesService client wrapper implementation, for convenient use.

FlexTemplatesServiceSettings

Settings for FlexTemplatesServiceClient instances.

GetJobExecutionDetailsRequest

Request to get job execution details.

GetJobMetricsRequest

Request to get job metrics.

GetJobRequest

Request to get the state of a Cloud Dataflow job.

GetSnapshotRequest

Request to get information about a snapshot

GetStageExecutionDetailsRequest

Request to get information about a particular execution stage of a job. Currently only tracked for Batch jobs.

GetTemplateRequest

A request to retrieve a Cloud Dataflow job template.

GetTemplateRequest.Types

Container for nested types declared in the GetTemplateRequest message type.

GetTemplateResponse

The response to a GetTemplate request.

GetTemplateResponse.Types

Container for nested types declared in the GetTemplateResponse message type.

InvalidTemplateParameters

Used in the error_details field of a google.rpc.Status message, this indicates problems with the template parameter.

InvalidTemplateParameters.Types

Container for nested types declared in the InvalidTemplateParameters message type.

InvalidTemplateParameters.Types.ParameterViolation

A specific template-parameter violation.

Job

Defines a job to be run by the Cloud Dataflow service.

JobExecutionDetails

Information about the execution of a job.

JobExecutionInfo

Additional information about how a Cloud Dataflow job will be executed that isn't contained in the submitted job.

JobExecutionStageInfo

Contains information about how a particular [google.dataflow.v1beta3.Step][google.dataflow.v1beta3.Step] will be executed.

JobMessage

A particular message pertaining to a Dataflow job.

JobMetadata

Metadata available primarily for filtering jobs. Will be included in the ListJob response and Job SUMMARY view.

JobMetrics

JobMetrics contains a collection of metrics describing the detailed progress of a Dataflow job. Metrics correspond to user-defined and system-defined metrics in the job.

This resource captures only the most recent values of each metric; time-series data can be queried for them (under the same metric names) from Cloud Monitoring.

JobsV1Beta3

Provides a method to create and modify Google Cloud Dataflow jobs. A Job is a multi-stage computation graph run by the Cloud Dataflow service.

JobsV1Beta3.JobsV1Beta3Base

Base class for server-side implementations of JobsV1Beta3

JobsV1Beta3.JobsV1Beta3Client

Client for JobsV1Beta3

JobsV1Beta3Client

JobsV1Beta3 client wrapper, for convenient use.

JobsV1Beta3ClientBuilder

Builder class for JobsV1Beta3Client to provide simple configuration of credentials, endpoint etc.

JobsV1Beta3ClientImpl

JobsV1Beta3 client wrapper implementation, for convenient use.

JobsV1Beta3Settings

Settings for JobsV1Beta3Client instances.

KeyRangeDataDiskAssignment

Data disk assignment information for a specific key-range of a sharded computation. Currently we only support UTF-8 character splits to simplify encoding into JSON.

KeyRangeLocation

Location information for a specific key-range of a sharded computation. Currently we only support UTF-8 character splits to simplify encoding into JSON.

LaunchFlexTemplateParameter

Launch FlexTemplate Parameter.

LaunchFlexTemplateRequest

A request to launch a Cloud Dataflow job from a FlexTemplate.

LaunchFlexTemplateResponse

Response to the request to launch a job from Flex Template.

LaunchTemplateParameters

Parameters to provide to the template being launched.

LaunchTemplateRequest

A request to launch a template.

LaunchTemplateResponse

Response to the request to launch a template.

ListJobMessagesRequest

Request to list job messages. Up to max_results messages will be returned in the time range specified starting with the oldest messages first. If no time range is specified the results with start with the oldest message.

ListJobMessagesResponse

Response to a request to list job messages.

ListJobsRequest

Request to list Cloud Dataflow jobs.

ListJobsRequest.Types

Container for nested types declared in the ListJobsRequest message type.

ListJobsResponse

Response to a request to list Cloud Dataflow jobs in a project. This might be a partial response, depending on the page size in the ListJobsRequest. However, if the project does not have any jobs, an instance of ListJobsResponse is not returned and the requests's response body is empty {}.

ListSnapshotsRequest

Request to list snapshots.

ListSnapshotsResponse

List of snapshots.

MessagesV1Beta3

The Dataflow Messages API is used for monitoring the progress of Dataflow jobs.

MessagesV1Beta3.MessagesV1Beta3Base

Base class for server-side implementations of MessagesV1Beta3

MessagesV1Beta3.MessagesV1Beta3Client

Client for MessagesV1Beta3

MessagesV1Beta3Client

MessagesV1Beta3 client wrapper, for convenient use.

MessagesV1Beta3ClientBuilder

Builder class for MessagesV1Beta3Client to provide simple configuration of credentials, endpoint etc.

MessagesV1Beta3ClientImpl

MessagesV1Beta3 client wrapper implementation, for convenient use.

MessagesV1Beta3Settings

Settings for MessagesV1Beta3Client instances.

MetricStructuredName

Identifies a metric, by describing the source which generated the metric.

MetricUpdate

Describes the state of a metric.

MetricsV1Beta3

The Dataflow Metrics API lets you monitor the progress of Dataflow jobs.

MetricsV1Beta3.MetricsV1Beta3Base

Base class for server-side implementations of MetricsV1Beta3

MetricsV1Beta3.MetricsV1Beta3Client

Client for MetricsV1Beta3

MetricsV1Beta3Client

MetricsV1Beta3 client wrapper, for convenient use.

MetricsV1Beta3ClientBuilder

Builder class for MetricsV1Beta3Client to provide simple configuration of credentials, endpoint etc.

MetricsV1Beta3ClientImpl

MetricsV1Beta3 client wrapper implementation, for convenient use.

MetricsV1Beta3Settings

Settings for MetricsV1Beta3Client instances.

MountedDataDisk

Describes mounted data disk.

Package

The packages that must be installed in order for a worker to run the steps of the Cloud Dataflow job that will be assigned to its worker pool.

This is the mechanism by which the Cloud Dataflow SDK causes code to be loaded onto the workers. For example, the Cloud Dataflow Java SDK might use this to install jars containing the user's code and all of the various dependencies (libraries, data files, etc.) required in order for that code to run.

ParameterMetadata

Metadata for a specific parameter.

PipelineDescription

A descriptive representation of submitted pipeline as well as the executed form. This data is provided by the Dataflow service for ease of visualizing the pipeline and interpreting Dataflow provided metrics.

ProgressTimeseries

Information about the progress of some component of job execution.

ProgressTimeseries.Types

Container for nested types declared in the ProgressTimeseries message type.

ProgressTimeseries.Types.Point

A point in the timeseries.

PubSubIODetails

Metadata for a Pub/Sub connector used by the job.

PubsubLocation

Identifies a pubsub location to use for transferring data into or out of a streaming Dataflow job.

PubsubSnapshotMetadata

Represents a Pubsub snapshot.

RuntimeEnvironment

The environment values to set at runtime.

RuntimeMetadata

RuntimeMetadata describing a runtime environment.

SDKInfo

SDK Information.

SDKInfo.Types

Container for nested types declared in the SDKInfo message type.

SdkHarnessContainerImage

Defines a SDK harness container for executing Dataflow pipelines.

SdkVersion

The version of the SDK used to run the job.

SdkVersion.Types

Container for nested types declared in the SdkVersion message type.

Snapshot

Represents a snapshot of a job.

SnapshotJobRequest

Request to create a snapshot of a job.

SnapshotsV1Beta3

Provides methods to manage snapshots of Google Cloud Dataflow jobs.

SnapshotsV1Beta3.SnapshotsV1Beta3Base

Base class for server-side implementations of SnapshotsV1Beta3

SnapshotsV1Beta3.SnapshotsV1Beta3Client

Client for SnapshotsV1Beta3

SnapshotsV1Beta3Client

SnapshotsV1Beta3 client wrapper, for convenient use.

SnapshotsV1Beta3ClientBuilder

Builder class for SnapshotsV1Beta3Client to provide simple configuration of credentials, endpoint etc.

SnapshotsV1Beta3ClientImpl

SnapshotsV1Beta3 client wrapper implementation, for convenient use.

SnapshotsV1Beta3Settings

Settings for SnapshotsV1Beta3Client instances.

SpannerIODetails

Metadata for a Spanner connector used by the job.

StageExecutionDetails

Information about the workers and work items within a stage.

StageSummary

Information about a particular execution stage of a job.

StateFamilyConfig

State family configuration.

Step

Defines a particular step within a Cloud Dataflow job.

A job consists of multiple steps, each of which performs some specific operation as part of the overall job. Data is typically passed from one step to another as part of the job.

Here's an example of a sequence of steps which together implement a Map-Reduce job:

Read a collection of data from some source, parsing the collection's elements.
Validate the elements.
Apply a user-defined function to map each element to some value and extract an element-specific key value.
Group elements with the same key into a single element with that key, transforming a multiply-keyed collection into a uniquely-keyed collection.
Write the elements out to some data sink.

Note that the Cloud Dataflow service may be used to run many different types of jobs, not just Map-Reduce.

StreamLocation

Describes a stream of data, either as input to be processed or as output of a streaming Dataflow job.

StreamingApplianceSnapshotConfig

Streaming appliance snapshot configuration.

StreamingComputationRanges

Describes full or partial data disk assignment information of the computation ranges.

StreamingSideInputLocation

Identifies the location of a streaming side input.

StreamingStageLocation

Identifies the location of a streaming computation stage, for stage-to-stage communication.

StructuredMessage

A rich message format, including a human readable string, a key for identifying the message, and structured data associated with the message for programmatic consumption.

StructuredMessage.Types

Container for nested types declared in the StructuredMessage message type.

StructuredMessage.Types.Parameter

Structured data associated with this message.

TaskRunnerSettings

Taskrunner configuration settings.

TemplateMetadata

Metadata describing a template.

TemplatesService

Provides a method to create Cloud Dataflow jobs from templates.

TemplatesService.TemplatesServiceBase

Base class for server-side implementations of TemplatesService

TemplatesService.TemplatesServiceClient

Client for TemplatesService

TemplatesServiceClient

TemplatesService client wrapper, for convenient use.

TemplatesServiceClientBuilder

Builder class for TemplatesServiceClient to provide simple configuration of credentials, endpoint etc.

TemplatesServiceClientImpl

TemplatesService client wrapper implementation, for convenient use.

TemplatesServiceSettings

Settings for TemplatesServiceClient instances.

TopologyConfig

Global topology of the streaming Dataflow job, including all computations and their sharded locations.

TransformSummary

Description of the type, names/ids, and input/outputs for a transform.

UpdateJobRequest

Request to update a Cloud Dataflow job.

WorkItemDetails

Information about an individual work item execution.

WorkerDetails

Information about a worker

WorkerPool

Describes one particular pool of Cloud Dataflow workers to be instantiated by the Cloud Dataflow service in order to perform the computations required by a job. Note that a workflow job may use multiple pools, in order to match the various computational requirements of the various stages of the job.

WorkerSettings

Provides data to pass through to the worker harness.