Google Cloud Dataflow SDK for Java, version 1.9.1
com.google.cloud.dataflow.sdk.runners
Class DataflowPipelineRunner
- java.lang.Object
-
- com.google.cloud.dataflow.sdk.runners.PipelineRunner<DataflowPipelineJob>
-
- com.google.cloud.dataflow.sdk.runners.DataflowPipelineRunner
-
public class DataflowPipelineRunner extends PipelineRunner<DataflowPipelineJob>
APipelineRunner
that executes the operations in the pipeline by first translating them to the Dataflow representation using theDataflowPipelineTranslator
and then submitting them to a Dataflow service for execution.Permissions
When reading from a Dataflow source or writing to a Dataflow sink usingDataflowPipelineRunner
, the Google cloudservices account and the Google compute engine service account of the GCP project running the Dataflow Job will need access to the corresponding source/sink.Please see Google Cloud Dataflow Security and Permissions for more details.
-
-
Nested Class Summary
Nested Classes Modifier and Type Class and Description static class
DataflowPipelineRunner.BatchBigQueryIONativeReadTranslator
Implements BigQueryIO Read translation for the Dataflow backend.
-
Field Summary
Fields Modifier and Type Field and Description static String
BATCH_WORKER_HARNESS_CONTAINER_IMAGE
static String
PROJECT_ID_REGEXP
Project IDs must contain lowercase letters, digits, or dashes.static String
STREAMING_WORKER_HARNESS_CONTAINER_IMAGE
-
Constructor Summary
Constructors Modifier Constructor and Description protected
DataflowPipelineRunner(DataflowPipelineOptions options)
-
Method Summary
All Methods Static Methods Instance Methods Concrete Methods Modifier and Type Method and Description <OutputT extends POutput,InputT extends PInput>
OutputTapply(PTransform<InputT,OutputT> transform, InputT input)
Applies the given transform to the input.protected static List<String>
detectClassPathResourcesToStage(ClassLoader classLoader)
Attempts to detect all the resources the class loader has access to.static DataflowPipelineRunner
fromOptions(PipelineOptions options)
Construct a runner from the provided options.DataflowPipelineTranslator
getTranslator()
Returns the DataflowPipelineTranslator associated with this object.DataflowPipelineJob
run(Pipeline pipeline)
Processes the given Pipeline, returning the results.void
setHooks(DataflowPipelineRunnerHooks hooks)
Sets callbacks to invoke during execution seeDataflowPipelineRunnerHooks
.String
toString()
-
-
-
Field Detail
-
BATCH_WORKER_HARNESS_CONTAINER_IMAGE
public static final String BATCH_WORKER_HARNESS_CONTAINER_IMAGE
- See Also:
- Constant Field Values
-
STREAMING_WORKER_HARNESS_CONTAINER_IMAGE
public static final String STREAMING_WORKER_HARNESS_CONTAINER_IMAGE
- See Also:
- Constant Field Values
-
PROJECT_ID_REGEXP
public static final String PROJECT_ID_REGEXP
Project IDs must contain lowercase letters, digits, or dashes. IDs must start with a letter and may not end with a dash. This regex isn't exact - this allows for patterns that would be rejected by the service, but this is sufficient for basic validation of project IDs.- See Also:
- Constant Field Values
-
-
Constructor Detail
-
DataflowPipelineRunner
protected DataflowPipelineRunner(DataflowPipelineOptions options)
-
-
Method Detail
-
fromOptions
public static DataflowPipelineRunner fromOptions(PipelineOptions options)
Construct a runner from the provided options.- Parameters:
options
- Properties that configure the runner.- Returns:
- The newly created runner.
-
apply
public <OutputT extends POutput,InputT extends PInput> OutputT apply(PTransform<InputT,OutputT> transform, InputT input)
Applies the given transform to the input. For transforms with customized definitions for the Dataflow pipeline runner, the application is intercepted and modified here.- Overrides:
apply
in classPipelineRunner<DataflowPipelineJob>
-
run
public DataflowPipelineJob run(Pipeline pipeline)
Description copied from class:PipelineRunner
Processes the given Pipeline, returning the results.- Specified by:
run
in classPipelineRunner<DataflowPipelineJob>
-
getTranslator
public DataflowPipelineTranslator getTranslator()
Returns the DataflowPipelineTranslator associated with this object.
-
setHooks
@Experimental public void setHooks(DataflowPipelineRunnerHooks hooks)
Sets callbacks to invoke during execution seeDataflowPipelineRunnerHooks
.
-
detectClassPathResourcesToStage
protected static List<String> detectClassPathResourcesToStage(ClassLoader classLoader)
Attempts to detect all the resources the class loader has access to. This does not recurse to class loader parents stopping it from pulling in resources from the system class loader.- Parameters:
classLoader
- The URLClassLoader to use to detect resources to stage.- Returns:
- A list of absolute paths to the resources the class loader uses.
- Throws:
IllegalArgumentException
- If either the class loader is not a URLClassLoader or one of the resources the class loader exposes is not a file resource.
-
-