Google Cloud Dataflow SDK for Java, version 1.9.1
Class DirectPipelineRunner
- java.lang.Object
-
- com.google.cloud.dataflow.sdk.runners.PipelineRunner<DirectPipelineRunner.EvaluationResults>
-
- com.google.cloud.dataflow.sdk.runners.DirectPipelineRunner
-
public class DirectPipelineRunner extends PipelineRunner<DirectPipelineRunner.EvaluationResults>
Executes the operations in the pipeline directly, in this process, without any optimization. Useful for small local execution and tests.Throws an exception from
run(com.google.cloud.dataflow.sdk.Pipeline)
if execution fails.Permissions
When reading from a Dataflow source or writing to a Dataflow sink usingDirectPipelineRunner
, the Cloud Platform account that you configured with the gcloud executable will need access to the corresponding source/sink.Please see Google Cloud Dataflow Security and Permissions for more details.
-
-
Nested Class Summary
Nested Classes Modifier and Type Class and Description static interface
DirectPipelineRunner.EvaluationContext
The interface provided to registered callbacks for interacting with theDirectPipelineRunner
, including reading and writing the values ofPCollection
s andPCollectionView
s.static interface
DirectPipelineRunner.EvaluationResults
The interface provided to registered callbacks for interacting with theDirectPipelineRunner
, including reading and writing the values ofPCollection
s andPCollectionView
s.static class
DirectPipelineRunner.TestCombineDoFn<K,InputT,AccumT,OutputT>
The implementation may split theCombine.KeyedCombineFn
into ADD, MERGE and EXTRACT phases ( seecom.google.cloud.dataflow.sdk.runners.worker.CombineValuesFn
).static interface
DirectPipelineRunner.TransformEvaluator<TransformT extends PTransform>
An evaluator of a PTransform.static class
DirectPipelineRunner.ValueWithMetadata<V>
An immutable (value, timestamp) pair, along with other metadata necessary for the implementation ofDirectPipelineRunner
.
-
Method Summary
All Methods Static Methods Instance Methods Concrete Methods Modifier and Type Method and Description <OutputT extends POutput,InputT extends PInput>
OutputTapply(PTransform<InputT,OutputT> transform, InputT input)
Applies a transform to the given input, returning the output.static DirectPipelineRunner
createForTest()
Constructs a runner with default properties for testing.static DirectPipelineRunner
fromOptions(PipelineOptions options)
Constructs a DirectPipelineRunner from the given options.DirectPipelineOptions
getPipelineOptions()
Get the options used in thisPipeline
.<TransformT extends PTransform<?,?>>
DirectPipelineRunner.TransformEvaluator<TransformT>getTransformEvaluator(Class<TransformT> transformClass)
Returns the TransformEvaluator to use for instances of the specified PTransform class, or null if none registered.static <TransformT extends PTransform<?,?>>
voidregisterDefaultTransformEvaluator(Class<TransformT> transformClass, DirectPipelineRunner.TransformEvaluator<? super TransformT> transformEvaluator)
Records that instances of the specified PTransform class should be evaluated by default by the corresponding TransformEvaluator.<TransformT extends PTransform<?,?>>
voidregisterTransformEvaluator(Class<TransformT> transformClass, DirectPipelineRunner.TransformEvaluator<TransformT> transformEvaluator)
Records that instances of the specified PTransform class should be evaluated by the corresponding TransformEvaluator.DirectPipelineRunner.EvaluationResults
run(Pipeline pipeline)
Processes the given Pipeline, returning the results.String
toString()
DirectPipelineRunner
withEncodabilityTesting(boolean enable)
Enable runtime testing to verify that all values can be encoded.DirectPipelineRunner
withSerializabilityTesting(boolean enable)
Enable runtime testing to verify that all functions andCoder
instances can be serialized.DirectPipelineRunner
withUnorderednessTesting(boolean enable)
Enable runtime testing to verify that functions do not depend on order of the elements.
-
-
-
Method Detail
-
registerDefaultTransformEvaluator
public static <TransformT extends PTransform<?,?>> void registerDefaultTransformEvaluator(Class<TransformT> transformClass, DirectPipelineRunner.TransformEvaluator<? super TransformT> transformEvaluator)
Records that instances of the specified PTransform class should be evaluated by default by the corresponding TransformEvaluator.
-
registerTransformEvaluator
public <TransformT extends PTransform<?,?>> void registerTransformEvaluator(Class<TransformT> transformClass, DirectPipelineRunner.TransformEvaluator<TransformT> transformEvaluator)
Records that instances of the specified PTransform class should be evaluated by the corresponding TransformEvaluator. Overrides any bindings specified byregisterDefaultTransformEvaluator(java.lang.Class<TransformT>, com.google.cloud.dataflow.sdk.runners.DirectPipelineRunner.TransformEvaluator<? super TransformT>)
.
-
getTransformEvaluator
public <TransformT extends PTransform<?,?>> DirectPipelineRunner.TransformEvaluator<TransformT> getTransformEvaluator(Class<TransformT> transformClass)
Returns the TransformEvaluator to use for instances of the specified PTransform class, or null if none registered.
-
fromOptions
public static DirectPipelineRunner fromOptions(PipelineOptions options)
Constructs a DirectPipelineRunner from the given options.
-
createForTest
public static DirectPipelineRunner createForTest()
Constructs a runner with default properties for testing.- Returns:
- The newly created runner.
-
withSerializabilityTesting
public DirectPipelineRunner withSerializabilityTesting(boolean enable)
Enable runtime testing to verify that all functions andCoder
instances can be serialized.Enabled by default.
This method modifies the
DirectPipelineRunner
instance and returns itself.
-
withEncodabilityTesting
public DirectPipelineRunner withEncodabilityTesting(boolean enable)
Enable runtime testing to verify that all values can be encoded.Enabled by default.
This method modifies the
DirectPipelineRunner
instance and returns itself.
-
withUnorderednessTesting
public DirectPipelineRunner withUnorderednessTesting(boolean enable)
Enable runtime testing to verify that functions do not depend on order of the elements.This is accomplished by randomizing the order of elements.
Enabled by default.
This method modifies the
DirectPipelineRunner
instance and returns itself.
-
apply
public <OutputT extends POutput,InputT extends PInput> OutputT apply(PTransform<InputT,OutputT> transform, InputT input)
Description copied from class:PipelineRunner
Applies a transform to the given input, returning the output.The default implementation calls PTransform.apply(input), but can be overridden to customize behavior for a particular runner.
- Overrides:
apply
in classPipelineRunner<DirectPipelineRunner.EvaluationResults>
-
run
public DirectPipelineRunner.EvaluationResults run(Pipeline pipeline)
Description copied from class:PipelineRunner
Processes the given Pipeline, returning the results.- Specified by:
run
in classPipelineRunner<DirectPipelineRunner.EvaluationResults>
-
getPipelineOptions
public DirectPipelineOptions getPipelineOptions()
Get the options used in thisPipeline
.
-
-