DirectPipelineRunner (Google Cloud Dataflow SDK 1.9.1 API)

Google Cloud Dataflow SDK for Java, version 1.9.1

Warning: Dataflow SDK 1.x for Java is unsupported as of October 16, 2018. After August 12, 2020, you can use unsupported SDKs temporarily while you upgrade your pipelines. For more information, see Using an unsupported SDK.

For information on migrating from the 1.x SDK, see Migrating from Dataflow SDK 1.x for Java.

Note: If you are using Cloud Dataflow SDK version 2.x, see the Apache Beam API reference documentation.

com.google.cloud.dataflow.sdk.runners

Class DirectPipelineRunner

• java.lang.Object
• • com.google.cloud.dataflow.sdk.runners.PipelineRunner<DirectPipelineRunner.EvaluationResults>
  • • com.google.cloud.dataflow.sdk.runners.DirectPipelineRunner

• ---

  
  

  public class DirectPipelineRunner
  extends PipelineRunner<DirectPipelineRunner.EvaluationResults>

  Executes the operations in the pipeline directly, in this process, without any optimization. Useful for small local execution and tests.

  Throws an exception from run(com.google.cloud.dataflow.sdk.Pipeline) if execution fails.

  Permissions

  When reading from a Dataflow source or writing to a Dataflow sink using DirectPipelineRunner, the Cloud Platform account that you configured with the gcloud executable will need access to the corresponding source/sink.

  Please see Google Cloud Dataflow Security and Permissions for more details.

• • Nested Class Summary

    Nested Classes 

    Modifier and Type	Class and Description
    static interface	DirectPipelineRunner.EvaluationContext The interface provided to registered callbacks for interacting with the DirectPipelineRunner, including reading and writing the values of PCollections and PCollectionViews.
    static interface	DirectPipelineRunner.EvaluationResults The interface provided to registered callbacks for interacting with the DirectPipelineRunner, including reading and writing the values of PCollections and PCollectionViews.
    static class	DirectPipelineRunner.TestCombineDoFn<K,InputT,AccumT,OutputT> The implementation may split the Combine.KeyedCombineFn into ADD, MERGE and EXTRACT phases ( see com.google.cloud.dataflow.sdk.runners.worker.CombineValuesFn).
    static interface	DirectPipelineRunner.TransformEvaluator<TransformT extends PTransform> An evaluator of a PTransform.
    static class	DirectPipelineRunner.ValueWithMetadata<V> An immutable (value, timestamp) pair, along with other metadata necessary for the implementation of DirectPipelineRunner.

  • Method Summary

    All Methods Static Methods Instance Methods Concrete Methods 

    Modifier and Type	Method and Description
    <OutputT extends POutput,InputT extends PInput> OutputT	apply(PTransform<InputT,OutputT> transform, InputT input) Applies a transform to the given input, returning the output.
    static DirectPipelineRunner	createForTest() Constructs a runner with default properties for testing.
    static DirectPipelineRunner	fromOptions(PipelineOptions options) Constructs a DirectPipelineRunner from the given options.
    DirectPipelineOptions	getPipelineOptions() Get the options used in this Pipeline.
    <TransformT extends PTransform<?,?>> DirectPipelineRunner.TransformEvaluator<TransformT>	getTransformEvaluator(Class<TransformT> transformClass) Returns the TransformEvaluator to use for instances of the specified PTransform class, or null if none registered.
    static <TransformT extends PTransform<?,?>> void	registerDefaultTransformEvaluator(Class<TransformT> transformClass, DirectPipelineRunner.TransformEvaluator<? super TransformT> transformEvaluator) Records that instances of the specified PTransform class should be evaluated by default by the corresponding TransformEvaluator.
    <TransformT extends PTransform<?,?>> void	registerTransformEvaluator(Class<TransformT> transformClass, DirectPipelineRunner.TransformEvaluator<TransformT> transformEvaluator) Records that instances of the specified PTransform class should be evaluated by the corresponding TransformEvaluator.
    DirectPipelineRunner.EvaluationResults	run(Pipeline pipeline) Processes the given Pipeline, returning the results.
    String	toString()
    DirectPipelineRunner	withEncodabilityTesting(boolean enable) Enable runtime testing to verify that all values can be encoded.
    DirectPipelineRunner	withSerializabilityTesting(boolean enable) Enable runtime testing to verify that all functions and Coder instances can be serialized.
    DirectPipelineRunner	withUnorderednessTesting(boolean enable) Enable runtime testing to verify that functions do not depend on order of the elements.

    • Methods inherited from class java.lang.Object

      clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

• • Method Detail

    • registerDefaultTransformEvaluator

      public static <TransformT extends PTransform<?,?>> void registerDefaultTransformEvaluator(Class<TransformT> transformClass,
                                                                                                DirectPipelineRunner.TransformEvaluator<? super TransformT> transformEvaluator)

      Records that instances of the specified PTransform class should be evaluated by default by the corresponding TransformEvaluator.

    • registerTransformEvaluator

      public <TransformT extends PTransform<?,?>> void registerTransformEvaluator(Class<TransformT> transformClass,
                                                                                  DirectPipelineRunner.TransformEvaluator<TransformT> transformEvaluator)

      Records that instances of the specified PTransform class should be evaluated by the corresponding TransformEvaluator. Overrides any bindings specified by registerDefaultTransformEvaluator(java.lang.Class<TransformT>, com.google.cloud.dataflow.sdk.runners.DirectPipelineRunner.TransformEvaluator<? super TransformT>).

    • getTransformEvaluator

      public <TransformT extends PTransform<?,?>> DirectPipelineRunner.TransformEvaluator<TransformT> getTransformEvaluator(Class<TransformT> transformClass)

      Returns the TransformEvaluator to use for instances of the specified PTransform class, or null if none registered.

    • fromOptions

      public static DirectPipelineRunner fromOptions(PipelineOptions options)

      Constructs a DirectPipelineRunner from the given options.

    • createForTest

      public static DirectPipelineRunner createForTest()

      Constructs a runner with default properties for testing.

      Returns:

      The newly created runner.

    • withSerializabilityTesting

      public DirectPipelineRunner withSerializabilityTesting(boolean enable)

      Enable runtime testing to verify that all functions and Coder instances can be serialized.

      Enabled by default.

      This method modifies the DirectPipelineRunner instance and returns itself.

    • withEncodabilityTesting

      public DirectPipelineRunner withEncodabilityTesting(boolean enable)

      Enable runtime testing to verify that all values can be encoded.

      Enabled by default.

      This method modifies the DirectPipelineRunner instance and returns itself.

    • withUnorderednessTesting

      public DirectPipelineRunner withUnorderednessTesting(boolean enable)

      Enable runtime testing to verify that functions do not depend on order of the elements.

      This is accomplished by randomizing the order of elements.

      Enabled by default.

      This method modifies the DirectPipelineRunner instance and returns itself.

    • apply

      public <OutputT extends POutput,InputT extends PInput> OutputT apply(PTransform<InputT,OutputT> transform,
                                                                           InputT input)

      Description copied from class: PipelineRunner
      Applies a transform to the given input, returning the output.

      The default implementation calls PTransform.apply(input), but can be overridden to customize behavior for a particular runner.

      Overrides:

      apply in class PipelineRunner<DirectPipelineRunner.EvaluationResults>

    • run

      public DirectPipelineRunner.EvaluationResults run(Pipeline pipeline)

      Description copied from class: PipelineRunner
      Processes the given Pipeline, returning the results.

      Specified by:

      run in class PipelineRunner<DirectPipelineRunner.EvaluationResults>

    • getPipelineOptions

      public DirectPipelineOptions getPipelineOptions()

      Get the options used in this Pipeline.

    • toString

      public String toString()

      Overrides:

      toString in class Object