DataflowPipelineRunner (Google Cloud Dataflow SDK 1.9.1 API)

Google Cloud Dataflow SDK for Java, version 1.9.1

Warning: Dataflow SDK 1.x for Java is unsupported as of October 16, 2018. After August 12, 2020, you can use unsupported SDKs temporarily while you upgrade your pipelines. For more information, see Using an unsupported SDK.

For information on migrating from the 1.x SDK, see Migrating from Dataflow SDK 1.x for Java.

Note: If you are using Cloud Dataflow SDK version 2.x, see the Apache Beam API reference documentation.

com.google.cloud.dataflow.sdk.runners

Class DataflowPipelineRunner

• java.lang.Object
• • com.google.cloud.dataflow.sdk.runners.PipelineRunner<DataflowPipelineJob>
  • • com.google.cloud.dataflow.sdk.runners.DataflowPipelineRunner

• ---

  
  

  public class DataflowPipelineRunner
  extends PipelineRunner<DataflowPipelineJob>

  A PipelineRunner that executes the operations in the pipeline by first translating them to the Dataflow representation using the DataflowPipelineTranslator and then submitting them to a Dataflow service for execution.

  Permissions

  When reading from a Dataflow source or writing to a Dataflow sink using DataflowPipelineRunner, the Google cloudservices account and the Google compute engine service account of the GCP project running the Dataflow Job will need access to the corresponding source/sink.

  Please see Google Cloud Dataflow Security and Permissions for more details.

• • Nested Class Summary

    Nested Classes 

    Modifier and Type	Class and Description
    static class	DataflowPipelineRunner.BatchBigQueryIONativeReadTranslator Implements BigQueryIO Read translation for the Dataflow backend.

  • Field Summary

    Fields 

    Modifier and Type	Field and Description
    static String	BATCH_WORKER_HARNESS_CONTAINER_IMAGE
    static String	PROJECT_ID_REGEXP Project IDs must contain lowercase letters, digits, or dashes.
    static String	STREAMING_WORKER_HARNESS_CONTAINER_IMAGE

  • Constructor Summary

    Constructors 

    Modifier	Constructor and Description
    protected	DataflowPipelineRunner(DataflowPipelineOptions options)

  • Method Summary

    All Methods Static Methods Instance Methods Concrete Methods 

    Modifier and Type	Method and Description
    <OutputT extends POutput,InputT extends PInput> OutputT	apply(PTransform<InputT,OutputT> transform, InputT input) Applies the given transform to the input.
    protected static List<String>	detectClassPathResourcesToStage(ClassLoader classLoader) Attempts to detect all the resources the class loader has access to.
    static DataflowPipelineRunner	fromOptions(PipelineOptions options) Construct a runner from the provided options.
    DataflowPipelineTranslator	getTranslator() Returns the DataflowPipelineTranslator associated with this object.
    DataflowPipelineJob	run(Pipeline pipeline) Processes the given Pipeline, returning the results.
    void	setHooks(DataflowPipelineRunnerHooks hooks) Sets callbacks to invoke during execution see DataflowPipelineRunnerHooks.
    String	toString()

    • Methods inherited from class java.lang.Object

      clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

• • Field Detail

    • BATCH_WORKER_HARNESS_CONTAINER_IMAGE

      public static final String BATCH_WORKER_HARNESS_CONTAINER_IMAGE

      See Also:

      Constant Field Values

    • STREAMING_WORKER_HARNESS_CONTAINER_IMAGE

      public static final String STREAMING_WORKER_HARNESS_CONTAINER_IMAGE

      See Also:

      Constant Field Values

    • PROJECT_ID_REGEXP

      public static final String PROJECT_ID_REGEXP

      Project IDs must contain lowercase letters, digits, or dashes. IDs must start with a letter and may not end with a dash. This regex isn't exact - this allows for patterns that would be rejected by the service, but this is sufficient for basic validation of project IDs.

      See Also:

      Constant Field Values

  • Constructor Detail

    • DataflowPipelineRunner

      protected DataflowPipelineRunner(DataflowPipelineOptions options)

  • Method Detail

    • fromOptions

      public static DataflowPipelineRunner fromOptions(PipelineOptions options)

      Construct a runner from the provided options.

      Parameters:

      options - Properties that configure the runner.

      Returns:

      The newly created runner.

    • apply

      public <OutputT extends POutput,InputT extends PInput> OutputT apply(PTransform<InputT,OutputT> transform,
                                                                           InputT input)

      Applies the given transform to the input. For transforms with customized definitions for the Dataflow pipeline runner, the application is intercepted and modified here.

      Overrides:

      apply in class PipelineRunner<DataflowPipelineJob>

    • run

      public DataflowPipelineJob run(Pipeline pipeline)

      Description copied from class: PipelineRunner
      Processes the given Pipeline, returning the results.

      Specified by:

      run in class PipelineRunner<DataflowPipelineJob>

    • getTranslator

      public DataflowPipelineTranslator getTranslator()

      Returns the DataflowPipelineTranslator associated with this object.

    • setHooks

      @Experimental
      public void setHooks(DataflowPipelineRunnerHooks hooks)

      Sets callbacks to invoke during execution see DataflowPipelineRunnerHooks.

    • toString

      public String toString()

      Overrides:

      toString in class Object

    • detectClassPathResourcesToStage

      protected static List<String> detectClassPathResourcesToStage(ClassLoader classLoader)

      Attempts to detect all the resources the class loader has access to. This does not recurse to class loader parents stopping it from pulling in resources from the system class loader.

      Parameters:

      classLoader - The URLClassLoader to use to detect resources to stage.

      Returns:

      A list of absolute paths to the resources the class loader uses.

      Throws:

      IllegalArgumentException - If either the class loader is not a URLClassLoader or one of the resources the class loader exposes is not a file resource.