IntraBundleParallelization (Google Cloud Dataflow SDK 1.9.1 API)

Google Cloud Dataflow SDK for Java, version 1.9.1

Warning: Dataflow SDK 1.x for Java is unsupported as of October 16, 2018. After August 12, 2020, you can use unsupported SDKs temporarily while you upgrade your pipelines. For more information, see Using an unsupported SDK.

For information on migrating from the 1.x SDK, see Migrating from Dataflow SDK 1.x for Java.

Note: If you are using Cloud Dataflow SDK version 2.x, see the Apache Beam API reference documentation.

com.google.cloud.dataflow.sdk.transforms

Class IntraBundleParallelization

• java.lang.Object
• • com.google.cloud.dataflow.sdk.transforms.IntraBundleParallelization

• ---

  
  

  public class IntraBundleParallelization
  extends Object

  Provides multi-threading of DoFns, using threaded execution to process multiple elements concurrently within a bundle.

  Note, that each Dataflow worker will already process multiple bundles concurrently and usage of this class is meant only for cases where processing elements from within a bundle is limited by blocking calls.

  CPU intensive or IO intensive tasks are in general a poor fit for parallelization. This is because a limited resource that is already maximally utilized does not benefit from sub-division of work. The parallelization will increase the amount of time to process each element yet the throughput for processing will remain relatively the same. For example, if the local disk (an IO resource) has a maximum write rate of 10 MiB/s, and processing each element requires to write 20 MiBs to disk, then processing one element to disk will take 2 seconds. Yet processing 3 elements concurrently (each getting an equal share of the maximum write rate) will take at least 6 seconds to complete (there is additional overhead in the extra parallelization).

  To parallelize a DoFn to 10 threads:

  
   PCollection<T> data = ...;
   data.apply(
     IntraBundleParallelization.of(new MyDoFn())
                               .withMaxParallelism(10)));
   

  An uncaught exception from the wrapped DoFn will result in the exception being rethrown in later calls to IntraBundleParallelization.MultiThreadedIntraBundleProcessingDoFn.processElement(com.google.cloud.dataflow.sdk.transforms.DoFn<InputT, OutputT>.ProcessContext) or a call to IntraBundleParallelization.MultiThreadedIntraBundleProcessingDoFn.finishBundle(com.google.cloud.dataflow.sdk.transforms.DoFn<InputT, OutputT>.Context).

• • Nested Class Summary

    Nested Classes 

    Modifier and Type	Class and Description
    static class	IntraBundleParallelization.Bound<InputT,OutputT> A PTransform that, when applied to a PCollection<InputT>, invokes a user-specified DoFn<InputT, OutputT> on all its elements, with all its outputs collected into an output PCollection<OutputT>.
    static class	IntraBundleParallelization.MultiThreadedIntraBundleProcessingDoFn<InputT,OutputT> A multi-threaded DoFn wrapper.
    static class	IntraBundleParallelization.Unbound An incomplete IntraBundleParallelization transform, with unbound input/output types.

  • Constructor Summary

    Constructors 

    Constructor and Description
    IntraBundleParallelization()

  • Method Summary

    All Methods Static Methods Concrete Methods 

    Modifier and Type	Method and Description
    static <InputT,OutputT> IntraBundleParallelization.Bound<InputT,OutputT>	of(DoFn<InputT,OutputT> doFn) Creates a IntraBundleParallelization PTransform for the given DoFn that processes elements using multiple threads.
    static IntraBundleParallelization.Unbound	withMaxParallelism(int maxParallelism) Creates a IntraBundleParallelization PTransform with the specified maximum concurrency level.

    • Methods inherited from class java.lang.Object

      clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

• • Constructor Detail

    • IntraBundleParallelization

      public IntraBundleParallelization()

  • Method Detail

    • of

      public static <InputT,OutputT> IntraBundleParallelization.Bound<InputT,OutputT> of(DoFn<InputT,OutputT> doFn)

      Creates a IntraBundleParallelization PTransform for the given DoFn that processes elements using multiple threads.

      Note that the specified doFn needs to be thread safe.

    • withMaxParallelism

      public static IntraBundleParallelization.Unbound withMaxParallelism(int maxParallelism)

      Creates a IntraBundleParallelization PTransform with the specified maximum concurrency level.