IntraBundleParallelization.MultiThreadedIntraBundleProcessingDoFn (Google Cloud Dataflow SDK 1.9.1 API)

Google Cloud Dataflow SDK for Java, version 1.9.1

Warning: Dataflow SDK 1.x for Java is unsupported as of October 16, 2018. After August 12, 2020, you can use unsupported SDKs temporarily while you upgrade your pipelines. For more information, see Using an unsupported SDK.

For information on migrating from the 1.x SDK, see Migrating from Dataflow SDK 1.x for Java.

Note: If you are using Cloud Dataflow SDK version 2.x, see the Apache Beam API reference documentation.

com.google.cloud.dataflow.sdk.transforms

Class IntraBundleParallelization.MultiThreadedIntraBundleProcessingDoFn<InputT,OutputT>

• java.lang.Object
• • com.google.cloud.dataflow.sdk.transforms.DoFn<InputT,OutputT>
  • • com.google.cloud.dataflow.sdk.transforms.IntraBundleParallelization.MultiThreadedIntraBundleProcessingDoFn<InputT,OutputT>

• Type Parameters:

  InputT - the type of the (main) input elements

  OutputT - the type of the (main) output elements

  All Implemented Interfaces:

  HasDisplayData, Serializable

  Enclosing class:

  IntraBundleParallelization

  ---

  
  

  public static class IntraBundleParallelization.MultiThreadedIntraBundleProcessingDoFn<InputT,OutputT>
  extends DoFn<InputT,OutputT>

  A multi-threaded DoFn wrapper.

  See Also:

  IntraBundleParallelization.of(DoFn), Serialized Form

• • Nested Class Summary

    • Nested classes/interfaces inherited from class com.google.cloud.dataflow.sdk.transforms.DoFn

      DoFn.Context, DoFn.ProcessContext, DoFn.RequiresWindowAccess

  • Constructor Summary

    Constructors 

    Constructor and Description
    MultiThreadedIntraBundleProcessingDoFn(DoFn<InputT,OutputT> doFn, int maxParallelism)

  • Method Summary

    All Methods Instance Methods Concrete Methods 

    Modifier and Type	Method and Description
    void	finishBundle(DoFn.Context c) Finishes processing this batch of elements.
    protected TypeDescriptor<InputT>	getInputTypeDescriptor() Returns a TypeDescriptor capturing what is known statically about the input type of this DoFn instance's most-derived class.
    protected TypeDescriptor<OutputT>	getOutputTypeDescriptor() Returns a TypeDescriptor capturing what is known statically about the output type of this DoFn instance's most-derived class.
    void	processElement(DoFn.ProcessContext c) Processes one input element.
    void	startBundle(DoFn.Context c) Prepares this DoFn instance for processing a batch of elements.

    • Methods inherited from class com.google.cloud.dataflow.sdk.transforms.DoFn

      createAggregator, createAggregator, getAllowedTimestampSkew, populateDisplayData

    • Methods inherited from class java.lang.Object

      clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

• • Constructor Detail

    • MultiThreadedIntraBundleProcessingDoFn

      public MultiThreadedIntraBundleProcessingDoFn(DoFn<InputT,OutputT> doFn,
                                                    int maxParallelism)

  • Method Detail

    • startBundle

      public void startBundle(DoFn.Context c)
                       throws Exception

      Description copied from class: DoFn
      Prepares this DoFn instance for processing a batch of elements.

      By default, does nothing.

      Overrides:

      startBundle in class DoFn<InputT,OutputT>

      Throws:

      Exception

    • processElement

      public void processElement(DoFn.ProcessContext c)
                          throws Exception

      Description copied from class: DoFn
      Processes one input element.

      The current element of the input PCollection is returned by c.element(). It should be considered immutable. The Dataflow runtime will not mutate the element, so it is safe to cache, etc. The element should not be mutated by any of the DoFn methods, because it may be cached elsewhere, retained by the Dataflow runtime, or used in other unspecified ways.

      A value is added to the main output PCollection by DoFn.Context.output(OutputT). Once passed to output the element should be considered immutable and not be modified in any way. It may be cached elsewhere, retained by the Dataflow runtime, or used in other unspecified ways.

      Specified by:

      processElement in class DoFn<InputT,OutputT>

      Throws:

      Exception

      See Also:

      DoFn.ProcessContext

    • finishBundle

      public void finishBundle(DoFn.Context c)
                        throws Exception

      Description copied from class: DoFn
      Finishes processing this batch of elements.

      By default, does nothing.

      Overrides:

      finishBundle in class DoFn<InputT,OutputT>

      Throws:

      Exception

    • getInputTypeDescriptor

      protected TypeDescriptor<InputT> getInputTypeDescriptor()

      Description copied from class: DoFn
      Returns a TypeDescriptor capturing what is known statically about the input type of this DoFn instance's most-derived class.

      See DoFn.getOutputTypeDescriptor() for more discussion.

      Overrides:

      getInputTypeDescriptor in class DoFn<InputT,OutputT>

    • getOutputTypeDescriptor

      protected TypeDescriptor<OutputT> getOutputTypeDescriptor()

      Description copied from class: DoFn
      Returns a TypeDescriptor capturing what is known statically about the output type of this DoFn instance's most-derived class.

      In the normal case of a concrete DoFn subclass with no generic type parameters of its own (including anonymous inner classes), this will be a complete non-generic type, which is good for choosing a default output Coder<OutputT> for the output PCollection<OutputT>.

      Overrides:

      getOutputTypeDescriptor in class DoFn<InputT,OutputT>