DoFn (Google Cloud Dataflow SDK 1.9.1 API)

Google Cloud Dataflow SDK for Java, version 1.9.1

Warning: Dataflow SDK 1.x for Java is unsupported as of October 16, 2018. After August 12, 2020, Dataflow will not run jobs using Dataflow 1.x and below. See Migrating from Dataflow SDK 1.x for Java for migration guidance.

Note: If you are using Cloud Dataflow SDK version 2.x, see the Apache Beam API reference documentation.

com.google.cloud.dataflow.sdk.transforms

Class DoFn<InputT,OutputT>

• java.lang.Object
• • com.google.cloud.dataflow.sdk.transforms.DoFn<InputT,OutputT>

• Type Parameters:

  InputT - the type of the (main) input elements

  OutputT - the type of the (main) output elements

  All Implemented Interfaces:

  HasDisplayData, Serializable

  Direct Known Subclasses:

  DirectPipelineRunner.TestCombineDoFn, IntraBundleParallelization.MultiThreadedIntraBundleProcessingDoFn, PubsubIO.Read.Bound.PubsubReader, PubsubIO.Write.Bound.PubsubWriter

  ---

  
  

  public abstract class DoFn<InputT,OutputT>
  extends Object
  implements Serializable, HasDisplayData

  The argument to ParDo providing the code to use to process elements of the input PCollection.

  See ParDo for more explanation, examples of use, and discussion of constraints on DoFns, including their serializability, lack of access to global shared mutable state, requirements for failure tolerance, and benefits of optimization.

  DoFns can be tested in the context of a particular Pipeline by running that Pipeline on sample input and then checking its output. Unit testing of a DoFn, separately from any ParDo transform or Pipeline, can be done via the DoFnTester harness.

  DoFnWithContext (currently experimental) offers an alternative mechanism for accessing DoFn.ProcessContext.window() without the need to implement DoFn.RequiresWindowAccess.

  See also processElement(com.google.cloud.dataflow.sdk.transforms.DoFn<InputT, OutputT>.ProcessContext) for details on implementing the transformation from InputT to OutputT.

  See Also:

  Serialized Form

• • Nested Class Summary

    Nested Classes 

    Modifier and Type	Class and Description
    class	DoFn.Context Information accessible to all methods in this DoFn.
    class	DoFn.ProcessContext Information accessible when running processElement(com.google.cloud.dataflow.sdk.transforms.DoFn<InputT, OutputT>.ProcessContext).
    static interface	DoFn.RequiresWindowAccess Interface for signaling that a DoFn needs to access the window the element is being processed in, via DoFn.ProcessContext.window().

  • Constructor Summary

    Constructors 

    Constructor and Description
    DoFn()

  • Method Summary

    All Methods Instance Methods Abstract Methods Concrete Methods Deprecated Methods 

    Modifier and Type	Method and Description
    protected <AggInputT,AggOutputT> Aggregator<AggInputT,AggOutputT>	createAggregator(String name, Combine.CombineFn<? super AggInputT,?,AggOutputT> combiner) Returns an Aggregator with aggregation logic specified by the Combine.CombineFn argument.
    protected <AggInputT> Aggregator<AggInputT,AggInputT>	createAggregator(String name, SerializableFunction<Iterable<AggInputT>,AggInputT> combiner) Returns an Aggregator with the aggregation logic specified by the SerializableFunction argument.
    void	finishBundle(DoFn.Context c) Finishes processing this batch of elements.
    Duration	getAllowedTimestampSkew() Deprecated.  does not interact well with the watermark.
    protected TypeDescriptor<InputT>	getInputTypeDescriptor() Returns a TypeDescriptor capturing what is known statically about the input type of this DoFn instance's most-derived class.
    protected TypeDescriptor<OutputT>	getOutputTypeDescriptor() Returns a TypeDescriptor capturing what is known statically about the output type of this DoFn instance's most-derived class.
    void	populateDisplayData(DisplayData.Builder builder) Register display data for the given transform or component.
    abstract void	processElement(DoFn.ProcessContext c) Processes one input element.
    void	startBundle(DoFn.Context c) Prepares this DoFn instance for processing a batch of elements.

    • Methods inherited from class java.lang.Object

      clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

• • Constructor Detail

    • DoFn

      public DoFn()

  • Method Detail

    • getAllowedTimestampSkew

      @Deprecated
      public Duration getAllowedTimestampSkew()

      Deprecated. does not interact well with the watermark.
      Returns the allowed timestamp skew duration, which is the maximum duration that timestamps can be shifted backward in DoFn.Context.outputWithTimestamp(OutputT, org.joda.time.Instant).

      The default value is Duration.ZERO, in which case timestamps can only be shifted forward to future. For infinite skew, return Duration.millis(Long.MAX_VALUE).

      Note that producing an element whose timestamp is less than the current timestamp may result in late data, i.e. returning a non-zero value here does not impact watermark calculations used for firing windows.

    • startBundle

      public void startBundle(DoFn.Context c)
                       throws Exception

      Prepares this DoFn instance for processing a batch of elements.

      By default, does nothing.

      Throws:

      Exception

    • processElement

      public abstract void processElement(DoFn.ProcessContext c)
                                   throws Exception

      Processes one input element.

      The current element of the input PCollection is returned by c.element(). It should be considered immutable. The Dataflow runtime will not mutate the element, so it is safe to cache, etc. The element should not be mutated by any of the DoFn methods, because it may be cached elsewhere, retained by the Dataflow runtime, or used in other unspecified ways.

      A value is added to the main output PCollection by DoFn.Context.output(OutputT). Once passed to output the element should be considered immutable and not be modified in any way. It may be cached elsewhere, retained by the Dataflow runtime, or used in other unspecified ways.

      Throws:

      Exception

      See Also:

      DoFn.ProcessContext

    • finishBundle

      public void finishBundle(DoFn.Context c)
                        throws Exception

      Finishes processing this batch of elements.

      By default, does nothing.

      Throws:

      Exception

    • populateDisplayData

      public void populateDisplayData(DisplayData.Builder builder)

      Register display data for the given transform or component.

      populateDisplayData(DisplayData.Builder) is invoked by Pipeline runners to collect display data via DisplayData.from(HasDisplayData). Implementations may call super.populateDisplayData(builder) in order to register display data in the current namespace, but should otherwise use subcomponent.populateDisplayData(builder) to use the namespace of the subcomponent.

      By default, does not register any display data. Implementors may override this method to provide their own display data.

      Specified by:

      populateDisplayData in interface HasDisplayData

      Parameters:

      builder - The builder to populate with display data.

      See Also:

      HasDisplayData

    • getInputTypeDescriptor

      protected TypeDescriptor<InputT> getInputTypeDescriptor()

      Returns a TypeDescriptor capturing what is known statically about the input type of this DoFn instance's most-derived class.

      See getOutputTypeDescriptor() for more discussion.

    • getOutputTypeDescriptor

      protected TypeDescriptor<OutputT> getOutputTypeDescriptor()

      Returns a TypeDescriptor capturing what is known statically about the output type of this DoFn instance's most-derived class.

      In the normal case of a concrete DoFn subclass with no generic type parameters of its own (including anonymous inner classes), this will be a complete non-generic type, which is good for choosing a default output Coder<OutputT> for the output PCollection<OutputT>.

    • createAggregator

      protected final <AggInputT,AggOutputT> Aggregator<AggInputT,AggOutputT> createAggregator(String name,
                                                                                               Combine.CombineFn<? super AggInputT,?,AggOutputT> combiner)

      Returns an Aggregator with aggregation logic specified by the Combine.CombineFn argument. The name provided must be unique across Aggregators created within the DoFn. Aggregators can only be created during pipeline construction.

      Parameters:

      name - the name of the aggregator

      combiner - the Combine.CombineFn to use in the aggregator

      Returns:

      an aggregator for the provided name and combiner in the scope of this DoFn

      Throws:

      NullPointerException - if the name or combiner is null

      IllegalArgumentException - if the given name collides with another aggregator in this scope

      IllegalStateException - if called during pipeline processing.

    • createAggregator

      protected final <AggInputT> Aggregator<AggInputT,AggInputT> createAggregator(String name,
                                                                                   SerializableFunction<Iterable<AggInputT>,AggInputT> combiner)

      Returns an Aggregator with the aggregation logic specified by the SerializableFunction argument. The name provided must be unique across Aggregators created within the DoFn. Aggregators can only be created during pipeline construction.

      Parameters:

      name - the name of the aggregator

      combiner - the SerializableFunction to use in the aggregator

      Returns:

      an aggregator for the provided name and combiner in the scope of this DoFn

      Throws:

      NullPointerException - if the name or combiner is null

      IllegalArgumentException - if the given name collides with another aggregator in this scope

      IllegalStateException - if called during pipeline processing.