OutputTimeFn (Google Cloud Dataflow SDK 1.9.1 API)

Google Cloud Dataflow SDK for Java, version 1.9.1

Warning: Dataflow SDK 1.x for Java is unsupported as of October 16, 2018. After August 12, 2020, you can use unsupported SDKs temporarily while you upgrade your pipelines. For more information, see Using an unsupported SDK.

For information on migrating from the 1.x SDK, see Migrating from Dataflow SDK 1.x for Java.

Note: If you are using Cloud Dataflow SDK version 2.x, see the Apache Beam API reference documentation.

com.google.cloud.dataflow.sdk.transforms.windowing

Class OutputTimeFn<W extends BoundedWindow>

• java.lang.Object
• • com.google.cloud.dataflow.sdk.transforms.windowing.OutputTimeFn<W>

• Type Parameters:

  W - the type of window. Contravariant: methods accepting any subtype of OutputTimeFn<W> should use the parameter type OutputTimeFn<? super W>.

  All Implemented Interfaces:

  Serializable

  Direct Known Subclasses:

  OutputTimeFn.Defaults, OutputTimeFn.DependsOnlyOnWindow

  ---

  
  

  @Experimental(value=OUTPUT_TIME)
  public abstract class OutputTimeFn<W extends BoundedWindow>
  extends Object
  implements Serializable

  (Experimental) A function from timestamps of input values to the timestamp for a computed value.

  The function is represented via three components:

  1. assignOutputTime(org.joda.time.Instant, W) calculates an output timestamp for any input value in a particular window.
  2. The output timestamps for all non-late input values within a window are combined according to combine(), a commutative and associative operation on the output timestamps.
  3. The output timestamp when windows merge is provided by merge().

  This abstract class cannot be subclassed directly, by design: it may grow in consumer-compatible ways that require mutually-exclusive default implementations. To create a concrete subclass, extend OutputTimeFn.Defaults or OutputTimeFn.DependsOnlyOnWindow. Note that as long as this class remains experimental, we may also choose to change it in arbitrary backwards-incompatible ways.

  See Also:

  Serialized Form

• • Nested Class Summary

    Nested Classes 

    Modifier and Type	Class and Description
    static class	OutputTimeFn.Defaults<W extends BoundedWindow> (Experimental) Default method implementations for OutputTimeFn where the output time depends on the input element timestamps and possibly the window.
    static class	OutputTimeFn.DependsOnlyOnWindow<W extends BoundedWindow> (Experimental) Default method implementations for OutputTimeFn when the output time depends only on the window.

  • Method Summary

    All Methods Instance Methods Abstract Methods 

    Modifier and Type	Method and Description
    abstract Instant	assignOutputTime(Instant inputTimestamp, W window) Returns the output timestamp to use for data depending on the given inputTimestamp in the specified window.
    abstract Instant	combine(Instant outputTime, Instant otherOutputTime) Combines the given output times, which must be from the same window, into an output time for a computed value.
    abstract boolean	dependsOnlyOnEarliestInputTimestamp() Returns true if the result of combination of many output timestamps actually depends only on the earliest.
    abstract boolean	dependsOnlyOnWindow() Returns true if the result does not depend on what outputs were combined but only the window they are in.
    abstract Instant	merge(W intoWindow, Iterable<? extends Instant> mergingTimestamps) Merges the given output times, presumed to be combined output times for windows that are merging, into an output time for the resultWindow.

    • Methods inherited from class java.lang.Object

      clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

• • Method Detail

    • assignOutputTime

      public abstract Instant assignOutputTime(Instant inputTimestamp,
                                               W window)

      Returns the output timestamp to use for data depending on the given inputTimestamp in the specified window.

      The result of this method must be between inputTimestamp and window.maxTimestamp() (inclusive on both sides).

      This function must be monotonic across input timestamps. Specifically, if A < B, then assignOutputTime(A, window) <= assignOutputTime(B, window).

      For a WindowFn that doesn't produce overlapping windows, this can (and typically should) just return inputTimestamp. In the presence of overlapping windows, it is suggested that the result in later overlapping windows is past the end of earlier windows so that the later windows don't prevent the watermark from progressing past the end of the earlier window.

      See the overview of OutputTimeFn for the consistency properties required between assignOutputTime(org.joda.time.Instant, W), combine(org.joda.time.Instant, org.joda.time.Instant), and merge(W, java.lang.Iterable<? extends org.joda.time.Instant>).

    • combine

      public abstract Instant combine(Instant outputTime,
                                      Instant otherOutputTime)

      Combines the given output times, which must be from the same window, into an output time for a computed value.

      • combine must be commutative: combine(a, b).equals(combine(b, a)).
      • combine must be associative: combine(a, combine(b, c)).equals(combine(combine(a, b), c)).

    • merge

      public abstract Instant merge(W intoWindow,
                                    Iterable<? extends Instant> mergingTimestamps)

      Merges the given output times, presumed to be combined output times for windows that are merging, into an output time for the resultWindow.

      When windows w1 and w2 merge to become a new window w1plus2, then merge(W, java.lang.Iterable<? extends org.joda.time.Instant>) must be implemented such that the output time is the same as if all timestamps were assigned in w1plus2. Formally:

      fn.merge(w, fn.assignOutputTime(t1, w1), fn.assignOutputTime(t2, w2))

      must be equal to

      fn.combine(fn.assignOutputTime(t1, w1plus2), fn.assignOutputTime(t2, w1plus2))

      If the assigned time depends only on the window, the correct implementation of merge() necessarily returns the result of assignOutputTime(t1, w1plus2) (which equals assignOutputTime(t2, w1plus2). Defaults for this case are provided by OutputTimeFn.DependsOnlyOnWindow.

      For many other OutputTimeFn implementations, such as taking the earliest or latest timestamp, this will be the same as combine(). Defaults for this case are provided by OutputTimeFn.Defaults.

    • dependsOnlyOnEarliestInputTimestamp

      public abstract boolean dependsOnlyOnEarliestInputTimestamp()

      Returns true if the result of combination of many output timestamps actually depends only on the earliest.

      This may allow optimizations when it is very efficient to retrieve the earliest timestamp to be combined.

    • dependsOnlyOnWindow

      public abstract boolean dependsOnlyOnWindow()

      Returns true if the result does not depend on what outputs were combined but only the window they are in. The canonical example is if all timestamps are sure to be the end of the window.

      This may allow optimizations, since it is typically very efficient to retrieve the window and combining output timestamps is not necessary.

      If the assigned output time for an implementation depends only on the window, consider extending OutputTimeFn.DependsOnlyOnWindow, which returns true here and also provides a framework for easily implementing a correct merge(W, java.lang.Iterable<? extends org.joda.time.Instant>), combine(org.joda.time.Instant, org.joda.time.Instant) and assignOutputTime(org.joda.time.Instant, W).