Google Cloud Dataflow SDK for Java, version 1.9.1
Class OutputTimeFn<W extends BoundedWindow>
- java.lang.Object
-
- com.google.cloud.dataflow.sdk.transforms.windowing.OutputTimeFn<W>
-
- Type Parameters:
W
- the type of window. Contravariant: methods accepting any subtype ofOutputTimeFn<W>
should use the parameter typeOutputTimeFn<? super W>
.
- All Implemented Interfaces:
- Serializable
- Direct Known Subclasses:
- OutputTimeFn.Defaults, OutputTimeFn.DependsOnlyOnWindow
@Experimental(value=OUTPUT_TIME) public abstract class OutputTimeFn<W extends BoundedWindow> extends Object implements Serializable
(Experimental) A function from timestamps of input values to the timestamp for a computed value.The function is represented via three components:
assignOutputTime(org.joda.time.Instant, W)
calculates an output timestamp for any input value in a particular window.- The output timestamps for all non-late input values within a window are combined
according to
combine()
, a commutative and associative operation on the output timestamps. - The output timestamp when windows merge is provided by
merge()
.
This abstract class cannot be subclassed directly, by design: it may grow in consumer-compatible ways that require mutually-exclusive default implementations. To create a concrete subclass, extend
OutputTimeFn.Defaults
orOutputTimeFn.DependsOnlyOnWindow
. Note that as long as this class remains experimental, we may also choose to change it in arbitrary backwards-incompatible ways.- See Also:
- Serialized Form
-
-
Nested Class Summary
Nested Classes Modifier and Type Class and Description static class
OutputTimeFn.Defaults<W extends BoundedWindow>
(Experimental) Default method implementations forOutputTimeFn
where the output time depends on the input element timestamps and possibly the window.static class
OutputTimeFn.DependsOnlyOnWindow<W extends BoundedWindow>
(Experimental) Default method implementations forOutputTimeFn
when the output time depends only on the window.
-
Method Summary
All Methods Instance Methods Abstract Methods Modifier and Type Method and Description abstract Instant
assignOutputTime(Instant inputTimestamp, W window)
Returns the output timestamp to use for data depending on the giveninputTimestamp
in the specifiedwindow
.abstract Instant
combine(Instant outputTime, Instant otherOutputTime)
Combines the given output times, which must be from the same window, into an output time for a computed value.abstract boolean
dependsOnlyOnEarliestInputTimestamp()
Returnstrue
if the result of combination of many output timestamps actually depends only on the earliest.abstract boolean
dependsOnlyOnWindow()
Returnstrue
if the result does not depend on what outputs were combined but only the window they are in.abstract Instant
merge(W intoWindow, Iterable<? extends Instant> mergingTimestamps)
Merges the given output times, presumed to be combined output times for windows that are merging, into an output time for theresultWindow
.
-
-
-
Method Detail
-
assignOutputTime
public abstract Instant assignOutputTime(Instant inputTimestamp, W window)
Returns the output timestamp to use for data depending on the giveninputTimestamp
in the specifiedwindow
.The result of this method must be between
inputTimestamp
andwindow.maxTimestamp()
(inclusive on both sides).This function must be monotonic across input timestamps. Specifically, if
A < B
, thenassignOutputTime(A, window) <= assignOutputTime(B, window)
.For a
WindowFn
that doesn't produce overlapping windows, this can (and typically should) just returninputTimestamp
. In the presence of overlapping windows, it is suggested that the result in later overlapping windows is past the end of earlier windows so that the later windows don't prevent the watermark from progressing past the end of the earlier window.See the overview of
OutputTimeFn
for the consistency properties required betweenassignOutputTime(org.joda.time.Instant, W)
,combine(org.joda.time.Instant, org.joda.time.Instant)
, andmerge(W, java.lang.Iterable<? extends org.joda.time.Instant>)
.
-
combine
public abstract Instant combine(Instant outputTime, Instant otherOutputTime)
Combines the given output times, which must be from the same window, into an output time for a computed value.combine
must be commutative:combine(a, b).equals(combine(b, a))
.combine
must be associative:combine(a, combine(b, c)).equals(combine(combine(a, b), c))
.
-
merge
public abstract Instant merge(W intoWindow, Iterable<? extends Instant> mergingTimestamps)
Merges the given output times, presumed to be combined output times for windows that are merging, into an output time for theresultWindow
.When windows
w1
andw2
merge to become a new windoww1plus2
, thenmerge(W, java.lang.Iterable<? extends org.joda.time.Instant>)
must be implemented such that the output time is the same as if all timestamps were assigned inw1plus2
. Formally:fn.merge(w, fn.assignOutputTime(t1, w1), fn.assignOutputTime(t2, w2))
must be equal to
fn.combine(fn.assignOutputTime(t1, w1plus2), fn.assignOutputTime(t2, w1plus2))
If the assigned time depends only on the window, the correct implementation of
merge()
necessarily returns the result ofassignOutputTime(t1, w1plus2)
(which equalsassignOutputTime(t2, w1plus2)
. Defaults for this case are provided byOutputTimeFn.DependsOnlyOnWindow
.For many other
OutputTimeFn
implementations, such as taking the earliest or latest timestamp, this will be the same ascombine()
. Defaults for this case are provided byOutputTimeFn.Defaults
.
-
dependsOnlyOnEarliestInputTimestamp
public abstract boolean dependsOnlyOnEarliestInputTimestamp()
Returnstrue
if the result of combination of many output timestamps actually depends only on the earliest.This may allow optimizations when it is very efficient to retrieve the earliest timestamp to be combined.
-
dependsOnlyOnWindow
public abstract boolean dependsOnlyOnWindow()
Returnstrue
if the result does not depend on what outputs were combined but only the window they are in. The canonical example is if all timestamps are sure to be the end of the window.This may allow optimizations, since it is typically very efficient to retrieve the window and combining output timestamps is not necessary.
If the assigned output time for an implementation depends only on the window, consider extending
OutputTimeFn.DependsOnlyOnWindow
, which returnstrue
here and also provides a framework for easily implementing a correctmerge(W, java.lang.Iterable<? extends org.joda.time.Instant>)
,combine(org.joda.time.Instant, org.joda.time.Instant)
andassignOutputTime(org.joda.time.Instant, W)
.
-
-