Google Cloud Dataflow SDK for Java, version 1.9.1
- Enclosing class:
public static enum PaneInfo.Timing extends Enum<PaneInfo.Timing>Enumerates the possibilities for the timing of this pane firing related to the input and output watermarks for its computation.
A window may fire multiple panes, and the timing of those panes generally follows the regular expression
EARLY* ON_TIME? LATE*. Generally a pane is considered:
EARLYif the system cannot be sure it has seen all data which may contribute to the pane's window.
ON_TIMEif the system predicts it has seen all the data which may contribute to the pane's window.
LATEif the system has encountered new data after predicting no more could arrive. It is possible an
ON_TIMEpane has already been emitted, in which case any following panes are considered
AfterWatermark.pastEndOfWindow()trigger may produce an
ON_TIMEpane. With merging
WindowFn's, windows may be merged to produce new windows that satisfy their own instance of the above regular expression. The only guarantee is that once a window produces a final pane, it will not be merged into any new windows.
The predictions above are made using the mechanism of watermarks. See
TimerInternalsfor more information about watermarks.
We can state some properties of
ON_TIMEpanes, but first need some definitions:
- We'll call a pipeline 'simple' if it does not use
DoFn.Context.outputWithTimestamp(OutputT, org.joda.time.Instant)in any
DoFn, and it uses the same
Window.Bound.withAllowedLateness(org.joda.time.Duration)argument value on all windows (or uses the default of
- We'll call an element 'locally late', from the point of view of a computation on a worker, if the element's timestamp is before the input watermark for that computation on that worker. The element is otherwise 'locally on-time'.
- We'll say 'the pane's timestamp' to mean the timestamp of the element produced to represent the pane's contents.
Then in simple pipelines:
- (Soundness) An
ON_TIMEpane can never cause a later computation to generate a
LATEpane. (If it did, it would imply a later computation's input watermark progressed ahead of an earlier stage's output watermark, which by design is not possible.)
- (Liveness) An
ON_TIMEpane is emitted as soon as possible after the input watermark passes the end of the pane's window.
- (Consistency) A pane with only locally on-time elements will always be
ON_TIME. And a
LATEpane cannot contain locally on-time elements.
However, note that:
ON_TIMEpane may contain locally late elements. It may even contain only locally late elements. Provided a locally late element finds its way into an
ON_TIMEpane its lateness becomes unobservable.
LATEpane does not necessarily cause any following computation panes to be marked as
Enum Constant Summary
Enum Constants Enum Constant and Description
EARLYPane was fired before the input watermark had progressed after the end of the window.
LATEPane was fired after the output watermark had progressed past the end of the window.
ON_TIMEPane was fired by a
AfterWatermark.pastEndOfWindow()trigger because the input watermark progressed after the end of the window.
UNKNOWNThis element was not produced in a triggered pane and its relation to input and output watermarks is unknown.
All Methods Static Methods Concrete Methods Modifier and Type Method and Description
valueOf(String name)Returns the enum constant of this type with the specified name.
values()Returns an array containing the constants of this enum type, in the order they are declared.
Methods inherited from class java.lang.Enum
clone, compareTo, equals, finalize, getDeclaringClass, hashCode, name, ordinal, toString, valueOf
Enum Constant Detail
public static final PaneInfo.Timing EARLYPane was fired before the input watermark had progressed after the end of the window.
public static final PaneInfo.Timing ON_TIMEPane was fired by a
AfterWatermark.pastEndOfWindow()trigger because the input watermark progressed after the end of the window. However the output watermark has not yet progressed after the end of the window. Thus it is still possible to assign a timestamp to the element representing this pane which cannot be considered locally late by any following computation.
public static final PaneInfo.Timing LATEPane was fired after the output watermark had progressed past the end of the window.
public static final PaneInfo.Timing UNKNOWN
public static PaneInfo.Timing values()Returns an array containing the constants of this enum type, in the order they are declared. This method may be used to iterate over the constants as follows:
for (PaneInfo.Timing c : PaneInfo.Timing.values()) System.out.println(c);
- an array containing the constants of this enum type, in the order they are declared
public static PaneInfo.Timing valueOf(String name)Returns the enum constant of this type with the specified name. The string must match exactly an identifier used to declare an enum constant in this type. (Extraneous whitespace characters are not permitted.)