Google Cloud Dataflow SDK for Java, version 1.9.1
Class CountingSource
- java.lang.Object
-
- com.google.cloud.dataflow.sdk.io.CountingSource
-
public class CountingSource extends Object
A source that produces longs. When used as aBoundedSource
,CountingSource
starts at0
and counts up to a specified maximum. When used as anUnboundedSource
, it counts up toLong.MAX_VALUE
and then never produces more output. (In practice, this limit should never be reached.)The bounded
CountingSource
is implemented based onOffsetBasedSource
andOffsetBasedSource.OffsetBasedReader
, so it performs efficient initial splitting and it supports dynamic work rebalancing.To produce a bounded
PCollection<Long>
, useupTo(long)
:Pipeline p = ... PTransform<PBegin, PCollection<Long>> producer = CountingInput.upTo(1000); PCollection<Long> bounded = p.apply(producer);
To produce an unbounded
PCollection<Long>
, useCountingInput.unbounded()
, callingCountingInput.UnboundedCountingInput.withTimestampFn(SerializableFunction)
to provide values with timestamps other thanInstant.now()
.Pipeline p = ... // To create an unbounded PCollection that uses processing time as the element timestamp. PCollection<Long> unbounded = p.apply(CountingInput.unbounded()); // Or, to create an unbounded source that uses a provided function to set the element timestamp. PCollection<Long> unboundedWithTimestamps = p.apply(CountingInput.unbounded().withTimestampFn(someFn));
-
-
Nested Class Summary
Nested Classes Modifier and Type Class and Description static class
CountingSource.CounterMark
The checkpoint for an unboundedCountingSource
is simply the last value produced.
-
Method Summary
All Methods Static Methods Concrete Methods Deprecated Methods Modifier and Type Method and Description static UnboundedSource<Long,CountingSource.CounterMark>
unbounded()
Deprecated.useCountingInput.unbounded()
insteadstatic UnboundedSource<Long,CountingSource.CounterMark>
unboundedWithTimestampFn(SerializableFunction<Long,Instant> timestampFn)
Deprecated.static BoundedSource<Long>
upTo(long numElements)
Deprecated.useCountingInput.upTo(long)
instead
-
-
-
Method Detail
-
upTo
@Deprecated public static BoundedSource<Long> upTo(long numElements)
Deprecated. useCountingInput.upTo(long)
insteadCreates aBoundedSource
that will produce the specified number of elements, from0
tonumElements - 1
.
-
unbounded
@Deprecated public static UnboundedSource<Long,CountingSource.CounterMark> unbounded()
Deprecated. useCountingInput.unbounded()
insteadCreates anUnboundedSource
that will produce numbers starting from0
up toLong.MAX_VALUE
.After
Long.MAX_VALUE
, the source never produces more output. (In practice, this limit should never be reached.)Elements in the resulting
PCollection<Long>
will have timestamps corresponding to processing time at element generation, provided byInstant.now()
.
-
unboundedWithTimestampFn
@Deprecated public static UnboundedSource<Long,CountingSource.CounterMark> unboundedWithTimestampFn(SerializableFunction<Long,Instant> timestampFn)
Deprecated. useCountingInput.unbounded()
and callCountingInput.UnboundedCountingInput.withTimestampFn(SerializableFunction)
insteadCreates anUnboundedSource
that will produce numbers starting from0
up toLong.MAX_VALUE
, with element timestamps supplied by the specified function.After
Long.MAX_VALUE
, the source never produces more output. (In practice, this limit should never be reached.)Note that the timestamps produced by
timestampFn
may not decrease.
-
-