Google Cloud Dataflow SDK for Java, version 1.9.1
Class CountingInput
- java.lang.Object
-
- com.google.cloud.dataflow.sdk.io.CountingInput
-
public class CountingInput extends Object
APTransform
that produces longs. When used to produce abounded
PCollection
,CountingInput
starts at0
and counts up to a specified maximum. When used to produce anunbounded
PCollection
, it counts up toLong.MAX_VALUE
and then never produces more output. (In practice, this limit should never be reached.)The bounded
CountingInput
is implemented based onOffsetBasedSource
andOffsetBasedSource.OffsetBasedReader
, so it performs efficient initial splitting and it supports dynamic work rebalancing.To produce a bounded
PCollection<Long>
, useupTo(long)
:Pipeline p = ... PTransform<PBegin, PCollection<Long>> producer = CountingInput.upTo(1000); PCollection<Long> bounded = p.apply(producer);
To produce an unbounded
PCollection<Long>
, useunbounded()
, callingCountingInput.UnboundedCountingInput.withTimestampFn(SerializableFunction)
to provide values with timestamps other thanInstant.now()
.Pipeline p = ... // To create an unbounded producer that uses processing time as the element timestamp. PCollection<Long> unbounded = p.apply(CountingInput.unbounded()); // Or, to create an unbounded source that uses a provided function to set the element timestamp. PCollection<Long> unboundedWithTimestamps = p.apply(CountingInput.unbounded().withTimestampFn(someFn));
-
-
Nested Class Summary
Nested Classes Modifier and Type Class and Description static class
CountingInput.BoundedCountingInput
APTransform
that will produce a specified number ofLongs
starting from 0.static class
CountingInput.UnboundedCountingInput
-
Constructor Summary
Constructors Constructor and Description CountingInput()
-
Method Summary
All Methods Static Methods Concrete Methods Modifier and Type Method and Description static CountingInput.UnboundedCountingInput
unbounded()
Creates anCountingInput.UnboundedCountingInput
that will produce numbers starting from0
up toLong.MAX_VALUE
.static CountingInput.BoundedCountingInput
upTo(long numElements)
Creates aCountingInput.BoundedCountingInput
that will produce the specified number of elements, from0
tonumElements - 1
.
-
-
-
Method Detail
-
upTo
public static CountingInput.BoundedCountingInput upTo(long numElements)
Creates aCountingInput.BoundedCountingInput
that will produce the specified number of elements, from0
tonumElements - 1
.
-
unbounded
public static CountingInput.UnboundedCountingInput unbounded()
Creates anCountingInput.UnboundedCountingInput
that will produce numbers starting from0
up toLong.MAX_VALUE
.After
Long.MAX_VALUE
, the transform never produces more output. (In practice, this limit should never be reached.)Elements in the resulting
PCollection<Long>
will by default have timestamps corresponding to processing time at element generation, provided byInstant.now()
. Use the transform returned byCountingInput.UnboundedCountingInput.withTimestampFn(SerializableFunction)
to control the output timestamps.
-
-