CountingInput (Google Cloud Dataflow SDK 1.9.1 API)

Google Cloud Dataflow SDK for Java, version 1.9.1

com.google.cloud.dataflow.sdk.io

Class CountingInput



  • public class CountingInput
    extends Object
    A PTransform that produces longs. When used to produce a bounded PCollection, CountingInput starts at 0 and counts up to a specified maximum. When used to produce an unbounded PCollection, it counts up to Long.MAX_VALUE and then never produces more output. (In practice, this limit should never be reached.)

    The bounded CountingInput is implemented based on OffsetBasedSource and OffsetBasedSource.OffsetBasedReader, so it performs efficient initial splitting and it supports dynamic work rebalancing.

    To produce a bounded PCollection<Long>, use upTo(long):

    
     Pipeline p = ...
     PTransform<PBegin, PCollection<Long>> producer = CountingInput.upTo(1000);
     PCollection<Long> bounded = p.apply(producer);
     

    To produce an unbounded PCollection<Long>, use unbounded(), calling CountingInput.UnboundedCountingInput.withTimestampFn(SerializableFunction) to provide values with timestamps other than Instant.now().

    
     Pipeline p = ...
    
     // To create an unbounded producer that uses processing time as the element timestamp.
     PCollection<Long> unbounded = p.apply(CountingInput.unbounded());
     // Or, to create an unbounded source that uses a provided function to set the element timestamp.
     PCollection<Long> unboundedWithTimestamps =
         p.apply(CountingInput.unbounded().withTimestampFn(someFn));
     


Was this page helpful? Let us know how we did:

Send feedback about...

Cloud Dataflow
Need help? Visit our support page.