CountingSource (Google Cloud Dataflow SDK 1.9.1 API)

Google Cloud Dataflow SDK for Java, version 1.9.1

com.google.cloud.dataflow.sdk.io

Class CountingSource



  • public class CountingSource
    extends Object
    A source that produces longs. When used as a BoundedSource, CountingSource starts at 0 and counts up to a specified maximum. When used as an UnboundedSource, it counts up to Long.MAX_VALUE and then never produces more output. (In practice, this limit should never be reached.)

    The bounded CountingSource is implemented based on OffsetBasedSource and OffsetBasedSource.OffsetBasedReader, so it performs efficient initial splitting and it supports dynamic work rebalancing.

    To produce a bounded PCollection<Long>, use upTo(long):

    
     Pipeline p = ...
     PTransform<PBegin, PCollection<Long>> producer = CountingInput.upTo(1000);
     PCollection<Long> bounded = p.apply(producer);
     

    To produce an unbounded PCollection<Long>, use CountingInput.unbounded(), calling CountingInput.UnboundedCountingInput.withTimestampFn(SerializableFunction) to provide values with timestamps other than Instant.now().

    
     Pipeline p = ...
    
     // To create an unbounded PCollection that uses processing time as the element timestamp.
     PCollection<Long> unbounded = p.apply(CountingInput.unbounded());
     // Or, to create an unbounded source that uses a provided function to set the element timestamp.
     PCollection<Long> unboundedWithTimestamps =
         p.apply(CountingInput.unbounded().withTimestampFn(someFn));
    
     


Monitor your resources on the go

Get the Google Cloud Console app to help you manage your projects.

Send feedback about...

Cloud Dataflow