CountingSource (Google Cloud Dataflow SDK 1.9.1 API)

Google Cloud Dataflow SDK for Java, version 1.9.1

Warning: Dataflow SDK 1.x for Java is unsupported as of October 16, 2018. After August 12, 2020, you can use unsupported SDKs temporarily while you upgrade your pipelines. For more information, see Using an unsupported SDK.

For information on migrating from the 1.x SDK, see Migrating from Dataflow SDK 1.x for Java.

Note: If you are using Cloud Dataflow SDK version 2.x, see the Apache Beam API reference documentation.

com.google.cloud.dataflow.sdk.io

Class CountingSource

• java.lang.Object
• • com.google.cloud.dataflow.sdk.io.CountingSource

• ---

  
  

  public class CountingSource
  extends Object

  A source that produces longs. When used as a BoundedSource, CountingSource starts at 0 and counts up to a specified maximum. When used as an UnboundedSource, it counts up to Long.MAX_VALUE and then never produces more output. (In practice, this limit should never be reached.)

  The bounded CountingSource is implemented based on OffsetBasedSource and OffsetBasedSource.OffsetBasedReader, so it performs efficient initial splitting and it supports dynamic work rebalancing.

  To produce a bounded PCollection<Long>, use upTo(long):

  
   Pipeline p = ...
   PTransform<PBegin, PCollection<Long>> producer = CountingInput.upTo(1000);
   PCollection<Long> bounded = p.apply(producer);
   

  To produce an unbounded PCollection<Long>, use CountingInput.unbounded(), calling CountingInput.UnboundedCountingInput.withTimestampFn(SerializableFunction) to provide values with timestamps other than Instant.now().

  
   Pipeline p = ...
  
   // To create an unbounded PCollection that uses processing time as the element timestamp.
   PCollection<Long> unbounded = p.apply(CountingInput.unbounded());
   // Or, to create an unbounded source that uses a provided function to set the element timestamp.
   PCollection<Long> unboundedWithTimestamps =
       p.apply(CountingInput.unbounded().withTimestampFn(someFn));
  
   

• • Nested Class Summary

    Nested Classes 

    Modifier and Type	Class and Description
    static class	CountingSource.CounterMark The checkpoint for an unbounded CountingSource is simply the last value produced.

  • Method Summary

    All Methods Static Methods Concrete Methods Deprecated Methods 

    Modifier and Type	Method and Description
    static UnboundedSource<Long,CountingSource.CounterMark>	unbounded() Deprecated.  use CountingInput.unbounded() instead
    static UnboundedSource<Long,CountingSource.CounterMark>	unboundedWithTimestampFn(SerializableFunction<Long,Instant> timestampFn) Deprecated.  use CountingInput.unbounded() and call CountingInput.UnboundedCountingInput.withTimestampFn(SerializableFunction) instead
    static BoundedSource<Long>	upTo(long numElements) Deprecated.  use CountingInput.upTo(long) instead

    • Methods inherited from class java.lang.Object

      clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

• • Method Detail

    • upTo

      @Deprecated
      public static BoundedSource<Long> upTo(long numElements)

      Deprecated. use CountingInput.upTo(long) instead
      Creates a BoundedSource that will produce the specified number of elements, from 0 to numElements - 1.

    • unbounded

      @Deprecated
      public static UnboundedSource<Long,CountingSource.CounterMark> unbounded()

      Deprecated. use CountingInput.unbounded() instead
      Creates an UnboundedSource that will produce numbers starting from 0 up to Long.MAX_VALUE.

      After Long.MAX_VALUE, the source never produces more output. (In practice, this limit should never be reached.)

      Elements in the resulting PCollection<Long> will have timestamps corresponding to processing time at element generation, provided by Instant.now().

    • unboundedWithTimestampFn

      @Deprecated
      public static UnboundedSource<Long,CountingSource.CounterMark> unboundedWithTimestampFn(SerializableFunction<Long,Instant> timestampFn)

      Deprecated. use CountingInput.unbounded() and call CountingInput.UnboundedCountingInput.withTimestampFn(SerializableFunction) instead
      Creates an UnboundedSource that will produce numbers starting from 0 up to Long.MAX_VALUE, with element timestamps supplied by the specified function.

      After Long.MAX_VALUE, the source never produces more output. (In practice, this limit should never be reached.)

      Note that the timestamps produced by timestampFn may not decrease.