CountingInput (Google Cloud Dataflow SDK 1.9.1 API)

Google Cloud Dataflow SDK for Java, version 1.9.1

Warning: Dataflow SDK 1.x for Java is unsupported as of October 16, 2018. After August 12, 2020, you can use unsupported SDKs temporarily while you upgrade your pipelines. For more information, see Using an unsupported SDK.

For information on migrating from the 1.x SDK, see Migrating from Dataflow SDK 1.x for Java.

Note: If you are using Cloud Dataflow SDK version 2.x, see the Apache Beam API reference documentation.

com.google.cloud.dataflow.sdk.io

Class CountingInput

• java.lang.Object
• • com.google.cloud.dataflow.sdk.io.CountingInput

• ---

  
  

  public class CountingInput
  extends Object

  A PTransform that produces longs. When used to produce a bounded PCollection, CountingInput starts at 0 and counts up to a specified maximum. When used to produce an unbounded PCollection, it counts up to Long.MAX_VALUE and then never produces more output. (In practice, this limit should never be reached.)

  The bounded CountingInput is implemented based on OffsetBasedSource and OffsetBasedSource.OffsetBasedReader, so it performs efficient initial splitting and it supports dynamic work rebalancing.

  To produce a bounded PCollection<Long>, use upTo(long):

  
   Pipeline p = ...
   PTransform<PBegin, PCollection<Long>> producer = CountingInput.upTo(1000);
   PCollection<Long> bounded = p.apply(producer);
   

  To produce an unbounded PCollection<Long>, use unbounded(), calling CountingInput.UnboundedCountingInput.withTimestampFn(SerializableFunction) to provide values with timestamps other than Instant.now().

  
   Pipeline p = ...
  
   // To create an unbounded producer that uses processing time as the element timestamp.
   PCollection<Long> unbounded = p.apply(CountingInput.unbounded());
   // Or, to create an unbounded source that uses a provided function to set the element timestamp.
   PCollection<Long> unboundedWithTimestamps =
       p.apply(CountingInput.unbounded().withTimestampFn(someFn));
   

• • Nested Class Summary

    Nested Classes 

    Modifier and Type	Class and Description
    static class	CountingInput.BoundedCountingInput A PTransform that will produce a specified number of Longs starting from 0.
    static class	CountingInput.UnboundedCountingInput A PTransform that will produce numbers starting from 0 up to Long.MAX_VALUE.

  • Constructor Summary

    Constructors 

    Constructor and Description
    CountingInput()

  • Method Summary

    All Methods Static Methods Concrete Methods 

    Modifier and Type	Method and Description
    static CountingInput.UnboundedCountingInput	unbounded() Creates an CountingInput.UnboundedCountingInput that will produce numbers starting from 0 up to Long.MAX_VALUE.
    static CountingInput.BoundedCountingInput	upTo(long numElements) Creates a CountingInput.BoundedCountingInput that will produce the specified number of elements, from 0 to numElements - 1.

    • Methods inherited from class java.lang.Object

      clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

• • Constructor Detail

    • CountingInput

      public CountingInput()

  • Method Detail

    • upTo

      public static CountingInput.BoundedCountingInput upTo(long numElements)

      Creates a CountingInput.BoundedCountingInput that will produce the specified number of elements, from 0 to numElements - 1.

    • unbounded

      public static CountingInput.UnboundedCountingInput unbounded()

      Creates an CountingInput.UnboundedCountingInput that will produce numbers starting from 0 up to Long.MAX_VALUE.

      After Long.MAX_VALUE, the transform never produces more output. (In practice, this limit should never be reached.)

      Elements in the resulting PCollection<Long> will by default have timestamps corresponding to processing time at element generation, provided by Instant.now(). Use the transform returned by CountingInput.UnboundedCountingInput.withTimestampFn(SerializableFunction) to control the output timestamps.