PubsubUnboundedSource (Google Cloud Dataflow SDK 1.9.1 API)

Google Cloud Dataflow SDK for Java, version 1.9.1

Note: If you are using Cloud Dataflow SDK version 2.x, see the Apache Beam API reference documentation.

com.google.cloud.dataflow.sdk.io

Class PubsubUnboundedSource<T>

• java.lang.Object
• • com.google.cloud.dataflow.sdk.transforms.PTransform<PBegin,PCollection<T>>
  • • com.google.cloud.dataflow.sdk.io.PubsubUnboundedSource<T>

• All Implemented Interfaces:

  HasDisplayData, Serializable

  ---

  
  

  public class PubsubUnboundedSource<T>
  extends PTransform<PBegin,PCollection<T>>

  A PTransform which streams messages from Pubsub.

  • The underlying implementation in an UnboundedSource which receives messages in batches and hands them out one at a time.
  • The watermark (either in Pubsub processing time or custom timestamp time) is estimated by keeping track of the minimum of the last minutes worth of messages. This assumes Pubsub delivers the oldest (in Pubsub processing time) available message at least once a minute, and that custom timestamps are 'mostly' monotonic with Pubsub processing time. Unfortunately both of those assumptions are fragile. Thus the estimated watermark may get ahead of the 'true' watermark and cause some messages to be late.
  • Checkpoints are used both to ACK received messages back to Pubsub (so that they may be retired on the Pubsub end), and to NACK already consumed messages should a checkpoint need to be restored (so that Pubsub will resend those messages promptly).
  • The backlog is determined by each reader using the messages which have been pulled from Pubsub but not yet consumed downstream. The backlog does not take account of any messages queued by Pubsub for the subscription. Unfortunately there is currently no API to determine the size of the Pubsub queue's backlog.
  • The subscription must already exist.
  • The subscription timeout is read whenever a reader is started. However it is not checked thereafter despite the timeout being user-changeable on-the-fly.
  • We log vital stats every 30 seconds.
  • Though some background threads may be used by the underlying transport all Pubsub calls are blocking. We rely on the underlying runner to allow multiple UnboundedSource.UnboundedReader instances to execute concurrently and thus hide latency.

  NOTE: This is not the implementation used when running on the Google Cloud Dataflow service.

  See Also:

  Serialized Form

• • Field Summary

    • Fields inherited from class com.google.cloud.dataflow.sdk.transforms.PTransform

      name

  • Constructor Summary

    Constructors 

    Constructor and Description

    PubsubUnboundedSource(com.google.cloud.dataflow.sdk.util.PubsubClient.PubsubClientFactory pubsubFactory, ValueProvider<com.google.cloud.dataflow.sdk.util.PubsubClient.ProjectPath> project, ValueProvider<com.google.cloud.dataflow.sdk.util.PubsubClient.TopicPath> topic, ValueProvider<com.google.cloud.dataflow.sdk.util.PubsubClient.SubscriptionPath> subscription, Coder<T> elementCoder, String timestampLabel, String idLabel) Construct an unbounded source to consume from the Pubsub subscription.

  • Method Summary

    All Methods Instance Methods Concrete Methods 

    Modifier and Type	Method and Description
    PCollection<T>	apply(PBegin input) Applies this PTransform on the given InputT, and returns its Output.
    Coder<T>	getElementCoder()
    String	getIdLabel()
    com.google.cloud.dataflow.sdk.util.PubsubClient.ProjectPath	getProject()
    com.google.cloud.dataflow.sdk.util.PubsubClient.SubscriptionPath	getSubscription()
    ValueProvider<com.google.cloud.dataflow.sdk.util.PubsubClient.SubscriptionPath>	getSubscriptionProvider()
    String	getTimestampLabel()
    com.google.cloud.dataflow.sdk.util.PubsubClient.TopicPath	getTopic()
    ValueProvider<com.google.cloud.dataflow.sdk.util.PubsubClient.TopicPath>	getTopicProvider()

    • Methods inherited from class com.google.cloud.dataflow.sdk.transforms.PTransform

      getDefaultOutputCoder, getDefaultOutputCoder, getDefaultOutputCoder, getKindString, getName, populateDisplayData, toString, validate

    • Methods inherited from class java.lang.Object

      clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

• • Constructor Detail

    • PubsubUnboundedSource

      public PubsubUnboundedSource(com.google.cloud.dataflow.sdk.util.PubsubClient.PubsubClientFactory pubsubFactory,
                                   @Nullable
                                   ValueProvider<com.google.cloud.dataflow.sdk.util.PubsubClient.ProjectPath> project,
                                   @Nullable
                                   ValueProvider<com.google.cloud.dataflow.sdk.util.PubsubClient.TopicPath> topic,
                                   @Nullable
                                   ValueProvider<com.google.cloud.dataflow.sdk.util.PubsubClient.SubscriptionPath> subscription,
                                   Coder<T> elementCoder,
                                   @Nullable
                                   String timestampLabel,
                                   @Nullable
                                   String idLabel)

      Construct an unbounded source to consume from the Pubsub subscription.

  • Method Detail

    • getElementCoder

      public Coder<T> getElementCoder()

    • getProject

      @Nullable
      public com.google.cloud.dataflow.sdk.util.PubsubClient.ProjectPath getProject()

    • getTopic

      @Nullable
      public com.google.cloud.dataflow.sdk.util.PubsubClient.TopicPath getTopic()

    • getTopicProvider

      @Nullable
      public ValueProvider<com.google.cloud.dataflow.sdk.util.PubsubClient.TopicPath> getTopicProvider()

    • getSubscription

      @Nullable
      public com.google.cloud.dataflow.sdk.util.PubsubClient.SubscriptionPath getSubscription()

    • getSubscriptionProvider

      @Nullable
      public ValueProvider<com.google.cloud.dataflow.sdk.util.PubsubClient.SubscriptionPath> getSubscriptionProvider()

    • getTimestampLabel

      @Nullable
      public String getTimestampLabel()

    • getIdLabel

      @Nullable
      public String getIdLabel()

    • apply

      public PCollection<T> apply(PBegin input)

      Description copied from class: PTransform
      Applies this PTransform on the given InputT, and returns its Output.

      Composite transforms, which are defined in terms of other transforms, should return the output of one of the composed transforms. Non-composite transforms, which do not apply any transforms internally, should return a new unbound output and register evaluators (via backend-specific registration methods).

      The default implementation throws an exception. A derived class must either implement apply, or else each runner must supply a custom implementation via PipelineRunner.apply(com.google.cloud.dataflow.sdk.transforms.PTransform<InputT, OutputT>, InputT).

      Overrides:

      apply in class PTransform<PBegin,PCollection<T>>