PCollection (Google Cloud Dataflow SDK 1.9.1 API)

Google Cloud Dataflow SDK for Java, version 1.9.1

Warning: Dataflow SDK 1.x for Java is unsupported as of October 16, 2018. After August 12, 2020, Dataflow will not run jobs using Dataflow 1.x and below. See Migrating from Dataflow SDK 1.x for Java for migration guidance.

Note: If you are using Cloud Dataflow SDK version 2.x, see the Apache Beam API reference documentation.

com.google.cloud.dataflow.sdk.values

Class PCollection<T>

• java.lang.Object
• • com.google.cloud.dataflow.sdk.values.POutputValueBase
  • • com.google.cloud.dataflow.sdk.values.PValueBase
    • • com.google.cloud.dataflow.sdk.values.TypedPValue<T>
      • • com.google.cloud.dataflow.sdk.values.PCollection<T>

• Type Parameters:

  T - the type of the elements of this PCollection

  All Implemented Interfaces:

  PInput, POutput, PValue

  ---

  
  

  public class PCollection<T>
  extends TypedPValue<T>

  A PCollection<T> is an immutable collection of values of type T. A PCollection can contain either a bounded or unbounded number of elements. Bounded and unbounded PCollections are produced as the output of PTransforms (including root PTransforms like Read and Create), and can be passed as the inputs of other PTransforms.

  Some root transforms produce bounded PCollections and others produce unbounded ones. For example, TextIO.Read reads a static set of files, so it produces a bounded PCollection. PubsubIO.Read, on the other hand, receives a potentially infinite stream of Pubsub messages, so it produces an unbounded PCollection.

  Each element in a PCollection may have an associated implicit timestamp. Readers assign timestamps to elements when they create PCollections, and other PTransforms propagate these timestamps from their input to their output. For example, PubsubIO.Read assigns pubsub message timestamps to elements, and TextIO.Read assigns the default value BoundedWindow.TIMESTAMP_MIN_VALUE to elements. User code can explicitly assign timestamps to elements with DoFn.Context.outputWithTimestamp(OutputT, org.joda.time.Instant).

  Additionally, a PCollection has an associated WindowFn and each element is assigned to a set of windows. By default, the windowing function is GlobalWindows and all elements are assigned into a single default window. This default can be overridden with the Window PTransform.

  See the individual PTransform subclasses for specific information on how they propagate timestamps and windowing.

• • Nested Class Summary

    Nested Classes 

    Modifier and Type	Class and Description
    static class	PCollection.IsBounded The enumeration of cases for whether a PCollection is bounded.

  • Method Summary

    All Methods Static Methods Instance Methods Concrete Methods 

    Modifier and Type	Method and Description
    <OutputT extends POutput> OutputT	apply(PTransform<? super PCollection<T>,OutputT> t) Like apply(String, PTransform) but defaulting to the name of the PTransform.
    <OutputT extends POutput> OutputT	apply(String name, PTransform<? super PCollection<T>,OutputT> t) Applies the given PTransform to this input PCollection, using name to identify this specific application of the transform.
    static <T> PCollection<T>	createPrimitiveOutputInternal(Pipeline pipeline, com.google.cloud.dataflow.sdk.util.WindowingStrategy<?,?> windowingStrategy, PCollection.IsBounded isBounded) Creates and returns a new PCollection for a primitive output.
    Coder<T>	getCoder() Returns the Coder used by this PCollection to encode and decode the values stored in it.
    String	getName() Returns the name of this PCollection.
    com.google.cloud.dataflow.sdk.util.WindowingStrategy<?,?>	getWindowingStrategy() Returns the WindowingStrategy of this PCollection.
    PCollection.IsBounded	isBounded()
    PCollection<T>	setCoder(Coder<T> coder) Sets the Coder used by this PCollection to encode and decode the values stored in it.
    PCollection<T>	setIsBoundedInternal(PCollection.IsBounded isBounded) Sets the PCollection.IsBounded of this PCollection.
    PCollection<T>	setName(String name) Sets the name of this PCollection.
    PCollection<T>	setTypeDescriptorInternal(TypeDescriptor<T> typeDescriptor) Sets the TypeDescriptor<T> for this PCollection<T>.
    PCollection<T>	setWindowingStrategyInternal(com.google.cloud.dataflow.sdk.util.WindowingStrategy<?,?> windowingStrategy) Sets the WindowingStrategy of this PCollection.

    • Methods inherited from class com.google.cloud.dataflow.sdk.values.TypedPValue

      finishSpecifying, getTypeDescriptor

    • Methods inherited from class com.google.cloud.dataflow.sdk.values.PValueBase

      expand, getKindString, isFinishedSpecifyingInternal, recordAsOutput, recordAsOutput, toString

    • Methods inherited from class com.google.cloud.dataflow.sdk.values.POutputValueBase

      finishSpecifyingOutput, getPipeline, getProducingTransformInternal

    • Methods inherited from class java.lang.Object

      clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

    • Methods inherited from interface com.google.cloud.dataflow.sdk.values.PValue

      getProducingTransformInternal

    • Methods inherited from interface com.google.cloud.dataflow.sdk.values.POutput

      expand, finishSpecifyingOutput, getPipeline, recordAsOutput

    • Methods inherited from interface com.google.cloud.dataflow.sdk.values.PInput

      expand, getPipeline

• • Method Detail

    • getName

      public String getName()

      Returns the name of this PCollection.

      By default, the name of a PCollection is based on the name of the PTransform that produces it. It can be specified explicitly by calling setName(java.lang.String).

      Specified by:

      getName in interface PValue

      Overrides:

      getName in class PValueBase

      Throws:

      IllegalStateException - if the name hasn't been set yet

    • setName

      public PCollection<T> setName(String name)

      Sets the name of this PCollection. Returns this.

      Overrides:

      setName in class PValueBase

      Throws:

      IllegalStateException - if this PCollection has already been finalized and may no longer be set. Once apply(com.google.cloud.dataflow.sdk.transforms.PTransform<? super com.google.cloud.dataflow.sdk.values.PCollection<T>, OutputT>) has been called, this will be the case.

    • getCoder

      public Coder<T> getCoder()

      Returns the Coder used by this PCollection to encode and decode the values stored in it.

      Overrides:

      getCoder in class TypedPValue<T>

      Throws:

      IllegalStateException - if the Coder hasn't been set, and couldn't be inferred.

    • setCoder

      public PCollection<T> setCoder(Coder<T> coder)

      Sets the Coder used by this PCollection to encode and decode the values stored in it. Returns this.

      Overrides:

      setCoder in class TypedPValue<T>

      Throws:

      IllegalStateException - if this PCollection has already been finalized and may no longer be set. Once apply(com.google.cloud.dataflow.sdk.transforms.PTransform<? super com.google.cloud.dataflow.sdk.values.PCollection<T>, OutputT>) has been called, this will be the case.

    • apply

      public <OutputT extends POutput> OutputT apply(PTransform<? super PCollection<T>,OutputT> t)

      Like apply(String, PTransform) but defaulting to the name of the PTransform.

      Returns:

      the output of the applied PTransform

    • apply

      public <OutputT extends POutput> OutputT apply(String name,
                                                     PTransform<? super PCollection<T>,OutputT> t)

      Applies the given PTransform to this input PCollection, using name to identify this specific application of the transform. This name is used in various places, including the monitoring UI, logging, and to stably identify this application node in the job graph.

      Returns:

      the output of the applied PTransform

    • getWindowingStrategy

      public com.google.cloud.dataflow.sdk.util.WindowingStrategy<?,?> getWindowingStrategy()

      Returns the WindowingStrategy of this PCollection.

    • isBounded

      public PCollection.IsBounded isBounded()

    • setTypeDescriptorInternal

      public PCollection<T> setTypeDescriptorInternal(TypeDescriptor<T> typeDescriptor)

      Sets the TypeDescriptor<T> for this PCollection<T>. This may allow the enclosing PCollectionTuple, PCollectionList, or PTransform<?, PCollection<T>>, etc., to provide more detailed reflective information.

      Overrides:

      setTypeDescriptorInternal in class TypedPValue<T>

    • setWindowingStrategyInternal

      public PCollection<T> setWindowingStrategyInternal(com.google.cloud.dataflow.sdk.util.WindowingStrategy<?,?> windowingStrategy)

      Sets the WindowingStrategy of this PCollection.

      For use by primitive transformations only.

    • setIsBoundedInternal

      public PCollection<T> setIsBoundedInternal(PCollection.IsBounded isBounded)

      Sets the PCollection.IsBounded of this PCollection.

      For use by internal transformations only.

    • createPrimitiveOutputInternal

      public static <T> PCollection<T> createPrimitiveOutputInternal(Pipeline pipeline,
                                                                     com.google.cloud.dataflow.sdk.util.WindowingStrategy<?,?> windowingStrategy,
                                                                     PCollection.IsBounded isBounded)

      Creates and returns a new PCollection for a primitive output.

      For use by primitive transformations only.