Partition (Google Cloud Dataflow SDK 1.9.1 API)

Google Cloud Dataflow SDK for Java, version 1.9.1

com.google.cloud.dataflow.sdk.transforms

Class Partition<T>

  • Type Parameters:
    T - the type of the elements of the input and output PCollections
    All Implemented Interfaces:
    HasDisplayData, Serializable

    public class Partition<T>
extends PTransform<PCollection<T>,PCollectionList<T>>
    Partition takes a PCollection<T> and a PartitionFn, uses the PartitionFn to split the elements of the input PCollection into N partitions, and returns a PCollectionList<T> that bundles N PCollection<T>s containing the split elements.

    Example of use: 

     
 PCollection<Student> students = ...;
 // Split students up into 10 partitions, by percentile:
 PCollectionList<Student> studentsByPercentile =
     students.apply(Partition.of(10, new PartitionFn<Student>() {
         public int partitionFor(Student student, int numPartitions) {
             return student.getPercentile()  // 0..99
                  * numPartitions / 100;
         }}))
 for (int i = 0; i < 10; i++) {
   PCollection<Student> partition = studentsByPercentile.get(i);
   ...
 }

    By default, the Coder of each of the PCollections in the output PCollectionList is the same as the Coder of the input PCollection.

    Each output element has the same timestamp and is in the same windows as its corresponding input element, and each output PCollection has the same WindowFn associated with it as the input.

    See Also:
    Serialized Form

    • Method Detail

      • of

        public static <T> Partition<T> of(int numPartitions,
                                  Partition.PartitionFn<? super T> partitionFn)
        Returns a new Partition PTransform that divides its input PCollection into the given number of partitions, using the given partitioning function.
        Parameters:
        numPartitions - the number of partitions to divide the input PCollection into
        partitionFn - the function to invoke on each element to choose its output partition
        Throws:
        IllegalArgumentException - if numPartitions <= 0

      • populateDisplayData

        public void populateDisplayData(DisplayData.Builder builder)
        Description copied from class: PTransform
        Register display data for the given transform or component.

        populateDisplayData(DisplayData.Builder) is invoked by Pipeline runners to collect display data via DisplayData.from(HasDisplayData). Implementations may call super.populateDisplayData(builder) in order to register display data in the current namespace, but should otherwise use subcomponent.populateDisplayData(builder) to use the namespace of the subcomponent.

        By default, does not register any display data. Implementors may override this method to provide their own display data.

        Specified by:
        populateDisplayData in interface HasDisplayData
        Overrides:
        populateDisplayData in class PTransform<PCollection<T>,PCollectionList<T>>
        Parameters:
        builder - The builder to populate with display data.
        See Also:
        HasDisplayData