CombineFns (Google Cloud Dataflow SDK 1.9.1 API)

Google Cloud Dataflow SDK for Java, version 1.9.1

Warning: Dataflow SDK 1.x for Java is unsupported as of October 16, 2018. After August 12, 2020, you can use unsupported SDKs temporarily while you upgrade your pipelines. For more information, see Using an unsupported SDK.

For information on migrating from the 1.x SDK, see Migrating from Dataflow SDK 1.x for Java.

Note: If you are using Cloud Dataflow SDK version 2.x, see the Apache Beam API reference documentation.

com.google.cloud.dataflow.sdk.transforms

Class CombineFns

• java.lang.Object
• • com.google.cloud.dataflow.sdk.transforms.CombineFns

• ---

  
  

  public class CombineFns
  extends Object

  Static utility methods that create combine function instances.

• • Nested Class Summary

    Nested Classes 

    Modifier and Type	Class and Description
    static class	CombineFns.CoCombineResult A tuple of outputs produced by a composed combine functions.
    static class	CombineFns.ComposeCombineFnBuilder A builder class to construct a composed CombineFnBase.GlobalCombineFn.
    static class	CombineFns.ComposedCombineFn<DataT> A composed Combine.CombineFn that applies multiple CombineFns.
    static class	CombineFns.ComposedCombineFnWithContext<DataT> A composed CombineWithContext.CombineFnWithContext that applies multiple CombineFnWithContexts.
    static class	CombineFns.ComposedKeyedCombineFn<DataT,K> A composed Combine.KeyedCombineFn that applies multiple KeyedCombineFns.
    static class	CombineFns.ComposedKeyedCombineFnWithContext<DataT,K> A composed CombineWithContext.KeyedCombineFnWithContext that applies multiple KeyedCombineFnWithContexts.
    static class	CombineFns.ComposeKeyedCombineFnBuilder A builder class to construct a composed CombineFnBase.PerKeyCombineFn.

  • Constructor Summary

    Constructors 

    Constructor and Description
    CombineFns()

  • Method Summary

    All Methods Static Methods Concrete Methods 

    Modifier and Type	Method and Description
    static CombineFns.ComposeCombineFnBuilder	compose() Returns a CombineFns.ComposeCombineFnBuilder to construct a composed CombineFnBase.GlobalCombineFn.
    static CombineFns.ComposeKeyedCombineFnBuilder	composeKeyed() Returns a CombineFns.ComposeKeyedCombineFnBuilder to construct a composed CombineFnBase.PerKeyCombineFn.

    • Methods inherited from class java.lang.Object

      clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

• • Constructor Detail

    • CombineFns

      public CombineFns()

  • Method Detail

    • composeKeyed

      public static CombineFns.ComposeKeyedCombineFnBuilder composeKeyed()

      Returns a CombineFns.ComposeKeyedCombineFnBuilder to construct a composed CombineFnBase.PerKeyCombineFn.

      The same TupleTag cannot be used in a composition multiple times.

      Example:

      
       PCollection<KV<K, Integer>> latencies = ...;
      
       TupleTag<Integer> maxLatencyTag = new TupleTag<Integer>();
       TupleTag<Double> meanLatencyTag = new TupleTag<Double>();
      
       SimpleFunction<Integer, Integer> identityFn =
           new SimpleFunction<Integer, Integer>() {
             public Integer apply(Integer input) {
                 return input;
             }};
       PCollection<KV<K, CoCombineResult>> maxAndMean = latencies.apply(
           Combine.perKey(
               CombineFns.composeKeyed()
                  .with(identityFn, new MaxIntegerFn(), maxLatencyTag)
                  .with(identityFn, new MeanFn<Integer>(), meanLatencyTag)));
      
       PCollection<T> finalResultCollection = maxAndMean
           .apply(ParDo.of(
               new DoFn<KV<K, CoCombineResult>, T>() {
                 public void processElement(ProcessContext c) throws Exception {
                   KV<K, CoCombineResult> e = c.element();
                   Integer maxLatency = e.getValue().get(maxLatencyTag);
                   Double meanLatency = e.getValue().get(meanLatencyTag);
                   .... Do Something ....
                   c.output(...some T...);
                 }
               }));
       

    • compose

      public static CombineFns.ComposeCombineFnBuilder compose()

      Returns a CombineFns.ComposeCombineFnBuilder to construct a composed CombineFnBase.GlobalCombineFn.

      The same TupleTag cannot be used in a composition multiple times.

      Example:

      
       PCollection<Integer> globalLatencies = ...;
      
       TupleTag<Integer> maxLatencyTag = new TupleTag<Integer>();
       TupleTag<Double> meanLatencyTag = new TupleTag<Double>();
      
       SimpleFunction<Integer, Integer> identityFn =
           new SimpleFunction<Integer, Integer>() {
             public Integer apply(Integer input) {
                 return input;
             }};
       PCollection<CoCombineResult> maxAndMean = globalLatencies.apply(
           Combine.globally(
               CombineFns.compose()
                  .with(identityFn, new MaxIntegerFn(), maxLatencyTag)
                  .with(identityFn, new MeanFn<Integer>(), meanLatencyTag)));
      
       PCollection<T> finalResultCollection = maxAndMean
           .apply(ParDo.of(
               new DoFn<CoCombineResult, T>() {
                 public void processElement(ProcessContext c) throws Exception {
                   CoCombineResult e = c.element();
                   Integer maxLatency = e.get(maxLatencyTag);
                   Double meanLatency = e.get(meanLatencyTag);
                   .... Do Something ....
                   c.output(...some T...);
                 }
               }));