CombineFns (Google Cloud Dataflow SDK 1.9.1 API)

Google Cloud Dataflow SDK for Java, version 1.9.1

com.google.cloud.dataflow.sdk.transforms

Class CombineFns



  • public class CombineFns
    extends Object
    Static utility methods that create combine function instances.
    • Constructor Detail

      • CombineFns

        public CombineFns()
    • Method Detail

      • composeKeyed

        public static CombineFns.ComposeKeyedCombineFnBuilder composeKeyed()
        Returns a CombineFns.ComposeKeyedCombineFnBuilder to construct a composed CombineFnBase.PerKeyCombineFn.

        The same TupleTag cannot be used in a composition multiple times.

        Example:

        
         PCollection<KV<K, Integer>> latencies = ...;
        
         TupleTag<Integer> maxLatencyTag = new TupleTag<Integer>();
         TupleTag<Double> meanLatencyTag = new TupleTag<Double>();
        
         SimpleFunction<Integer, Integer> identityFn =
             new SimpleFunction<Integer, Integer>() {
               public Integer apply(Integer input) {
                   return input;
               }};
         PCollection<KV<K, CoCombineResult>> maxAndMean = latencies.apply(
             Combine.perKey(
                 CombineFns.composeKeyed()
                    .with(identityFn, new MaxIntegerFn(), maxLatencyTag)
                    .with(identityFn, new MeanFn<Integer>(), meanLatencyTag)));
        
         PCollection<T> finalResultCollection = maxAndMean
             .apply(ParDo.of(
                 new DoFn<KV<K, CoCombineResult>, T>() {
                   public void processElement(ProcessContext c) throws Exception {
                     KV<K, CoCombineResult> e = c.element();
                     Integer maxLatency = e.getValue().get(maxLatencyTag);
                     Double meanLatency = e.getValue().get(meanLatencyTag);
                     .... Do Something ....
                     c.output(...some T...);
                   }
                 }));
         
      • compose

        public static CombineFns.ComposeCombineFnBuilder compose()
        Returns a CombineFns.ComposeCombineFnBuilder to construct a composed CombineFnBase.GlobalCombineFn.

        The same TupleTag cannot be used in a composition multiple times.

        Example:

        
         PCollection<Integer> globalLatencies = ...;
        
         TupleTag<Integer> maxLatencyTag = new TupleTag<Integer>();
         TupleTag<Double> meanLatencyTag = new TupleTag<Double>();
        
         SimpleFunction<Integer, Integer> identityFn =
             new SimpleFunction<Integer, Integer>() {
               public Integer apply(Integer input) {
                   return input;
               }};
         PCollection<CoCombineResult> maxAndMean = globalLatencies.apply(
             Combine.globally(
                 CombineFns.compose()
                    .with(identityFn, new MaxIntegerFn(), maxLatencyTag)
                    .with(identityFn, new MeanFn<Integer>(), meanLatencyTag)));
        
         PCollection<T> finalResultCollection = maxAndMean
             .apply(ParDo.of(
                 new DoFn<CoCombineResult, T>() {
                   public void processElement(ProcessContext c) throws Exception {
                     CoCombineResult e = c.element();
                     Integer maxLatency = e.get(maxLatencyTag);
                     Double meanLatency = e.get(meanLatencyTag);
                     .... Do Something ....
                     c.output(...some T...);
                   }
                 }));
         


Monitor your resources on the go

Get the Google Cloud Console app to help you manage your projects.

Send feedback about...

Cloud Dataflow