Google Cloud Dataflow SDK for Java, version 1.9.1
com.google.cloud.dataflow.sdk.transforms
Class CombineFns
- java.lang.Object
-
- com.google.cloud.dataflow.sdk.transforms.CombineFns
-
public class CombineFns extends Object
Static utility methods that create combine function instances.
-
-
Nested Class Summary
Nested Classes Modifier and Type Class and Description static class
CombineFns.CoCombineResult
A tuple of outputs produced by a composed combine functions.static class
CombineFns.ComposeCombineFnBuilder
A builder class to construct a composedCombineFnBase.GlobalCombineFn
.static class
CombineFns.ComposedCombineFn<DataT>
A composedCombine.CombineFn
that applies multipleCombineFns
.static class
CombineFns.ComposedCombineFnWithContext<DataT>
A composedCombineWithContext.CombineFnWithContext
that applies multipleCombineFnWithContexts
.static class
CombineFns.ComposedKeyedCombineFn<DataT,K>
A composedCombine.KeyedCombineFn
that applies multipleKeyedCombineFns
.static class
CombineFns.ComposedKeyedCombineFnWithContext<DataT,K>
A composedCombineWithContext.KeyedCombineFnWithContext
that applies multipleKeyedCombineFnWithContexts
.static class
CombineFns.ComposeKeyedCombineFnBuilder
A builder class to construct a composedCombineFnBase.PerKeyCombineFn
.
-
Constructor Summary
Constructors Constructor and Description CombineFns()
-
Method Summary
All Methods Static Methods Concrete Methods Modifier and Type Method and Description static CombineFns.ComposeCombineFnBuilder
compose()
Returns aCombineFns.ComposeCombineFnBuilder
to construct a composedCombineFnBase.GlobalCombineFn
.static CombineFns.ComposeKeyedCombineFnBuilder
composeKeyed()
Returns aCombineFns.ComposeKeyedCombineFnBuilder
to construct a composedCombineFnBase.PerKeyCombineFn
.
-
-
-
Method Detail
-
composeKeyed
public static CombineFns.ComposeKeyedCombineFnBuilder composeKeyed()
Returns aCombineFns.ComposeKeyedCombineFnBuilder
to construct a composedCombineFnBase.PerKeyCombineFn
.The same
TupleTag
cannot be used in a composition multiple times.Example:
PCollection<KV<K, Integer>> latencies = ...; TupleTag<Integer> maxLatencyTag = new TupleTag<Integer>(); TupleTag<Double> meanLatencyTag = new TupleTag<Double>(); SimpleFunction<Integer, Integer> identityFn = new SimpleFunction<Integer, Integer>() { public Integer apply(Integer input) { return input; }}; PCollection<KV<K, CoCombineResult>> maxAndMean = latencies.apply( Combine.perKey( CombineFns.composeKeyed() .with(identityFn, new MaxIntegerFn(), maxLatencyTag) .with(identityFn, new MeanFn<Integer>(), meanLatencyTag))); PCollection<T> finalResultCollection = maxAndMean .apply(ParDo.of( new DoFn<KV<K, CoCombineResult>, T>() { public void processElement(ProcessContext c) throws Exception { KV<K, CoCombineResult> e = c.element(); Integer maxLatency = e.getValue().get(maxLatencyTag); Double meanLatency = e.getValue().get(meanLatencyTag); .... Do Something .... c.output(...some T...); } }));
-
compose
public static CombineFns.ComposeCombineFnBuilder compose()
Returns aCombineFns.ComposeCombineFnBuilder
to construct a composedCombineFnBase.GlobalCombineFn
.The same
TupleTag
cannot be used in a composition multiple times.Example:
PCollection<Integer> globalLatencies = ...; TupleTag<Integer> maxLatencyTag = new TupleTag<Integer>(); TupleTag<Double> meanLatencyTag = new TupleTag<Double>(); SimpleFunction<Integer, Integer> identityFn = new SimpleFunction<Integer, Integer>() { public Integer apply(Integer input) { return input; }}; PCollection<CoCombineResult> maxAndMean = globalLatencies.apply( Combine.globally( CombineFns.compose() .with(identityFn, new MaxIntegerFn(), maxLatencyTag) .with(identityFn, new MeanFn<Integer>(), meanLatencyTag))); PCollection<T> finalResultCollection = maxAndMean .apply(ParDo.of( new DoFn<CoCombineResult, T>() { public void processElement(ProcessContext c) throws Exception { CoCombineResult e = c.element(); Integer maxLatency = e.get(maxLatencyTag); Double meanLatency = e.get(meanLatencyTag); .... Do Something .... c.output(...some T...); } }));
-
-