Google Cloud Dataflow SDK for Java, version 1.9.1
Class DoFn.Context
- java.lang.Object
-
- com.google.cloud.dataflow.sdk.transforms.DoFn.Context
-
- Direct Known Subclasses:
- DoFn.ProcessContext
public abstract class DoFn.Context extends Object
Information accessible to all methods in thisDoFn
. Used primarily to output elements.
-
-
Constructor Summary
Constructors Constructor and Description Context()
-
Method Summary
All Methods Instance Methods Abstract Methods Concrete Methods Modifier and Type Method and Description protected abstract <AggInputT,AggOutputT>
Aggregator<AggInputT,AggOutputT>createAggregatorInternal(String name, Combine.CombineFn<AggInputT,?,AggOutputT> combiner)
Creates anAggregator
in theDoFn
context with the specified name and aggregation logic specified byCombine.CombineFn
.abstract PipelineOptions
getPipelineOptions()
abstract void
output(OutputT output)
Adds the given element to the main outputPCollection
.abstract void
outputWithTimestamp(OutputT output, Instant timestamp)
Adds the given element to the main outputPCollection
, with the given timestamp.protected void
setupDelegateAggregators()
Sets upAggregator
s created by theDoFn
so they are usable within this context.abstract <T> void
sideOutput(TupleTag<T> tag, T output)
Adds the given element to the side outputPCollection
with the given tag.abstract <T> void
sideOutputWithTimestamp(TupleTag<T> tag, T output, Instant timestamp)
Adds the given element to the specified side outputPCollection
, with the given timestamp.
-
-
-
Method Detail
-
getPipelineOptions
public abstract PipelineOptions getPipelineOptions()
Returns thePipelineOptions
specified with thePipelineRunner
invoking thisDoFn
. ThePipelineOptions
will be the default running viaDoFnTester
.
-
output
public abstract void output(OutputT output)
Adds the given element to the main outputPCollection
.Once passed to
output
the element should be considered immutable and not be modified in any way. It may be cached or retained by the Dataflow runtime or later steps in the pipeline, or used in other unspecified ways.If invoked from
processElement
, the output element will have the same timestamp and be in the same windows as the input element passed toprocessElement
.If invoked from
startBundle
orfinishBundle
, this will attempt to use theWindowFn
of the inputPCollection
to determine what windows the element should be in, throwing an exception if theWindowFn
attempts to access any information about the input element. The output element will have a timestamp of negative infinity.
-
outputWithTimestamp
public abstract void outputWithTimestamp(OutputT output, Instant timestamp)
Adds the given element to the main outputPCollection
, with the given timestamp.Once passed to
outputWithTimestamp
the element should not be modified in any way.If invoked from
processElement
, the timestamp must not be older than the input element's timestamp minusgetAllowedTimestampSkew
. The output element will be in the same windows as the input element.If invoked from
startBundle
orfinishBundle
, this will attempt to use theWindowFn
of the inputPCollection
to determine what windows the element should be in, throwing an exception if theWindowFn
attempts to access any information about the input element except for the timestamp.
-
sideOutput
public abstract <T> void sideOutput(TupleTag<T> tag, T output)
Adds the given element to the side outputPCollection
with the given tag.Once passed to
sideOutput
the element should not be modified in any way.The caller of
ParDo
useswithOutputTags
to specify the tags of side outputs that it consumes. Non-consumed side outputs, e.g., outputs for monitoring purposes only, don't necessarily need to be specified.The output element will have the same timestamp and be in the same windows as the input element passed to
processElement
.If invoked from
startBundle
orfinishBundle
, this will attempt to use theWindowFn
of the inputPCollection
to determine what windows the element should be in, throwing an exception if theWindowFn
attempts to access any information about the input element. The output element will have a timestamp of negative infinity.
-
sideOutputWithTimestamp
public abstract <T> void sideOutputWithTimestamp(TupleTag<T> tag, T output, Instant timestamp)
Adds the given element to the specified side outputPCollection
, with the given timestamp.Once passed to
sideOutputWithTimestamp
the element should not be modified in any way.If invoked from
processElement
, the timestamp must not be older than the input element's timestamp minusgetAllowedTimestampSkew
. The output element will be in the same windows as the input element.If invoked from
startBundle
orfinishBundle
, this will attempt to use theWindowFn
of the inputPCollection
to determine what windows the element should be in, throwing an exception if theWindowFn
attempts to access any information about the input element except for the timestamp.
-
createAggregatorInternal
@Experimental(value=AGGREGATOR) protected abstract <AggInputT,AggOutputT> Aggregator<AggInputT,AggOutputT> createAggregatorInternal(String name, Combine.CombineFn<AggInputT,?,AggOutputT> combiner)
Creates anAggregator
in theDoFn
context with the specified name and aggregation logic specified byCombine.CombineFn
.For internal use only.
- Parameters:
name
- the name of the aggregatorcombiner
- theCombine.CombineFn
to use in the aggregator- Returns:
- an aggregator for the provided name and
Combine.CombineFn
in this context
-
setupDelegateAggregators
@Experimental(value=AGGREGATOR) protected final void setupDelegateAggregators()
Sets upAggregator
s created by theDoFn
so they are usable within this context.This method should be called by runners before
DoFn.startBundle(com.google.cloud.dataflow.sdk.transforms.DoFn<InputT, OutputT>.Context)
is executed.
-
-