Google Cloud Dataflow SDK for Java, version 1.9.1
com.google.cloud.dataflow.sdk.runners
Interface DataflowPipelineTranslator.TranslationContext
-
- Enclosing class:
- DataflowPipelineTranslator
public static interface DataflowPipelineTranslator.TranslationContext
The interface provided to registered callbacks for interacting with theDataflowPipelineRunner
, including reading and writing the values ofPCollection
s and side inputs (PCollectionView
s).
-
-
Method Summary
All Methods Instance Methods Abstract Methods Modifier and Type Method and Description void
addCollectionToSingletonOutput(String name, PValue inputValue, PValue outputValue)
Adds an output with the given name to the previously added CollectionToSingleton Dataflow step, consuming the specified inputPValue
and producing the specified outputPValue
.void
addEncodingInput(Coder<?> value)
Sets the encoding for the current Dataflow step.void
addInput(String name, Boolean value)
Adds an input with the given name and value to the current Dataflow step.void
addInput(String name, List<? extends Map<String,Object>> elements)
Adds an input that is a list of objects.void
addInput(String name, Long value)
Adds an input with the given name and value to the current Dataflow step.void
addInput(String name, Map<String,Object> elements)
Adds an input that is a dictionary of strings to objects.void
addInput(String name, PInput value)
Adds an input with the given name to the previously added Dataflow step, coming from the specified input PValue.void
addInput(String name, String value)
Adds an input with the given name and value to the current Dataflow step.void
addOutput(String name, PValue value)
Adds an output with the given name to the previously added Dataflow step, producing the specified outputPValue
, including itsCoder
if aTypedPValue
.void
addStep(PTransform<?,?> transform, String type)
Adds a step to the Dataflow workflow for the given transform, with the given Dataflow step type.void
addStep(PTransform<?,? extends PValue> transform, com.google.api.services.dataflow.model.Step step)
Adds a pre-defined step to the Dataflow workflow.void
addValueOnlyOutput(String name, PValue value)
Adds an output with the given name to the previously added Dataflow step, producing the specified outputPValue
, including itsCoder
if aTypedPValue
.com.google.cloud.dataflow.sdk.util.OutputReference
asOutputReference(PValue value)
Encode a PValue reference as an output reference.String
getFullName(PTransform<?,?> transform)
Returns the full name of the currently being translated transform.<InputT extends PInput>
InputTgetInput(PTransform<InputT,?> transform)
Returns the input of the currently being translated transform.<OutputT extends POutput>
OutputTgetOutput(PTransform<?,OutputT> transform)
Returns the output of the currently being translated transform.DataflowPipelineOptions
getPipelineOptions()
Returns the configured pipeline options.
-
-
-
Method Detail
-
getPipelineOptions
DataflowPipelineOptions getPipelineOptions()
Returns the configured pipeline options.
-
getInput
<InputT extends PInput> InputT getInput(PTransform<InputT,?> transform)
Returns the input of the currently being translated transform.
-
getOutput
<OutputT extends POutput> OutputT getOutput(PTransform<?,OutputT> transform)
Returns the output of the currently being translated transform.
-
getFullName
String getFullName(PTransform<?,?> transform)
Returns the full name of the currently being translated transform.
-
addStep
void addStep(PTransform<?,?> transform, String type)
Adds a step to the Dataflow workflow for the given transform, with the given Dataflow step type. This step becomes "current" for the purpose ofaddInput(java.lang.String, java.lang.Boolean)
andaddOutput(java.lang.String, com.google.cloud.dataflow.sdk.values.PValue)
.
-
addStep
void addStep(PTransform<?,? extends PValue> transform, com.google.api.services.dataflow.model.Step step)
Adds a pre-defined step to the Dataflow workflow. The given PTransform should be consistent with the Step, in terms of input, output and coder types.This is a low-level operation, when using this method it is up to the caller to ensure that names do not collide.
-
addEncodingInput
void addEncodingInput(Coder<?> value)
Sets the encoding for the current Dataflow step.
-
addInput
void addInput(String name, Boolean value)
Adds an input with the given name and value to the current Dataflow step.
-
addInput
void addInput(String name, String value)
Adds an input with the given name and value to the current Dataflow step.
-
addInput
void addInput(String name, Long value)
Adds an input with the given name and value to the current Dataflow step.
-
addInput
void addInput(String name, PInput value)
Adds an input with the given name to the previously added Dataflow step, coming from the specified input PValue.
-
addInput
void addInput(String name, Map<String,Object> elements)
Adds an input that is a dictionary of strings to objects.
-
addInput
void addInput(String name, List<? extends Map<String,Object>> elements)
Adds an input that is a list of objects.
-
addOutput
void addOutput(String name, PValue value)
Adds an output with the given name to the previously added Dataflow step, producing the specified outputPValue
, including itsCoder
if aTypedPValue
. If thePValue
is aPCollection
, wraps its coder inside aWindowedValueCoder
.
-
addValueOnlyOutput
void addValueOnlyOutput(String name, PValue value)
Adds an output with the given name to the previously added Dataflow step, producing the specified outputPValue
, including itsCoder
if aTypedPValue
. If thePValue
is aPCollection
, wraps its coder inside aValueOnlyCoder
.
-
addCollectionToSingletonOutput
void addCollectionToSingletonOutput(String name, PValue inputValue, PValue outputValue)
Adds an output with the given name to the previously added CollectionToSingleton Dataflow step, consuming the specified inputPValue
and producing the specified outputPValue
. This step requires special treatment for its output encoding.
-
asOutputReference
com.google.cloud.dataflow.sdk.util.OutputReference asOutputReference(PValue value)
Encode a PValue reference as an output reference.
-
-