Google Cloud Dataflow SDK for Java, version 1.9.1
Class PCollectionList<T>
- java.lang.Object
-
- com.google.cloud.dataflow.sdk.values.PCollectionList<T>
-
- Type Parameters:
T
- the type of the elements of all thePCollections
in this list
public class PCollectionList<T> extends Object implements PInput, POutput
APCollectionList<T>
is an immutable list of homogeneously typedPCollection<T>s
. APCollectionList
is used, for instance, as the input toFlatten
or the output ofPartition
.PCollectionLists can be created and accessed like follows:
PCollection<String> pc1 = ...; PCollection<String> pc2 = ...; PCollection<String> pc3 = ...; // Create a PCollectionList with three PCollections: PCollectionList<String> pcs = PCollectionList.of(pc1).and(pc2).and(pc3); // Create an empty PCollectionList: Pipeline p = ...; PCollectionList<String> pcs2 = PCollectionList.<String>empty(p); // Get PCollections out of a PCollectionList, by index (origin 0): PCollection<String> pcX = pcs.get(1); PCollection<String> pcY = pcs.get(0); PCollection<String> pcZ = pcs.get(2); // Get a list of all PCollections in a PCollectionList: List<PCollection<String>> allPcs = pcs.getAll();
-
-
Method Summary
All Methods Static Methods Instance Methods Concrete Methods Modifier and Type Method and Description PCollectionList<T>
and(Iterable<PCollection<T>> pcs)
Returns a newPCollectionList
that has all thePCollections
of thisPCollectionList
plus the givenPCollections
appended to the end, in order.PCollectionList<T>
and(PCollection<T> pc)
Returns a newPCollectionList
that has all thePCollections
of thisPCollectionList
plus the givenPCollection
appended to the end.<OutputT extends POutput>
OutputTapply(PTransform<PCollectionList<T>,OutputT> t)
Likeapply(String, PTransform)
but defaulting to the name of thePTransform
.<OutputT extends POutput>
OutputTapply(String name, PTransform<PCollectionList<T>,OutputT> t)
Applies the givenPTransform
to this inputPCollectionList
, usingname
to identify this specific application of the transform.static <T> PCollectionList<T>
empty(Pipeline pipeline)
Returns an emptyPCollectionList
that is part of the givenPipeline
.Collection<? extends PValue>
expand()
void
finishSpecifying()
After building, finalizes thisPInput
to make it ready for being used as an input to aPTransform
.void
finishSpecifyingOutput()
As part of applying the producingPTransform
, finalizes this output to make it ready for being used as an input and for running.PCollection<T>
get(int index)
Returns thePCollection
at the given index (origin zero).List<PCollection<T>>
getAll()
Returns an immutable List of all thePCollections
in thisPCollectionList
.Pipeline
getPipeline()
static <T> PCollectionList<T>
of(Iterable<PCollection<T>> pcs)
Returns aPCollectionList
containing the givenPCollections
, in order.static <T> PCollectionList<T>
of(PCollection<T> pc)
Returns a singletonPCollectionList
containing the givenPCollection
.void
recordAsOutput(AppliedPTransform<?,?,?> transform)
Records that thisPOutput
is an output of the givenPTransform
.int
size()
Returns the number ofPCollections
in thisPCollectionList
.
-
-
-
Method Detail
-
empty
public static <T> PCollectionList<T> empty(Pipeline pipeline)
Returns an emptyPCollectionList
that is part of the givenPipeline
.Longer
PCollectionLists
can be created by callingand(com.google.cloud.dataflow.sdk.values.PCollection<T>)
on the result.
-
of
public static <T> PCollectionList<T> of(PCollection<T> pc)
Returns a singletonPCollectionList
containing the givenPCollection
.Longer
PCollectionLists
can be created by callingand(com.google.cloud.dataflow.sdk.values.PCollection<T>)
on the result.
-
of
public static <T> PCollectionList<T> of(Iterable<PCollection<T>> pcs)
Returns aPCollectionList
containing the givenPCollections
, in order.The argument list cannot be empty.
All the
PCollections
in the resultingPCollectionList
must be part of the samePipeline
.Longer PCollectionLists can be created by calling
and(com.google.cloud.dataflow.sdk.values.PCollection<T>)
on the result.
-
and
public PCollectionList<T> and(PCollection<T> pc)
Returns a newPCollectionList
that has all thePCollections
of thisPCollectionList
plus the givenPCollection
appended to the end.All the
PCollections
in the resultingPCollectionList
must be part of the samePipeline
.
-
and
public PCollectionList<T> and(Iterable<PCollection<T>> pcs)
Returns a newPCollectionList
that has all thePCollections
of thisPCollectionList
plus the givenPCollections
appended to the end, in order.All the
PCollections
in the resultingPCollectionList
must be part of the samePipeline
.
-
size
public int size()
Returns the number ofPCollections
in thisPCollectionList
.
-
get
public PCollection<T> get(int index)
Returns thePCollection
at the given index (origin zero).- Throws:
IndexOutOfBoundsException
- if the index is out of the range[0..size()-1]
.
-
getAll
public List<PCollection<T>> getAll()
Returns an immutable List of all thePCollections
in thisPCollectionList
.
-
apply
public <OutputT extends POutput> OutputT apply(PTransform<PCollectionList<T>,OutputT> t)
Likeapply(String, PTransform)
but defaulting to the name of thePTransform
.
-
apply
public <OutputT extends POutput> OutputT apply(String name, PTransform<PCollectionList<T>,OutputT> t)
Applies the givenPTransform
to this inputPCollectionList
, usingname
to identify this specific application of the transform. This name is used in various places, including the monitoring UI, logging, and to stably identify this application node in the job graph.- Returns:
- the output of the applied
PTransform
-
getPipeline
public Pipeline getPipeline()
Description copied from interface:PInput
- Specified by:
getPipeline
in interfacePInput
- Specified by:
getPipeline
in interfacePOutput
-
expand
public Collection<? extends PValue> expand()
Description copied from interface:PInput
Expands thisPInput
into a list of its component outputPValues
.- A
PValue
expands to itself. - A tuple or list of
PValues
(such asPCollectionTuple
orPCollectionList
) expands to its componentPValue PValues
.
Not intended to be invoked directly by user code.
- A
-
recordAsOutput
public void recordAsOutput(AppliedPTransform<?,?,?> transform)
Description copied from interface:POutput
Records that thisPOutput
is an output of the givenPTransform
.For a compound
POutput
, it is advised to call this method on each componentPOutput
.This is not intended to be invoked by user code, but is automatically invoked as part of applying the producing
PTransform
.- Specified by:
recordAsOutput
in interfacePOutput
-
finishSpecifying
public void finishSpecifying()
Description copied from interface:PInput
After building, finalizes thisPInput
to make it ready for being used as an input to aPTransform
.Automatically invoked whenever
apply()
is invoked on thisPInput
, so users do not normally call this explicitly.- Specified by:
finishSpecifying
in interfacePInput
-
finishSpecifyingOutput
public void finishSpecifyingOutput()
Description copied from interface:POutput
As part of applying the producingPTransform
, finalizes this output to make it ready for being used as an input and for running.This includes ensuring that all
PCollections
haveCoders
specified or defaulted.Automatically invoked whenever this
POutput
is used as aPInput
to anotherPTransform
, or if never used as aPInput
, whenPipeline.run()
is called, so users do not normally call this explicitly.- Specified by:
finishSpecifyingOutput
in interfacePOutput
-
-