Google Cloud Dataflow SDK for Java, version 1.9.1
Class TextIO.Write
- java.lang.Object
-
- com.google.cloud.dataflow.sdk.io.TextIO.Write
-
- Enclosing class:
- TextIO
public static class TextIO.Write extends Object
APTransform
that writes aPCollection
to text file (or multiple text files matching a sharding pattern), with each element of the input collection encoded into its own line.
-
-
Nested Class Summary
Nested Classes Modifier and Type Class and Description static class
TextIO.Write.Bound<T>
A PTransform that writes a bounded PCollection to a text file (or multiple text files matching a sharding pattern), with each PCollection element being encoded into its own line.
-
Constructor Summary
Constructors Constructor and Description Write()
-
Method Summary
All Methods Static Methods Concrete Methods Modifier and Type Method and Description static TextIO.Write.Bound<String>
named(String name)
Returns a transform for writing to text files with the given step name.static TextIO.Write.Bound<String>
to(String prefix)
Returns a transform for writing to text files that writes to the file(s) with the given prefix.static TextIO.Write.Bound<String>
to(ValueProvider<String> prefix)
Liketo(String)
, but with aValueProvider
.static <T> TextIO.Write.Bound<T>
withCoder(Coder<T> coder)
Returns a transform for writing to text files that uses the givenCoder
to encode each of the elements of the inputPCollection
into an output text line.static TextIO.Write.Bound<String>
withFooter(String footer)
Returns a transform for writing to text files that adds a footer string to the files it writes.static TextIO.Write.Bound<String>
withHeader(String header)
Returns a transform for writing to text files that adds a header string to the files it writes.static TextIO.Write.Bound<String>
withNumShards(int numShards)
Returns a transform for writing to text files that uses the provided shard count.static TextIO.Write.Bound<String>
withoutSharding()
Returns a transform for writing to text files that forces a single file as output.static TextIO.Write.Bound<String>
withoutValidation()
Returns a transform for writing to text files that has Cloud Storage path validation on pipeline creation disabled.static TextIO.Write.Bound<String>
withShardNameTemplate(String shardTemplate)
Returns a transform for writing to text files that uses the given shard name template.static TextIO.Write.Bound<String>
withSuffix(String nameExtension)
Returns a transform for writing to text files that appends the specified suffix to the created files.
-
-
-
Method Detail
-
named
public static TextIO.Write.Bound<String> named(String name)
Returns a transform for writing to text files with the given step name.
-
to
public static TextIO.Write.Bound<String> to(String prefix)
Returns a transform for writing to text files that writes to the file(s) with the given prefix. This can be a local filename (if running locally), or a Google Cloud Storage filename of the form"gs://<bucket>/<filepath>"
(if running locally or via the Google Cloud Dataflow service).The files written will begin with this prefix, followed by a shard identifier (see
TextIO.Write.Bound.withNumShards(int)
, and end in a common extension, if given byTextIO.Write.Bound.withSuffix(String)
.
-
to
public static TextIO.Write.Bound<String> to(ValueProvider<String> prefix)
Liketo(String)
, but with aValueProvider
.
-
withSuffix
public static TextIO.Write.Bound<String> withSuffix(String nameExtension)
Returns a transform for writing to text files that appends the specified suffix to the created files.
-
withNumShards
public static TextIO.Write.Bound<String> withNumShards(int numShards)
Returns a transform for writing to text files that uses the provided shard count.Constraining the number of shards is likely to reduce the performance of a pipeline. Setting this value is not recommended unless you require a specific number of output files.
- Parameters:
numShards
- the number of shards to use, or 0 to let the system decide.
-
withShardNameTemplate
public static TextIO.Write.Bound<String> withShardNameTemplate(String shardTemplate)
Returns a transform for writing to text files that uses the given shard name template.See
ShardNameTemplate
for a description of shard templates.
-
withoutSharding
public static TextIO.Write.Bound<String> withoutSharding()
Returns a transform for writing to text files that forces a single file as output.
-
withCoder
public static <T> TextIO.Write.Bound<T> withCoder(Coder<T> coder)
Returns a transform for writing to text files that uses the givenCoder
to encode each of the elements of the inputPCollection
into an output text line.By default, uses
StringUtf8Coder
, which writes input Java strings directly as output lines.- Type Parameters:
T
- the type of the elements of the inputPCollection
-
withoutValidation
public static TextIO.Write.Bound<String> withoutValidation()
Returns a transform for writing to text files that has Cloud Storage path validation on pipeline creation disabled.This can be useful in the case where the Cloud Storage output location does not exist at the pipeline creation time, but is expected to be available at execution time.
-
withHeader
public static TextIO.Write.Bound<String> withHeader(@Nullable String header)
Returns a transform for writing to text files that adds a header string to the files it writes. Note that a newline character will be added after the header.A
null
value will clear any previously configured header.- Parameters:
header
- the string to be added as file header
-
withFooter
public static TextIO.Write.Bound<String> withFooter(@Nullable String footer)
Returns a transform for writing to text files that adds a footer string to the files it writes. Note that a newline character will be added after the header.A
null
value will clear any previously configured footer.- Parameters:
footer
- the string to be added as file footer
-
-