AvroIO.Write (Google Cloud Dataflow SDK 1.9.1 API)

Google Cloud Dataflow SDK for Java, version 1.9.1

Warning: Dataflow SDK 1.x for Java is unsupported as of October 16, 2018. After August 12, 2020, Dataflow will not run jobs using Dataflow 1.x and below. See Migrating from Dataflow SDK 1.x for Java for migration guidance.

Note: If you are using Cloud Dataflow SDK version 2.x, see the Apache Beam API reference documentation.

com.google.cloud.dataflow.sdk.io

Class AvroIO.Write

• java.lang.Object
• • com.google.cloud.dataflow.sdk.io.AvroIO.Write

• Enclosing class:

  AvroIO

  ---

  
  

  public static class AvroIO.Write
  extends Object

  A root PTransform that writes a PCollection to an Avro file (or multiple Avro files matching a sharding pattern).

• • Nested Class Summary

    Nested Classes 

    Modifier and Type	Class and Description
    static class	AvroIO.Write.Bound<T> A PTransform that writes a bounded PCollection to an Avro file (or multiple Avro files matching a sharding pattern).

  • Method Summary

    All Methods Static Methods Concrete Methods 

    Modifier and Type	Method and Description
    static AvroIO.Write.Bound<GenericRecord>	named(String name) Returns a PTransform with the given step name.
    static AvroIO.Write.Bound<GenericRecord>	to(String prefix) Returns a PTransform that writes to the file(s) with the given prefix.
    static AvroIO.Write.Bound<GenericRecord>	withNumShards(int numShards) Returns a PTransform that uses the provided shard count.
    static AvroIO.Write.Bound<GenericRecord>	withoutSharding() Returns a PTransform that forces a single file as output.
    static AvroIO.Write.Bound<GenericRecord>	withoutValidation() Returns a PTransform that writes Avro file(s) that has GCS path validation on pipeline creation disabled.
    static <T> AvroIO.Write.Bound<T>	withSchema(Class<T> type) Returns a PTransform that writes Avro file(s) containing records whose type is the specified Avro-generated class.
    static AvroIO.Write.Bound<GenericRecord>	withSchema(Schema schema) Returns a PTransform that writes Avro file(s) containing records of the specified schema.
    static AvroIO.Write.Bound<GenericRecord>	withSchema(String schema) Returns a PTransform that writes Avro file(s) containing records of the specified schema in a JSON-encoded string form.
    static AvroIO.Write.Bound<GenericRecord>	withShardNameTemplate(String shardTemplate) Returns a PTransform that uses the given shard name template.
    static AvroIO.Write.Bound<GenericRecord>	withSuffix(String filenameSuffix) Returns a PTransform that writes to the file(s) with the given filename suffix.

    • Methods inherited from class java.lang.Object

      clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

• • Method Detail

    • named

      public static AvroIO.Write.Bound<GenericRecord> named(String name)

      Returns a PTransform with the given step name.

    • to

      public static AvroIO.Write.Bound<GenericRecord> to(String prefix)

      Returns a PTransform that writes to the file(s) with the given prefix. This can be a local filename (if running locally), or a Google Cloud Storage filename of the form "gs://<bucket>/<filepath>" (if running locally or via the Google Cloud Dataflow service).

      The files written will begin with this prefix, followed by a shard identifier (see AvroIO.Write.Bound.withNumShards(int), and end in a common extension, if given by AvroIO.Write.Bound.withSuffix(java.lang.String).

    • withSuffix

      public static AvroIO.Write.Bound<GenericRecord> withSuffix(String filenameSuffix)

      Returns a PTransform that writes to the file(s) with the given filename suffix.

    • withNumShards

      public static AvroIO.Write.Bound<GenericRecord> withNumShards(int numShards)

      Returns a PTransform that uses the provided shard count.

      Constraining the number of shards is likely to reduce the performance of a pipeline. Setting this value is not recommended unless you require a specific number of output files.

      Parameters:

      numShards - the number of shards to use, or 0 to let the system decide.

    • withShardNameTemplate

      public static AvroIO.Write.Bound<GenericRecord> withShardNameTemplate(String shardTemplate)

      Returns a PTransform that uses the given shard name template.

      See ShardNameTemplate for a description of shard templates.

    • withoutSharding

      public static AvroIO.Write.Bound<GenericRecord> withoutSharding()

      Returns a PTransform that forces a single file as output.

      Constraining the number of shards is likely to reduce the performance of a pipeline. Setting this value is not recommended unless you require a specific number of output files.

    • withSchema

      public static <T> AvroIO.Write.Bound<T> withSchema(Class<T> type)

      Returns a PTransform that writes Avro file(s) containing records whose type is the specified Avro-generated class.

      Type Parameters:

      T - the type of the elements of the input PCollection

    • withSchema

      public static AvroIO.Write.Bound<GenericRecord> withSchema(Schema schema)

      Returns a PTransform that writes Avro file(s) containing records of the specified schema.

    • withSchema

      public static AvroIO.Write.Bound<GenericRecord> withSchema(String schema)

      Returns a PTransform that writes Avro file(s) containing records of the specified schema in a JSON-encoded string form.

    • withoutValidation

      public static AvroIO.Write.Bound<GenericRecord> withoutValidation()

      Returns a PTransform that writes Avro file(s) that has GCS path validation on pipeline creation disabled.

      This can be useful in the case where the GCS output location does not exist at the pipeline creation time, but is expected to be available at execution time.