AvroIO.Read (Google Cloud Dataflow SDK 1.9.1 API)

Google Cloud Dataflow SDK for Java, version 1.9.1

Warning: Dataflow SDK 1.x for Java is unsupported as of October 16, 2018. See Migrating from Dataflow SDK 1.x for Java for migration guidance.

Note: If you are using Cloud Dataflow SDK version 2.x, see the Apache Beam API reference documentation.

com.google.cloud.dataflow.sdk.io

Class AvroIO.Read

• java.lang.Object
• • com.google.cloud.dataflow.sdk.io.AvroIO.Read

• Enclosing class:

  AvroIO

  ---

  
  

  public static class AvroIO.Read
  extends Object

  A root PTransform that reads from an Avro file (or multiple Avro files matching a pattern) and returns a PCollection containing the decoding of each record.

• • Nested Class Summary

    Nested Classes 

    Modifier and Type	Class and Description
    static class	AvroIO.Read.Bound<T> A PTransform that reads from an Avro file (or multiple Avro files matching a pattern) and returns a bounded PCollection containing the decoding of each record.

  • Method Summary

    All Methods Static Methods Concrete Methods 

    Modifier and Type	Method and Description
    static AvroIO.Read.Bound<GenericRecord>	from(String filepattern) Returns a PTransform that reads from the file(s) with the given name or pattern.
    static AvroIO.Read.Bound<GenericRecord>	named(String name) Returns a PTransform with the given step name.
    static AvroIO.Read.Bound<GenericRecord>	withoutValidation() Returns a PTransform that reads Avro file(s) that has GCS path validation on pipeline creation disabled.
    static <T> AvroIO.Read.Bound<T>	withSchema(Class<T> type) Returns a PTransform that reads Avro file(s) containing records whose type is the specified Avro-generated class.
    static AvroIO.Read.Bound<GenericRecord>	withSchema(Schema schema) Returns a PTransform that reads Avro file(s) containing records of the specified schema.
    static AvroIO.Read.Bound<GenericRecord>	withSchema(String schema) Returns a PTransform that reads Avro file(s) containing records of the specified schema in a JSON-encoded string form.

    • Methods inherited from class java.lang.Object

      clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

• • Method Detail

    • named

      public static AvroIO.Read.Bound<GenericRecord> named(String name)

      Returns a PTransform with the given step name.

    • from

      public static AvroIO.Read.Bound<GenericRecord> from(String filepattern)

      Returns a PTransform that reads from the file(s) with the given name or pattern. This can be a local filename or filename pattern (if running locally), or a Google Cloud Storage filename or filename pattern of the form "gs://<bucket>/<filepath>" (if running locally or via the Google Cloud Dataflow service). Standard Java Filesystem glob patterns ("*", "?", "[..]") are supported.

    • withSchema

      public static <T> AvroIO.Read.Bound<T> withSchema(Class<T> type)

      Returns a PTransform that reads Avro file(s) containing records whose type is the specified Avro-generated class.

      Type Parameters:

      T - the type of the decoded elements, and the elements of the resulting PCollection

    • withSchema

      public static AvroIO.Read.Bound<GenericRecord> withSchema(Schema schema)

      Returns a PTransform that reads Avro file(s) containing records of the specified schema.

    • withSchema

      public static AvroIO.Read.Bound<GenericRecord> withSchema(String schema)

      Returns a PTransform that reads Avro file(s) containing records of the specified schema in a JSON-encoded string form.

    • withoutValidation

      public static AvroIO.Read.Bound<GenericRecord> withoutValidation()

      Returns a PTransform that reads Avro file(s) that has GCS path validation on pipeline creation disabled.

      This can be useful in the case where the GCS input location does not exist at the pipeline creation time, but is expected to be available at execution time.