AvroIO (Google Cloud Dataflow SDK 1.9.1 API)

Google Cloud Dataflow SDK for Java, version 1.9.1

com.google.cloud.dataflow.sdk.io

Class AvroIO


  • public class AvroIO
extends Object
    PTransforms for reading and writing Avro files.

    To read a PCollection from one or more Avro files, use AvroIO.Read, specifying AvroIO.Read.from(java.lang.String) to specify the path of the file(s) to read from (e.g., a local filename or filename pattern if running locally, or a Google Cloud Storage filename or filename pattern of the form "gs://<bucket>/<filepath>"), and optionally AvroIO.Read.named(java.lang.String) to specify the name of the pipeline step.

    It is required to specify AvroIO.Read.withSchema(java.lang.Class<T>). To read specific records, such as Avro-generated classes, provide an Avro-generated class type. To read GenericRecords, provide either a Schema object or an Avro schema in a JSON-encoded string form. An exception will be thrown if a record doesn't match the specified schema.

    For example: 

     
 Pipeline p = ...;

 // A simple Read of a local file (only runs locally):
 PCollection<AvroAutoGenClass> records =
     p.apply(AvroIO.Read.from("/path/to/file.avro")
                        .withSchema(AvroAutoGenClass.class));

 // A Read from a GCS file (runs locally and via the Google Cloud
 // Dataflow service):
 Schema schema = new Schema.Parser().parse(new File("schema.avsc"));
 PCollection<GenericRecord> records =
     p.apply(AvroIO.Read.named("ReadFromAvro")
                        .from("gs://my_bucket/path/to/records-*.avro")
                        .withSchema(schema));

    To write a PCollection to one or more Avro files, use AvroIO.Write, specifying AvroIO.Write.to(java.lang.String) to specify the path of the file to write to (e.g., a local filename or sharded filename pattern if running locally, or a Google Cloud Storage filename or sharded filename pattern of the form "gs://<bucket>/<filepath>"), and optionally AvroIO.Write.named(java.lang.String) to specify the name of the pipeline step.

    It is required to specify AvroIO.Write.withSchema(java.lang.Class<T>). To write specific records, such as Avro-generated classes, provide an Avro-generated class type. To write GenericRecords, provide either a Schema object or a schema in a JSON-encoded string form. An exception will be thrown if a record doesn't match the specified schema.

    For example: 

     
 // A simple Write to a local file (only runs locally):
 PCollection<AvroAutoGenClass> records = ...;
 records.apply(AvroIO.Write.to("/path/to/file.avro")
                           .withSchema(AvroAutoGenClass.class));

 // A Write to a sharded GCS file (runs locally and via the Google Cloud
 // Dataflow service):
 Schema schema = new Schema.Parser().parse(new File("schema.avsc"));
 PCollection<GenericRecord> records = ...;
 records.apply(AvroIO.Write.named("WriteToAvro")
                           .to("gs://my_bucket/path/to/numbers")
                           .withSchema(schema)
                           .withSuffix(".avro"));

    Permissions

    Permission requirements depend on the PipelineRunner that is used to execute the Dataflow job. Please refer to the documentation of corresponding PipelineRunners for more details.


Oliko tästä sivusta apua? Kerro mielipiteesi

Palautteen aihe:

Tämä sivu
Dokumentaatio
Cloud Dataflow
Tuote
Tarvitsetko apua? Siirry tukisivullemme.