com.google.cloud.dataflow.sdk.io (Google Cloud Dataflow SDK 1.9.1 API)

Google Cloud Dataflow SDK for Java, version 1.9.1

Warning: Cloud Dataflow SDK 1.x for Java is unsupported as of October 16, 2018. See Migrating from Cloud Dataflow SDK 1.x for Java for migration guidance.

Note: If you are using Cloud Dataflow SDK version 2.x, see the Apache Beam API reference documentation.

Package com.google.cloud.dataflow.sdk.io

Defines transforms for reading and writing common storage formats, including AvroIO, BigQueryIO, and TextIO.

See: Description

• Interface Summary 

  Interface	Description
  CompressedSource.DecompressingChannelFactory	Factory interface for creating channels that decompress the content of an underlying channel.
  UnboundedSource.CheckpointMark	A marker representing the progress and state of an UnboundedSource.UnboundedReader.

• Class Summary 

  Class	Description
  AvroIO	PTransforms for reading and writing Avro files.
  AvroIO.Read	A root PTransform that reads from an Avro file (or multiple Avro files matching a pattern) and returns a PCollection containing the decoding of each record.
  AvroIO.Read.Bound<T>	A PTransform that reads from an Avro file (or multiple Avro files matching a pattern) and returns a bounded PCollection containing the decoding of each record.
  AvroIO.Write	A root PTransform that writes a PCollection to an Avro file (or multiple Avro files matching a sharding pattern).
  AvroIO.Write.Bound<T>	A PTransform that writes a bounded PCollection to an Avro file (or multiple Avro files matching a sharding pattern).
  AvroSource<T>	A FileBasedSource for reading Avro files.
  AvroSource.AvroReader<T>	A BlockBasedSource.BlockBasedReader for reading blocks from Avro files.
  BigQueryIO	PTransforms for reading and writing BigQuery tables.
  BigQueryIO.Read	A PTransform that reads from a BigQuery table and returns a PCollection of TableRows containing each of the rows of the table.
  BigQueryIO.Read.Bound	A PTransform that reads from a BigQuery table and returns a bounded PCollection of TableRows.
  BigQueryIO.Write	A PTransform that writes a PCollection containing TableRows to a BigQuery table.
  BigQueryIO.Write.Bound	A PTransform that can write either a bounded or unbounded PCollection of TableRows to a BigQuery table.
  BlockBasedSource<T>	A BlockBasedSource is a FileBasedSource where a file consists of blocks of records.
  BlockBasedSource.Block<T>	A Block represents a block of records that can be read.
  BlockBasedSource.BlockBasedReader<T>	A Reader that reads records from a BlockBasedSource.
  BoundedSource<T>	A Source that reads a finite amount of input and, because of that, supports some additional operations.
  BoundedSource.BoundedReader<T>	A Reader that reads a bounded amount of input and supports some additional operations, such as progress estimation and dynamic work rebalancing.
  CompressedSource<T>	A Source that reads from compressed files.
  CompressedSource.CompressedReader<T>	Reader for a CompressedSource.
  CountingInput	A PTransform that produces longs.
  CountingInput.BoundedCountingInput	A PTransform that will produce a specified number of Longs starting from 0.
  CountingInput.UnboundedCountingInput	A PTransform that will produce numbers starting from 0 up to Long.MAX_VALUE.
  CountingSource	A source that produces longs.
  CountingSource.CounterMark	The checkpoint for an unbounded CountingSource is simply the last value produced.
  DatastoreIO	Deprecated replaced by DatastoreIO
  DatastoreIO.DatastoreReader	A Source.Reader over the records from a query of the datastore.
  DatastoreIO.Sink	A Sink that writes a PCollection containing Entities to a Datastore kind.
  DatastoreIO.Source	A Source that reads the result rows of a Datastore query as Entity objects.
  FileBasedSink<T>	Abstract Sink for file-based output.
  FileBasedSink.FileBasedWriteOperation<T>	Abstract Sink.WriteOperation that manages the process of writing to a FileBasedSink.
  FileBasedSink.FileBasedWriter<T>	Abstract Sink.Writer that writes a bundle to a FileBasedSink.
  FileBasedSink.FileResult	Result of a single bundle write.
  FileBasedSource<T>	A common base class for all file-based Sources.
  FileBasedSource.FileBasedReader<T>	A reader that implements code common to readers of FileBasedSources.
  OffsetBasedSource<T>	A BoundedSource that uses offsets to define starting and ending positions.
  OffsetBasedSource.OffsetBasedReader<T>	A Source.Reader that implements code common to readers of all OffsetBasedSources.
  PubsubIO	Read and Write PTransforms for Cloud Pub/Sub streams.
  PubsubIO.PubsubSubscription	Class representing a Cloud Pub/Sub Subscription.
  PubsubIO.PubsubTopic	Class representing a Cloud Pub/Sub Topic.
  PubsubIO.Read	A PTransform that continuously reads from a Cloud Pub/Sub stream and returns a PCollection of Strings containing the items from the stream.
  PubsubIO.Read.Bound<T>	A PTransform that reads from a Cloud Pub/Sub source and returns a unbounded PCollection containing the items from the stream.
  PubsubIO.Write	A PTransform that continuously writes a PCollection of Strings to a Cloud Pub/Sub stream.
  PubsubIO.Write.Bound<T>	A PTransform that writes an unbounded PCollection of Strings to a Cloud Pub/Sub stream.
  PubsubUnboundedSink<T>	A PTransform which streams messages to Pubsub.
  PubsubUnboundedSource<T>	A PTransform which streams messages from Pubsub.
  Read	A PTransform for reading from a Source.
  Read.Bounded<T>	PTransform that reads from a BoundedSource.
  Read.Builder	Helper class for building Read transforms.
  Read.Unbounded<T>	PTransform that reads from a UnboundedSource.
  ShardNameTemplate	Standard shard naming templates.
  Sink<T>	A Sink represents a resource that can be written to using the Write transform.
  Sink.WriteOperation<T,WriteT>	A Sink.WriteOperation defines the process of a parallel write of objects to a Sink.
  Sink.Writer<T,WriteT>	A Writer writes a bundle of elements from a PCollection to a sink.
  Source<T>	Base class for defining input formats and creating a Source for reading the input.
  Source.Reader<T>	The interface that readers of custom input sources must implement.
  TextIO	PTransforms for reading and writing text files.
  TextIO.Read	A PTransform that reads from a text file (or multiple text files matching a pattern) and returns a PCollection containing the decoding of each of the lines of the text file(s).
  TextIO.Read.Bound<T>	A PTransform that reads from one or more text files and returns a bounded PCollection containing one element for each line of the input files.
  TextIO.Write	A PTransform that writes a PCollection to text file (or multiple text files matching a sharding pattern), with each element of the input collection encoded into its own line.
  TextIO.Write.Bound<T>	A PTransform that writes a bounded PCollection to a text file (or multiple text files matching a sharding pattern), with each PCollection element being encoded into its own line.
  UnboundedSource<OutputT,CheckpointMarkT extends UnboundedSource.CheckpointMark>	A Source that reads an unbounded amount of input and, because of that, supports some additional operations such as checkpointing, watermarks, and record ids.
  UnboundedSource.UnboundedReader<OutputT>	A Reader that reads an unbounded amount of input.
  Write	A PTransform that writes to a Sink.
  Write.Bound<T>	A PTransform that writes to a Sink.
  XmlSink	A Sink that outputs records as XML-formatted elements.
  XmlSink.Bound<T>	A FileBasedSink that writes objects as XML elements.
  XmlSink.XmlWriteOperation<T>	Sink.WriteOperation for XML Sinks.
  XmlSink.XmlWriter<T>	A Sink.Writer that can write objects as XML elements.
  XmlSource<T>	A source that can be used to read XML files.

• Enum Summary 

  Enum	Description
  BigQueryIO.Write.CreateDisposition	An enumeration type for the BigQuery create disposition strings.
  BigQueryIO.Write.WriteDisposition	An enumeration type for the BigQuery write disposition strings.
  CompressedSource.CompressionMode	Default compression types supported by the CompressedSource.
  FileBasedSink.FileBasedWriteOperation.TemporaryFileRetention	Options for handling of temporary output files.
  FileBasedSource.Mode	A given FileBasedSource represents a file resource of one of these types.
  TextIO.CompressionType	Possible text file compression types.

Package com.google.cloud.dataflow.sdk.io Description

Defines transforms for reading and writing common storage formats, including AvroIO, BigQueryIO, and TextIO.

The classes in this package provide Read transforms that create PCollections from existing storage:

 PCollection<TableRow> inputData = pipeline.apply(
     BigQueryIO.Read.named("Read")
                    .from("clouddataflow-readonly:samples.weather_stations");
 

and Write transforms that persist PCollections to external storage:

 
 PCollection<Integer> numbers = ...;
 numbers.apply(TextIO.Write.named("WriteNumbers")
                           .to("gs://my_bucket/path/to/numbers"));