Interface Summary Interface Description CompressedSource.DecompressingChannelFactoryFactory interface for creating channels that decompress the content of an underlying channel. UnboundedSource.CheckpointMarkA marker representing the progress and state of an
Class Summary Class Description AvroIO
PTransforms for reading and writing Avro files.
AvroIO.Read AvroIO.Read.Bound<T> AvroIO.Write AvroIO.Write.Bound<T> AvroSource<T>A
FileBasedSourcefor reading Avro files.
BlockBasedSource.BlockBasedReaderfor reading blocks from Avro files.
BigQueryIO BigQueryIO.Read BigQueryIO.Read.Bound BigQueryIO.Write BigQueryIO.Write.Bound BlockBasedSource<T>A
FileBasedSourcewhere a file consists of blocks of records.
Blockrepresents a block of records that can be read.
Readerthat reads records from a
Sourcethat reads a finite amount of input and, because of that, supports some additional operations.
Readerthat reads a bounded amount of input and supports some additional operations, such as progress estimation and dynamic work rebalancing.
CompressedSource<T>A Source that reads from compressed files. CompressedSource.CompressedReader<T>Reader for a
PTransformthat produces longs.
CountingInput.BoundedCountingInput CountingInput.UnboundedCountingInput CountingSourceA source that produces longs. CountingSource.CounterMarkThe checkpoint for an unbounded
CountingSourceis simply the last value produced.
DatastoreIO Deprecatedreplaced by
Source.Readerover the records from a query of the datastore.
Sourcethat reads the result rows of a Datastore query as
Sinkfor file-based output.
FileBasedSink.FileBasedWriteOperation<T> FileBasedSink.FileBasedWriter<T> FileBasedSink.FileResultResult of a single bundle write. FileBasedSource<T>A common base class for all file-based
readerthat implements code common to readers of
BoundedSourcethat uses offsets to define starting and ending positions.
OffsetBasedSource.OffsetBasedReader<T> PubsubIORead and Write
PTransforms for Cloud Pub/Sub streams.
PubsubIO.PubsubSubscriptionClass representing a Cloud Pub/Sub Subscription. PubsubIO.PubsubTopicClass representing a Cloud Pub/Sub Topic. PubsubIO.Read PubsubIO.Read.Bound<T> PubsubIO.Write PubsubIO.Write.Bound<T> PubsubUnboundedSink<T>A PTransform which streams messages to Pubsub. PubsubUnboundedSource<T>A PTransform which streams messages from Pubsub. Read Read.Bounded<T> Read.BuilderHelper class for building
Read.Unbounded<T> ShardNameTemplateStandard shard naming templates. Sink<T>A
Sinkrepresents a resource that can be written to using the
Sink.WriteOperationdefines the process of a parallel write of objects to a Sink.
Sink.Writer<T,WriteT>A Writer writes a bundle of elements from a PCollection to a sink. Source<T>Base class for defining input formats and creating a
Sourcefor reading the input.
Source.Reader<T>The interface that readers of custom input sources must implement. TextIO
PTransforms for reading and writing text files.
TextIO.Read TextIO.Read.Bound<T> TextIO.Write TextIO.Write.Bound<T>A PTransform that writes a bounded PCollection to a text file (or multiple text files matching a sharding pattern), with each PCollection element being encoded into its own line. UnboundedSource<OutputT,CheckpointMarkT extends UnboundedSource.CheckpointMark>A
Sourcethat reads an unbounded amount of input and, because of that, supports some additional operations such as checkpointing, watermarks, and record ids.
Readerthat reads an unbounded amount of input.
Write Write.Bound<T> XmlSinkA
Sinkthat outputs records as XML-formatted elements.
FileBasedSinkthat writes objects as XML elements.
Sink.Writerthat can write objects as XML elements.
XmlSource<T>A source that can be used to read XML files.
Enum Summary Enum Description BigQueryIO.Write.CreateDispositionAn enumeration type for the BigQuery create disposition strings. BigQueryIO.Write.WriteDispositionAn enumeration type for the BigQuery write disposition strings. CompressedSource.CompressionModeDefault compression types supported by the
FileBasedSink.FileBasedWriteOperation.TemporaryFileRetentionOptions for handling of temporary output files. FileBasedSource.ModeA given
FileBasedSourcerepresents a file resource of one of these types.
TextIO.CompressionTypePossible text file compression types.
Package com.google.cloud.dataflow.sdk.io Description
The classes in this package provide
Read transforms that create PCollections
from existing storage:
PCollection<TableRow> inputData = pipeline.apply( BigQueryIO.Read.named("Read") .from("clouddataflow-readonly:samples.weather_stations");
Writetransforms that persist PCollections to external storage:
PCollection<Integer> numbers = ...; numbers.apply(TextIO.Write.named("WriteNumbers") .to("gs://my_bucket/path/to/numbers"));