Sink.Writer (Google Cloud Dataflow SDK 1.9.1 API)

Google Cloud Dataflow SDK for Java, version 1.9.1

Warning: Dataflow SDK 1.x for Java is unsupported as of October 16, 2018. See Migrating from Dataflow SDK 1.x for Java for migration guidance.

Note: If you are using Cloud Dataflow SDK version 2.x, see the Apache Beam API reference documentation.

com.google.cloud.dataflow.sdk.io

Class Sink.Writer<T,WriteT>

• java.lang.Object
• • com.google.cloud.dataflow.sdk.io.Sink.Writer<T,WriteT>

• Type Parameters:

  T - The type of object to write

  WriteT - The writer results type (e.g., the bundle's output filename, as String)

  Direct Known Subclasses:

  FileBasedSink.FileBasedWriter

  Enclosing class:

  Sink<T>

  ---

  
  

  public abstract static class Sink.Writer<T,WriteT>
  extends Object

  A Writer writes a bundle of elements from a PCollection to a sink. open(java.lang.String) is called before writing begins and close() is called after all elements in the bundle have been written. write(T) writes an element to the sink.

  Note that any access to static members or methods of a Writer must be thread-safe, as multiple instances of a Writer may be instantiated in different threads on the same worker.

  See Sink for more detailed documentation about the process of writing to a Sink.

• • Constructor Summary

    Constructors 

    Constructor and Description
    Writer()

  • Method Summary

    All Methods Instance Methods Abstract Methods 

    Modifier and Type	Method and Description
    abstract WriteT	close() Finishes writing the bundle.
    abstract Sink.WriteOperation<T,WriteT>	getWriteOperation() Returns the write operation this writer belongs to.
    abstract void	open(String uId) Performs bundle initialization.
    abstract void	write(T value) Called for each value in the bundle.

    • Methods inherited from class java.lang.Object

      clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

• • Constructor Detail

    • Writer

      public Writer()

  • Method Detail

    • open

      public abstract void open(String uId)
                         throws Exception

      Performs bundle initialization. For example, creates a temporary file for writing or initializes any state that will be used across calls to write(T).

      The unique id that is given to open should be used to ensure that the writer's output does not interfere with the output of other Writers, as a bundle may be executed many times for fault tolerance. See Sink for more information about bundle ids.

      Throws:

      Exception

    • write

      public abstract void write(T value)
                          throws Exception

      Called for each value in the bundle.

      Throws:

      Exception

    • close

      public abstract WriteT close()
                            throws Exception

      Finishes writing the bundle. Closes any resources used for writing the bundle.

      Returns a writer result that will be used in the Sink.WriteOperation's finalization. The result should contain some way to identify the output of this bundle (using the bundle id). Sink.WriteOperation.finalize(java.lang.Iterable<WriteT>, com.google.cloud.dataflow.sdk.options.PipelineOptions) will use the writer result to identify successful writes. See Sink for more information about bundle ids.

      Returns:

      the writer result

      Throws:

      Exception

    • getWriteOperation

      public abstract Sink.WriteOperation<T,WriteT> getWriteOperation()

      Returns the write operation this writer belongs to.