FileBasedSink.FileBasedWriter (Google Cloud Dataflow SDK 1.9.1 API)

Google Cloud Dataflow SDK for Java, version 1.9.1

Warning: Dataflow SDK 1.x for Java is unsupported as of October 16, 2018. After August 12, 2020, you can use unsupported SDKs temporarily while you upgrade your pipelines. For more information, see Using an unsupported SDK.

For information on migrating from the 1.x SDK, see Migrating from Dataflow SDK 1.x for Java.

Note: If you are using Cloud Dataflow SDK version 2.x, see the Apache Beam API reference documentation.

com.google.cloud.dataflow.sdk.io

Class FileBasedSink.FileBasedWriter<T>

• java.lang.Object
• • com.google.cloud.dataflow.sdk.io.Sink.Writer<T,FileBasedSink.FileResult>
  • • com.google.cloud.dataflow.sdk.io.FileBasedSink.FileBasedWriter<T>

• Type Parameters:

  T - the type of values to write.

  Direct Known Subclasses:

  XmlSink.XmlWriter

  Enclosing class:

  FileBasedSink<T>

  ---

  
  

  public abstract static class FileBasedSink.FileBasedWriter<T>
  extends Sink.Writer<T,FileBasedSink.FileResult>

  Abstract Sink.Writer that writes a bundle to a FileBasedSink. Subclass implementations provide a method that can write a single value to a WritableByteChannel (Sink.Writer.write(T)).

  Subclass implementations may also override methods that write headers and footers before and after the values in a bundle, respectively, as well as provide a MIME type for the output channel.

  Multiple FileBasedWriter instances may be created on the same worker, and therefore any access to static members or methods should be thread safe.

• • Field Summary

    Fields 

    Modifier and Type	Field and Description
    protected String	mimeType The MIME type used in the creation of the output channel (if the file system supports it).

  • Constructor Summary

    Constructors 

    Constructor and Description
    FileBasedWriter(FileBasedSink.FileBasedWriteOperation<T> writeOperation) Construct a new FileBasedWriter with a base filename.

  • Method Summary

    All Methods Instance Methods Abstract Methods Concrete Methods 

    Modifier and Type	Method and Description
    FileBasedSink.FileResult	close() Closes the channel and return the bundle result.
    FileBasedSink.FileBasedWriteOperation<T>	getWriteOperation() Return the FileBasedWriteOperation that this Writer belongs to.
    void	open(String uId) Opens the channel.
    protected abstract void	prepareWrite(WritableByteChannel channel) Called with the channel that a subclass will write its header, footer, and values to.
    protected void	writeFooter() Writes footer at the end of output files.
    protected void	writeHeader() Writes header at the beginning of output files.

    • Methods inherited from class com.google.cloud.dataflow.sdk.io.Sink.Writer

      write

    • Methods inherited from class java.lang.Object

      clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

• • Field Detail

    • mimeType

      protected String mimeType

      The MIME type used in the creation of the output channel (if the file system supports it).

      GCS, for example, supports writing files with Content-Type metadata.

      May be overridden. Default is MimeTypes.TEXT. See MimeTypes for other options.

  • Constructor Detail

    • FileBasedWriter

      public FileBasedWriter(FileBasedSink.FileBasedWriteOperation<T> writeOperation)

      Construct a new FileBasedWriter with a base filename.

  • Method Detail

    • prepareWrite

      protected abstract void prepareWrite(WritableByteChannel channel)
                                    throws Exception

      Called with the channel that a subclass will write its header, footer, and values to. Subclasses should either keep a reference to the channel provided or create and keep a reference to an appropriate object that they will use to write to it.

      Called before any subsequent calls to writeHeader, writeFooter, and write.

      Throws:

      Exception

    • writeHeader

      protected void writeHeader()
                          throws Exception

      Writes header at the beginning of output files. Nothing by default; subclasses may override.

      Throws:

      Exception

    • writeFooter

      protected void writeFooter()
                          throws Exception

      Writes footer at the end of output files. Nothing by default; subclasses may override.

      Throws:

      Exception

    • open

      public final void open(String uId)
                      throws Exception

      Opens the channel.

      Specified by:

      open in class Sink.Writer<T,FileBasedSink.FileResult>

      Throws:

      Exception

    • close

      public final FileBasedSink.FileResult close()
                                           throws Exception

      Closes the channel and return the bundle result.

      Specified by:

      close in class Sink.Writer<T,FileBasedSink.FileResult>

      Returns:

      the writer result

      Throws:

      Exception

    • getWriteOperation

      public FileBasedSink.FileBasedWriteOperation<T> getWriteOperation()

      Return the FileBasedWriteOperation that this Writer belongs to.

      Specified by:

      getWriteOperation in class Sink.Writer<T,FileBasedSink.FileResult>