FileBasedSource (Google Cloud Dataflow SDK 1.9.1 API)

Google Cloud Dataflow SDK for Java, version 1.9.1

Warning: Dataflow SDK 1.x for Java is unsupported as of October 16, 2018. See Migrating from Dataflow SDK 1.x for Java for migration guidance.

Note: If you are using Cloud Dataflow SDK version 2.x, see the Apache Beam API reference documentation.

com.google.cloud.dataflow.sdk.io

Class FileBasedSource<T>

• java.lang.Object
• • com.google.cloud.dataflow.sdk.io.Source<T>
  • • com.google.cloud.dataflow.sdk.io.BoundedSource<T>
    • • com.google.cloud.dataflow.sdk.io.OffsetBasedSource<T>
      • • com.google.cloud.dataflow.sdk.io.FileBasedSource<T>

• Type Parameters:

  T - Type of records represented by the source.

  All Implemented Interfaces:

  HasDisplayData, Serializable

  Direct Known Subclasses:

  BlockBasedSource, CompressedSource, XmlSource

  ---

  
  

  public abstract class FileBasedSource<T>
  extends OffsetBasedSource<T>

  A common base class for all file-based Sources. Extend this class to implement your own file-based custom source.

  A file-based Source is a Source backed by a file pattern defined as a Java glob, a single file, or a offset range for a single file. See OffsetBasedSource and RangeTracker for semantics of offset ranges.

  This source stores a String that is an IOChannelFactory specification for a file or file pattern. There should be an IOChannelFactory defined for the file specification provided. Please refer to IOChannelUtils and IOChannelFactory for more information on this.

  In addition to the methods left abstract from BoundedSource, subclasses must implement methods to create a sub-source and a reader for a range of a single file - createForSubrangeOfFile(java.lang.String, long, long) and createSingleFileReader(com.google.cloud.dataflow.sdk.options.PipelineOptions). Please refer to XmlSource for an example implementation of FileBasedSource.

  See Also:

  Serialized Form

• • Nested Class Summary

    Nested Classes 

    Modifier and Type	Class and Description
    static class	FileBasedSource.FileBasedReader<T> A reader that implements code common to readers of FileBasedSources.
    static class	FileBasedSource.Mode A given FileBasedSource represents a file resource of one of these types.

    • Nested classes/interfaces inherited from class com.google.cloud.dataflow.sdk.io.OffsetBasedSource

      OffsetBasedSource.OffsetBasedReader<T>

    • Nested classes/interfaces inherited from class com.google.cloud.dataflow.sdk.io.BoundedSource

      BoundedSource.BoundedReader<T>

    • Nested classes/interfaces inherited from class com.google.cloud.dataflow.sdk.io.Source

      Source.Reader<T>

  • Constructor Summary

    Constructors 

    Constructor and Description
    FileBasedSource(String fileOrPatternSpec, long minBundleSize) Create a FileBaseSource based on a file or a file pattern specification.
    FileBasedSource(String fileName, long minBundleSize, long startOffset, long endOffset) Create a FileBasedSource based on a single file.
    FileBasedSource(ValueProvider<String> fileOrPatternSpec, long minBundleSize) Create a FileBaseSource based on a file or a file pattern specification.

  • Method Summary

    All Methods Static Methods Instance Methods Abstract Methods Concrete Methods 

    Modifier and Type	Method and Description
    protected abstract FileBasedSource<T>	createForSubrangeOfFile(String fileName, long start, long end) Creates and returns a new FileBasedSource of the same type as the current FileBasedSource backed by a given file and an offset range.
    BoundedSource.BoundedReader<T>	createReader(PipelineOptions options) Returns a new BoundedSource.BoundedReader that reads from this source.
    protected abstract FileBasedSource.FileBasedReader<T>	createSingleFileReader(PipelineOptions options) Creates and returns an instance of a FileBasedReader implementation for the current source assuming the source represents a single file.
    FileBasedSource<T>	createSourceForSubrange(long start, long end) Returns an OffsetBasedSource for a subrange of the current source.
    protected static Collection<String>	expandFilePattern(String fileOrPatternSpec)
    long	getEstimatedSizeBytes(PipelineOptions options) An estimate of the total size (in bytes) of the data that would be read from this source.
    String	getFileOrPatternSpec()
    ValueProvider<String>	getFileOrPatternSpecProvider()
    long	getMaxEndOffset(PipelineOptions options) Returns the actual ending offset of the current source.
    FileBasedSource.Mode	getMode()
    protected boolean	isSplittable() Determines whether a file represented by this source is can be split into bundles.
    void	populateDisplayData(DisplayData.Builder builder) Register display data for the given transform or component.
    List<? extends FileBasedSource<T>>	splitIntoBundles(long desiredBundleSizeBytes, PipelineOptions options) Splits the source into bundles of approximately desiredBundleSizeBytes.
    String	toString()
    void	validate() Checks that this source is valid, before it can be used in a pipeline.

    • Methods inherited from class com.google.cloud.dataflow.sdk.io.OffsetBasedSource

      allowsDynamicSplitting, getBytesPerOffset, getEndOffset, getMinBundleSize, getStartOffset

    • Methods inherited from class com.google.cloud.dataflow.sdk.io.BoundedSource

      producesSortedKeys

    • Methods inherited from class com.google.cloud.dataflow.sdk.io.Source

      getDefaultOutputCoder

    • Methods inherited from class java.lang.Object

      clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

• • Constructor Detail

    • FileBasedSource

      public FileBasedSource(String fileOrPatternSpec,
                             long minBundleSize)

      Create a FileBaseSource based on a file or a file pattern specification. This constructor must be used when creating a new FileBasedSource for a file pattern.

      See OffsetBasedSource for a detailed description of minBundleSize.

      Parameters:

      fileOrPatternSpec - IOChannelFactory specification of file or file pattern represented by the FileBasedSource.

      minBundleSize - minimum bundle size in bytes.

    • FileBasedSource

      public FileBasedSource(ValueProvider<String> fileOrPatternSpec,
                             long minBundleSize)

      Create a FileBaseSource based on a file or a file pattern specification. Same as the String constructor, but accepting a ValueProvider to allow for runtime configuration of the source.

    • FileBasedSource

      public FileBasedSource(String fileName,
                             long minBundleSize,
                             long startOffset,
                             long endOffset)

      Create a FileBasedSource based on a single file. This constructor must be used when creating a new FileBasedSource for a subrange of a single file. Additionally, this constructor must be used to create new FileBasedSources when subclasses implement the method createForSubrangeOfFile(java.lang.String, long, long).

      See OffsetBasedSource for detailed descriptions of minBundleSize, startOffset, and endOffset.

      Parameters:

      fileName - IOChannelFactory specification of the file represented by the FileBasedSource.

      minBundleSize - minimum bundle size in bytes.

      startOffset - starting byte offset.

      endOffset - ending byte offset. If the specified value >= #getMaxEndOffset() it implies #getMaxEndOffSet().

  • Method Detail

    • getFileOrPatternSpec

      public final String getFileOrPatternSpec()

    • getFileOrPatternSpecProvider

      public final ValueProvider<String> getFileOrPatternSpecProvider()

    • getMode

      public final FileBasedSource.Mode getMode()

    • createSourceForSubrange

      public final FileBasedSource<T> createSourceForSubrange(long start,
                                                              long end)

      Description copied from class: OffsetBasedSource
      Returns an OffsetBasedSource for a subrange of the current source. The subrange [start, end) must be within the range [startOffset, endOffset) of the current source, i.e. startOffset <= start < end <= endOffset.

      Specified by:

      createSourceForSubrange in class OffsetBasedSource<T>

    • createForSubrangeOfFile

      protected abstract FileBasedSource<T> createForSubrangeOfFile(String fileName,
                                                                    long start,
                                                                    long end)

      Creates and returns a new FileBasedSource of the same type as the current FileBasedSource backed by a given file and an offset range. When current source is being split, this method is used to generate new sub-sources. When creating the source subclasses must call the constructor FileBasedSource(String, long, long, long) of FileBasedSource with corresponding parameter values passed here.

      Parameters:

      fileName - file backing the new FileBasedSource.

      start - starting byte offset of the new FileBasedSource.

      end - ending byte offset of the new FileBasedSource. May be Long.MAX_VALUE, in which case it will be inferred using getMaxEndOffset(com.google.cloud.dataflow.sdk.options.PipelineOptions).

    • createSingleFileReader

      protected abstract FileBasedSource.FileBasedReader<T> createSingleFileReader(PipelineOptions options)

      Creates and returns an instance of a FileBasedReader implementation for the current source assuming the source represents a single file. File patterns will be handled by FileBasedSource implementation automatically.

    • getEstimatedSizeBytes

      public final long getEstimatedSizeBytes(PipelineOptions options)
                                       throws Exception

      Description copied from class: BoundedSource
      An estimate of the total size (in bytes) of the data that would be read from this source. This estimate is in terms of external storage size, before any decompression or other processing done by the reader.

      Overrides:

      getEstimatedSizeBytes in class OffsetBasedSource<T>

      Throws:

      Exception

    • populateDisplayData

      public void populateDisplayData(DisplayData.Builder builder)

      Description copied from class: Source
      Register display data for the given transform or component.

      populateDisplayData(DisplayData.Builder) is invoked by Pipeline runners to collect display data via DisplayData.from(HasDisplayData). Implementations may call super.populateDisplayData(builder) in order to register display data in the current namespace, but should otherwise use subcomponent.populateDisplayData(builder) to use the namespace of the subcomponent.

      By default, does not register any display data. Implementors may override this method to provide their own display data.

      Specified by:

      populateDisplayData in interface HasDisplayData

      Overrides:

      populateDisplayData in class OffsetBasedSource<T>

      Parameters:

      builder - The builder to populate with display data.

      See Also:

      HasDisplayData

    • splitIntoBundles

      public final List<? extends FileBasedSource<T>> splitIntoBundles(long desiredBundleSizeBytes,
                                                                       PipelineOptions options)
                                                                throws Exception

      Description copied from class: BoundedSource
      Splits the source into bundles of approximately desiredBundleSizeBytes.

      Overrides:

      splitIntoBundles in class OffsetBasedSource<T>

      Throws:

      Exception

    • isSplittable

      protected boolean isSplittable()
                              throws Exception

      Determines whether a file represented by this source is can be split into bundles.

      By default, a file is splittable if it is on a file system that supports efficient read seeking. Subclasses may override to provide different behavior.

      Throws:

      Exception

    • createReader

      public final BoundedSource.BoundedReader<T> createReader(PipelineOptions options)
                                                        throws IOException

      Description copied from class: BoundedSource
      Returns a new BoundedSource.BoundedReader that reads from this source.

      Specified by:

      createReader in class BoundedSource<T>

      Throws:

      IOException

    • toString

      public String toString()

      Overrides:

      toString in class OffsetBasedSource<T>

    • validate

      public void validate()

      Description copied from class: Source
      Checks that this source is valid, before it can be used in a pipeline.

      It is recommended to use Preconditions for implementing this method.

      Overrides:

      validate in class OffsetBasedSource<T>

    • getMaxEndOffset

      public final long getMaxEndOffset(PipelineOptions options)
                                 throws Exception

      Description copied from class: OffsetBasedSource
      Returns the actual ending offset of the current source. The value returned by this function will be used to clip the end of the range [startOffset, endOffset) such that the range used is [startOffset, min(endOffset, maxEndOffset)).

      As an example in which OffsetBasedSource is used to implement a file source, suppose that this source was constructed with an endOffset of Long.MAX_VALUE to indicate that a file should be read to the end. Then this function should determine the actual, exact size of the file in bytes and return it.

      Specified by:

      getMaxEndOffset in class OffsetBasedSource<T>

      Throws:

      Exception

    • expandFilePattern

      protected static final Collection<String> expandFilePattern(String fileOrPatternSpec)
                                                           throws IOException

      Throws:

      IOException