CompressedSource (Google Cloud Dataflow SDK 1.9.1 API)

Google Cloud Dataflow SDK for Java, version 1.9.1

Warning: Dataflow SDK 1.x for Java is unsupported as of October 16, 2018. After August 12, 2020, you can use unsupported SDKs temporarily while you upgrade your pipelines. For more information, see Using an unsupported SDK.

For information on migrating from the 1.x SDK, see Migrating from Dataflow SDK 1.x for Java.

Note: If you are using Cloud Dataflow SDK version 2.x, see the Apache Beam API reference documentation.

com.google.cloud.dataflow.sdk.io

Class CompressedSource<T>

• java.lang.Object
• • com.google.cloud.dataflow.sdk.io.Source<T>
  • • com.google.cloud.dataflow.sdk.io.BoundedSource<T>
    • • com.google.cloud.dataflow.sdk.io.OffsetBasedSource<T>
      • • com.google.cloud.dataflow.sdk.io.FileBasedSource<T>
        • • com.google.cloud.dataflow.sdk.io.CompressedSource<T>

• Type Parameters:

  T - The type to read from the compressed file.

  All Implemented Interfaces:

  HasDisplayData, Serializable

  ---

  
  

  @Experimental(value=SOURCE_SINK)
  public class CompressedSource<T>
  extends FileBasedSource<T>

  A Source that reads from compressed files. A CompressedSources wraps a delegate FileBasedSource that is able to read the decompressed file format.

  For example, use the following to read from a gzip-compressed XML file:

   
   XmlSource mySource = XmlSource.from(...);
   PCollection<T> collection = p.apply(Read.from(CompressedSource
       .from(mySource)
       .withDecompression(CompressedSource.CompressionMode.GZIP)));
    

  Supported compression algorithms are CompressedSource.CompressionMode.GZIP and CompressedSource.CompressionMode.BZIP2. User-defined compression types are supported by implementing CompressedSource.DecompressingChannelFactory.

  By default, the compression algorithm is selected from those supported in CompressedSource.CompressionMode based on the file name provided to the source, namely ".bz2" indicates CompressedSource.CompressionMode.BZIP2 and ".gz" indicates CompressedSource.CompressionMode.GZIP. If the file name does not match any of the supported algorithms, it is assumed to be uncompressed data.

  See Also:

  Serialized Form

• • Nested Class Summary

    Nested Classes 

    Modifier and Type	Class and Description
    static class	CompressedSource.CompressedReader<T> Reader for a CompressedSource.
    static class	CompressedSource.CompressionMode Default compression types supported by the CompressedSource.
    static interface	CompressedSource.DecompressingChannelFactory Factory interface for creating channels that decompress the content of an underlying channel.

    • Nested classes/interfaces inherited from class com.google.cloud.dataflow.sdk.io.FileBasedSource

      FileBasedSource.FileBasedReader<T>, FileBasedSource.Mode

    • Nested classes/interfaces inherited from class com.google.cloud.dataflow.sdk.io.OffsetBasedSource

      OffsetBasedSource.OffsetBasedReader<T>

    • Nested classes/interfaces inherited from class com.google.cloud.dataflow.sdk.io.BoundedSource

      BoundedSource.BoundedReader<T>

    • Nested classes/interfaces inherited from class com.google.cloud.dataflow.sdk.io.Source

      Source.Reader<T>

  • Method Summary

    All Methods Static Methods Instance Methods Concrete Methods 

    Modifier and Type	Method and Description
    protected FileBasedSource<T>	createForSubrangeOfFile(String fileName, long start, long end) Creates a CompressedSource for a subrange of a file.
    protected FileBasedSource.FileBasedReader<T>	createSingleFileReader(PipelineOptions options) Creates a FileBasedReader to read a single file.
    static <T> CompressedSource<T>	from(FileBasedSource<T> sourceDelegate) Creates a CompressedSource from an underlying FileBasedSource.
    CompressedSource.DecompressingChannelFactory	getChannelFactory()
    Coder<T>	getDefaultOutputCoder() Returns the delegate source's default output coder.
    protected boolean	isSplittable() Determines whether a single file represented by this source is splittable.
    void	populateDisplayData(DisplayData.Builder builder) Register display data for the given transform or component.
    boolean	producesSortedKeys(PipelineOptions options) Returns whether the delegate source produces sorted keys.
    static <T> Read.Bounded<T>	readFromSource(FileBasedSource<T> sourceDelegate, CompressedSource.DecompressingChannelFactory channelFactory) Creates a Read transform that reads from that reads from the underlying FileBasedSource sourceDelegate after decompressing it with a CompressedSource.DecompressingChannelFactory.
    void	validate() Validates that the delegate source is a valid source and that the channel factory is not null.
    CompressedSource<T>	withDecompression(CompressedSource.DecompressingChannelFactory channelFactory) Return a CompressedSource that is like this one but will decompress its underlying file with the given CompressedSource.DecompressingChannelFactory.

    • Methods inherited from class com.google.cloud.dataflow.sdk.io.FileBasedSource

      createReader, createSourceForSubrange, expandFilePattern, getEstimatedSizeBytes, getFileOrPatternSpec, getFileOrPatternSpecProvider, getMaxEndOffset, getMode, splitIntoBundles, toString

    • Methods inherited from class com.google.cloud.dataflow.sdk.io.OffsetBasedSource

      allowsDynamicSplitting, getBytesPerOffset, getEndOffset, getMinBundleSize, getStartOffset

    • Methods inherited from class java.lang.Object

      clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

• • Method Detail

    • readFromSource

      public static <T> Read.Bounded<T> readFromSource(FileBasedSource<T> sourceDelegate,
                                                       CompressedSource.DecompressingChannelFactory channelFactory)

      Creates a Read transform that reads from that reads from the underlying FileBasedSource sourceDelegate after decompressing it with a CompressedSource.DecompressingChannelFactory.

    • from

      public static <T> CompressedSource<T> from(FileBasedSource<T> sourceDelegate)

      Creates a CompressedSource from an underlying FileBasedSource. The type of compression used will be based on the file name extension unless explicitly configured via withDecompression(com.google.cloud.dataflow.sdk.io.CompressedSource.DecompressingChannelFactory).

    • withDecompression

      public CompressedSource<T> withDecompression(CompressedSource.DecompressingChannelFactory channelFactory)

      Return a CompressedSource that is like this one but will decompress its underlying file with the given CompressedSource.DecompressingChannelFactory.

    • validate

      public void validate()

      Validates that the delegate source is a valid source and that the channel factory is not null.

      Overrides:

      validate in class FileBasedSource<T>

    • createForSubrangeOfFile

      protected FileBasedSource<T> createForSubrangeOfFile(String fileName,
                                                           long start,
                                                           long end)

      Creates a CompressedSource for a subrange of a file. Called by superclass to create a source for a single file.

      Specified by:

      createForSubrangeOfFile in class FileBasedSource<T>

      Parameters:

      fileName - file backing the new FileBasedSource.

      start - starting byte offset of the new FileBasedSource.

      end - ending byte offset of the new FileBasedSource. May be Long.MAX_VALUE, in which case it will be inferred using FileBasedSource.getMaxEndOffset(com.google.cloud.dataflow.sdk.options.PipelineOptions).

    • isSplittable

      protected final boolean isSplittable()
                                    throws Exception

      Determines whether a single file represented by this source is splittable. Returns true if we are using the default decompression factory and and it determines from the requested file name that the file is not compressed.

      Overrides:

      isSplittable in class FileBasedSource<T>

      Throws:

      Exception

    • createSingleFileReader

      protected final FileBasedSource.FileBasedReader<T> createSingleFileReader(PipelineOptions options)

      Creates a FileBasedReader to read a single file.

      Uses the delegate source to create a single file reader for the delegate source. Utilizes the default decompression channel factory to not wrap the source reader if the file name does not represent a compressed file allowing for splitting of the source.

      Specified by:

      createSingleFileReader in class FileBasedSource<T>

    • producesSortedKeys

      public final boolean producesSortedKeys(PipelineOptions options)
                                       throws Exception

      Returns whether the delegate source produces sorted keys.

      Specified by:

      producesSortedKeys in class BoundedSource<T>

      Throws:

      Exception

    • populateDisplayData

      public void populateDisplayData(DisplayData.Builder builder)

      Description copied from class: Source
      Register display data for the given transform or component.

      populateDisplayData(DisplayData.Builder) is invoked by Pipeline runners to collect display data via DisplayData.from(HasDisplayData). Implementations may call super.populateDisplayData(builder) in order to register display data in the current namespace, but should otherwise use subcomponent.populateDisplayData(builder) to use the namespace of the subcomponent.

      By default, does not register any display data. Implementors may override this method to provide their own display data.

      Specified by:

      populateDisplayData in interface HasDisplayData

      Overrides:

      populateDisplayData in class FileBasedSource<T>

      Parameters:

      builder - The builder to populate with display data.

      See Also:

      HasDisplayData

    • getDefaultOutputCoder

      public final Coder<T> getDefaultOutputCoder()

      Returns the delegate source's default output coder.

      Specified by:

      getDefaultOutputCoder in class Source<T>

    • getChannelFactory

      public final CompressedSource.DecompressingChannelFactory getChannelFactory()