CloudBigtableIO.SourceWithKeys (Apache Beam + Cloud Bigtable Connector 1.12.0 API)

com.google.cloud.bigtable.beam

Class CloudBigtableIO.SourceWithKeys

• java.lang.Object
• • org.apache.beam.sdk.io.Source<T>
  • • org.apache.beam.sdk.io.BoundedSource<Result>
    • • com.google.cloud.bigtable.beam.CloudBigtableIO.SourceWithKeys

• All Implemented Interfaces:

  Serializable, HasDisplayData

  Enclosing class:

  CloudBigtableIO

  ---

  
  

  protected static class CloudBigtableIO.SourceWithKeys
  extends BoundedSource<Result>

  A BoundedSource for a Cloud Bigtable Table with a start/stop key range, along with a potential filter via a Scan.

  See Also:

  Serialized Form

• • Nested Class Summary

    • Nested classes/interfaces inherited from class org.apache.beam.sdk.io.BoundedSource

      BoundedSource.BoundedReader<T>

    • Nested classes/interfaces inherited from class org.apache.beam.sdk.io.Source

      Source.Reader<T>

  • Field Summary

    Fields 

    Modifier and Type	Field and Description
    protected static long	SIZED_BASED_MAX_SPLIT_COUNT
    protected static org.slf4j.Logger	SOURCE_LOG

  • Constructor Summary

    Constructors 

    Modifier	Constructor and Description
    protected	SourceWithKeys(CloudBigtableScanConfiguration configuration, long estimatedSize)

  • Method Summary

    All Methods Static Methods Instance Methods Concrete Methods 

    Modifier and Type	Method and Description
    BoundedSource.BoundedReader<Result>	createReader(PipelineOptions options) Creates a reader that will scan the entire table based on the Scan in the configuration.
    protected CloudBigtableScanConfiguration	getConfiguration()
    long	getEstimatedSize()
    long	getEstimatedSizeBytes(PipelineOptions options) Gets an estimate of the size of the source.
    Coder<Result>	getOutputCoder()
    List<com.google.bigtable.repackaged.com.google.cloud.bigtable.data.v2.models.KeyOffset>	getSampleRowKeys() Performs a call to get sample row keys from CloudBigtableServiceImpl.getSampleRowKeys(CloudBigtableTableConfiguration) if they are not yet cached.
    protected List<CloudBigtableIO.SourceWithKeys>	getSplits(long desiredBundleSizeBytes)
    protected static boolean	isWithinRange(byte[] scanStartKey, byte[] scanEndKey, byte[] startKey, byte[] endKey) Checks if the range of the region is within the range of the scan.
    void	populateDisplayData(DisplayData.Builder builder)
    protected List<CloudBigtableIO.SourceWithKeys>	split(long regionSize, long desiredBundleSizeBytes, byte[] startKey, byte[] stopKey) Splits the region based on the start and stop key.
    List<? extends BoundedSource<Result>>	split(long desiredBundleSizeBytes, PipelineOptions options) Splits the bundle based on the assumption that the data is distributed evenly between startKey and stopKey.
    String	toString()
    void	validate() Validates the existence of the table in the configuration.

    • Methods inherited from class org.apache.beam.sdk.io.Source

      getDefaultOutputCoder

    • Methods inherited from class java.lang.Object

      clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

• • Field Detail

    • SOURCE_LOG

      protected static final org.slf4j.Logger SOURCE_LOG

    • SIZED_BASED_MAX_SPLIT_COUNT

      protected static final long SIZED_BASED_MAX_SPLIT_COUNT

      See Also:

      Constant Field Values

  • Constructor Detail

    • SourceWithKeys

      protected SourceWithKeys(CloudBigtableScanConfiguration configuration,
                               long estimatedSize)

  • Method Detail

    • getEstimatedSizeBytes

      public long getEstimatedSizeBytes(PipelineOptions options)

      Gets an estimate of the size of the source.

      NOTE: This value is a guesstimate. It could be significantly off, especially if there is aScan selected in the configuration. It will also be off if the start and stop keys are calculated via CloudBigtableIO.Source.split(long, PipelineOptions).

      Parameters:

      options - The pipeline options.

      Returns:

      The estimated size of the source, in bytes.

    • getEstimatedSize

      public long getEstimatedSize()

    • split

      public List<? extends BoundedSource<Result>> split(long desiredBundleSizeBytes,
                                                         PipelineOptions options)
                                                  throws Exception

      Splits the bundle based on the assumption that the data is distributed evenly between startKey and stopKey. That assumption may not be correct for any specific start/stop key combination.

      This method is called internally by Beam. Do not call it directly.

      Specified by:

      split in class BoundedSource<Result>

      Parameters:

      desiredBundleSizeBytes - The desired size for each bundle, in bytes.

      options - The pipeline options.

      Returns:

      A list of sources split into groups.

      Throws:

      Exception

    • getOutputCoder

      public Coder<Result> getOutputCoder()

    • toString

      public String toString()

      Overrides:

      toString in class Object

    • getSplits

      protected List<CloudBigtableIO.SourceWithKeys> getSplits(long desiredBundleSizeBytes)
                                                        throws Exception

      Throws:

      Exception

    • isWithinRange

      protected static boolean isWithinRange(byte[] scanStartKey,
                                             byte[] scanEndKey,
                                             byte[] startKey,
                                             byte[] endKey)

      Checks if the range of the region is within the range of the scan.

    • getSampleRowKeys

      public List<com.google.bigtable.repackaged.com.google.cloud.bigtable.data.v2.models.KeyOffset> getSampleRowKeys()
                                                                                                               throws IOException

      Performs a call to get sample row keys from CloudBigtableServiceImpl.getSampleRowKeys(CloudBigtableTableConfiguration) if they are not yet cached. The sample row keys give information about tablet key boundaries and estimated sizes.

      Throws:

      IOException

    • validate

      public void validate()

      Validates the existence of the table in the configuration.

      Overrides:

      validate in class Source<Result>

    • split

      protected List<CloudBigtableIO.SourceWithKeys> split(long regionSize,
                                                           long desiredBundleSizeBytes,
                                                           byte[] startKey,
                                                           byte[] stopKey)
                                                    throws IOException

      Splits the region based on the start and stop key. Uses Bytes.split(byte[], byte[], int) under the covers.

      Throws:

      IOException

    • createReader

      public BoundedSource.BoundedReader<Result> createReader(PipelineOptions options)

      Creates a reader that will scan the entire table based on the Scan in the configuration.

      Specified by:

      createReader in class BoundedSource<Result>

      Returns:

      A reader for the table.

    • getConfiguration

      protected CloudBigtableScanConfiguration getConfiguration()

      Returns:

      the configuration

    • populateDisplayData

      public void populateDisplayData(DisplayData.Builder builder)

      Specified by:

      populateDisplayData in interface HasDisplayData

      Overrides:

      populateDisplayData in class Source<Result>