SourceTestUtils (Google Cloud Dataflow SDK 1.9.1 API)

Google Cloud Dataflow SDK for Java, version 1.9.1

com.google.cloud.dataflow.sdk.testing

Class SourceTestUtils

    • Constructor Detail

      • SourceTestUtils

        public SourceTestUtils()
    • Method Detail

      • createStructuralValues

        public static <T> List<com.google.cloud.dataflow.sdk.testing.SourceTestUtils.ReadableStructuralValue<T>> createStructuralValues(Coder<T> coder,
                                                                                                                                        List<T> list)
                                                                                                                                 throws Exception
        Testing utilities below depend on standard assertions and matchers to compare elements read by sources. In general the elements may not implement equals/hashCode properly, however every source has a Coder and every Coder can produce a Coder.structuralValue(T) whose equals/hashCode is consistent with equality of encoded format. So we use this Coder.structuralValue(T) to compare elements read by sources.
        Throws:
        Exception
      • readFromUnstartedReader

        public static <T> List<T> readFromUnstartedReader(Source.Reader<T> reader)
                                                   throws IOException
        Reads all elements from the given unstarted Source.Reader.
        Throws:
        IOException
      • readFromStartedReader

        public static <T> List<T> readFromStartedReader(Source.Reader<T> reader)
                                                 throws IOException
        Reads all elements from the given started Source.Reader.
        Throws:
        IOException
      • readNItemsFromUnstartedReader

        public static <T> List<T> readNItemsFromUnstartedReader(Source.Reader<T> reader,
                                                                int n)
                                                         throws IOException
        Read elements from a Source.Reader until n elements are read.
        Throws:
        IOException
      • readNItemsFromStartedReader

        public static <T> List<T> readNItemsFromStartedReader(Source.Reader<T> reader,
                                                              int n)
                                                       throws IOException
        Read elements from a Source.Reader that has already had Source.Reader#start called on it, until n elements are read.
        Throws:
        IOException
      • readRemainingFromReader

        public static <T> List<T> readRemainingFromReader(Source.Reader<T> reader,
                                                          boolean started)
                                                   throws IOException
        Read all remaining elements from a Source.Reader.
        Throws:
        IOException
      • assertSourcesEqualReferenceSource

        public static <T> void assertSourcesEqualReferenceSource(BoundedSource<T> referenceSource,
                                                                 List<? extends BoundedSource<T>> sources,
                                                                 PipelineOptions options)
                                                          throws Exception
        Given a reference Source and a list of Sources, assert that the union of the records read from the list of sources is equal to the records read from the reference source.
        Throws:
        Exception
      • assertUnstartedReaderReadsSameAsItsSource

        public static <T> void assertUnstartedReaderReadsSameAsItsSource(BoundedSource.BoundedReader<T> reader,
                                                                         PipelineOptions options)
                                                                  throws Exception
        Assert that a Reader returns a Source that, when read from, produces the same records as the reader.
        Throws:
        Exception
      • assertSplitAtFractionSucceedsAndConsistent

        public static <T> void assertSplitAtFractionSucceedsAndConsistent(BoundedSource<T> source,
                                                                          int numItemsToReadBeforeSplit,
                                                                          double splitFraction,
                                                                          PipelineOptions options)
                                                                   throws Exception
        Verifies some consistency properties of BoundedSource.BoundedReader#splitAtFraction on the given source. Equivalent to the following pseudocode:
           Reader reader = source.createReader();
           read N items from reader;
           Source residual = reader.splitAtFraction(splitFraction);
           Source primary = reader.getCurrentSource();
           assert: items in primary == items we read so far
                                       + items we'll get by continuing to read from reader;
           assert: items in original source == items in primary + items in residual
         
        Throws:
        Exception
      • assertSplitAtFractionFails

        public static <T> void assertSplitAtFractionFails(BoundedSource<T> source,
                                                          int numItemsToReadBeforeSplit,
                                                          double splitFraction,
                                                          PipelineOptions options)
                                                   throws Exception
        Asserts that the source's reader fails to splitAtFraction(fraction) after reading numItemsToReadBeforeSplit items.
        Throws:
        Exception
      • assertSplitAtFractionExhaustive

        public static <T> void assertSplitAtFractionExhaustive(BoundedSource<T> source,
                                                               PipelineOptions options)
                                                        throws Exception
        Asserts that for each possible start position, BoundedSource.BoundedReader#splitAtFraction at every interesting fraction (halfway between two fractions that differ by at least one item) can be called successfully and the results are consistent if a split succeeds. Verifies multithreaded splitting as well.
        Throws:
        Exception


Send feedback about...

Cloud Dataflow