TestPipeline (Google Cloud Dataflow SDK 1.9.1 API)

Google Cloud Dataflow SDK for Java, version 1.9.1

Warning: Dataflow SDK 1.x for Java is unsupported as of October 16, 2018. See Migrating from Dataflow SDK 1.x for Java for migration guidance.

Note: If you are using Cloud Dataflow SDK version 2.x, see the Apache Beam API reference documentation.

com.google.cloud.dataflow.sdk.testing

Class TestPipeline

• java.lang.Object
• • com.google.cloud.dataflow.sdk.Pipeline
  • • com.google.cloud.dataflow.sdk.testing.TestPipeline

• ---

  
  

  public class TestPipeline
  extends Pipeline

  A creator of test pipelines that can be used inside of tests that can be configured to run locally or against the live service.

  It is recommended to tag hand-selected tests for this purpose using the @RunnableOnService category annotation, and use a category aware test executor such as JUnit. Each test run against the service will spin up and tear down VMs.

  In order to run tests on Google Cloud Dataflow, the following conditions must be met:

  • System property runIntegrationTestOnService must be set to true.
  • System property dataflowOptions must contain a JSON delimited list of pipeline options with all options that the Google Cloud Dataflow service requires. For example:

     [
         "--project=mygcpproject",
         "--stagingLocation=gs://mygcsbucket/path"
         ]

    Note that the set of pipeline options required can also be test specific.
  • Jars containing the SDK and test classes must be available on the classpath.

  Use DataflowAssert for tests, as it integrates with this test harness in both direct and remote execution modes. For example:

  
   Pipeline p = TestPipeline.create();
   PCollection<Integer> output = ...
  
   DataflowAssert.that(output)
       .containsInAnyOrder(1, 2, 3, 4);
   p.run();
   

• • Nested Class Summary

    • Nested classes/interfaces inherited from class com.google.cloud.dataflow.sdk.Pipeline

      Pipeline.PipelineExecutionException, Pipeline.PipelineVisitor

  • Method Summary

    All Methods Static Methods Instance Methods Concrete Methods 

    Modifier and Type	Method and Description
    static TestPipeline	create() Creates and returns a new test pipeline.
    static TestPipeline	fromOptions(PipelineOptions options)
    PipelineResult	run() Runs this TestPipeline, unwrapping any AssertionError that is raised during testing.
    boolean	supportsDynamicWorkRebalancing() Returns whether a TestPipeline supports dynamic work rebalancing, and thus tests of dynamic work rebalancing are expected to pass.
    static PipelineOptions	testingPipelineOptions() Creates PipelineOptions for testing.
    String	toString()

    • Methods inherited from class com.google.cloud.dataflow.sdk.Pipeline

      addValueInternal, apply, apply, applyTransform, applyTransform, begin, create, getCoderRegistry, getFullNameForTesting, getOptions, getRunner, setCoderRegistry, traverseTopologically

    • Methods inherited from class java.lang.Object

      clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

• • Method Detail

    • create

      public static TestPipeline create()

      Creates and returns a new test pipeline.

      Use DataflowAssert to add tests, then call Pipeline.run() to execute the pipeline and check the tests.

    • fromOptions

      public static TestPipeline fromOptions(PipelineOptions options)

    • supportsDynamicWorkRebalancing

      public boolean supportsDynamicWorkRebalancing()

      Returns whether a TestPipeline supports dynamic work rebalancing, and thus tests of dynamic work rebalancing are expected to pass.

    • run

      public PipelineResult run()

      Runs this TestPipeline, unwrapping any AssertionError that is raised during testing.

      Overrides:

      run in class Pipeline

    • toString

      public String toString()

      Overrides:

      toString in class Pipeline

    • testingPipelineOptions

      public static PipelineOptions testingPipelineOptions()

      Creates PipelineOptions for testing.