Google Cloud Dataflow SDK for Java, version 1.9.1
Class TestPipeline
- java.lang.Object
-
- com.google.cloud.dataflow.sdk.Pipeline
-
- com.google.cloud.dataflow.sdk.testing.TestPipeline
-
public class TestPipeline extends Pipeline
A creator of test pipelines that can be used inside of tests that can be configured to run locally or against the live service.It is recommended to tag hand-selected tests for this purpose using the
@RunnableOnService
category annotation, and use a category aware test executor such as JUnit. Each test run against the service will spin up and tear down VMs.In order to run tests on Google Cloud Dataflow, the following conditions must be met:
- System property
runIntegrationTestOnService
must be set to true. - System property
dataflowOptions
must contain a JSON delimited list of pipeline options with all options that the Google Cloud Dataflow service requires. For example:
Note that the set of pipeline options required can also be test specific.[ "--project=mygcpproject", "--stagingLocation=gs://mygcsbucket/path" ]
- Jars containing the SDK and test classes must be available on the classpath.
Use
DataflowAssert
for tests, as it integrates with this test harness in both direct and remote execution modes. For example:Pipeline p = TestPipeline.create(); PCollection<Integer> output = ... DataflowAssert.that(output) .containsInAnyOrder(1, 2, 3, 4); p.run();
- System property
-
-
Nested Class Summary
-
Nested classes/interfaces inherited from class com.google.cloud.dataflow.sdk.Pipeline
Pipeline.PipelineExecutionException, Pipeline.PipelineVisitor
-
-
Method Summary
All Methods Static Methods Instance Methods Concrete Methods Modifier and Type Method and Description static TestPipeline
create()
Creates and returns a new test pipeline.static TestPipeline
fromOptions(PipelineOptions options)
PipelineResult
run()
Runs thisTestPipeline
, unwrapping anyAssertionError
that is raised during testing.boolean
supportsDynamicWorkRebalancing()
Returns whether aTestPipeline
supports dynamic work rebalancing, and thus tests of dynamic work rebalancing are expected to pass.static PipelineOptions
testingPipelineOptions()
CreatesPipelineOptions
for testing.String
toString()
-
Methods inherited from class com.google.cloud.dataflow.sdk.Pipeline
addValueInternal, apply, apply, applyTransform, applyTransform, begin, create, getCoderRegistry, getFullNameForTesting, getOptions, getRunner, setCoderRegistry, traverseTopologically
-
-
-
-
Method Detail
-
create
public static TestPipeline create()
Creates and returns a new test pipeline.Use
DataflowAssert
to add tests, then callPipeline.run()
to execute the pipeline and check the tests.
-
fromOptions
public static TestPipeline fromOptions(PipelineOptions options)
-
supportsDynamicWorkRebalancing
public boolean supportsDynamicWorkRebalancing()
Returns whether aTestPipeline
supports dynamic work rebalancing, and thus tests of dynamic work rebalancing are expected to pass.
-
run
public PipelineResult run()
Runs thisTestPipeline
, unwrapping anyAssertionError
that is raised during testing.
-
testingPipelineOptions
public static PipelineOptions testingPipelineOptions()
CreatesPipelineOptions
for testing.
-
-