DoFnTester (Google Cloud Dataflow SDK 1.9.1 API)

Google Cloud Dataflow SDK for Java, version 1.9.1

java.lang.Object
- com.google.cloud.dataflow.sdk.transforms.DoFnTester<InputT,OutputT>

Type Parameters:: InputT - the type of the DoFn's (main) input elements; OutputT - the type of the DoFn's (main) output elements

public class DoFnTester<InputT,OutputT>
extends Object

A harness for unit-testing a DoFn.

For example:

 
 DoFn<InputT, OutputT> fn = ...;

 DoFnTester<InputT, OutputT> fnTester = DoFnTester.of(fn);

 // Set arguments shared across all batches:
 fnTester.setSideInputs(...);      // If fn takes side inputs.
 fnTester.setSideOutputTags(...);  // If fn writes to side outputs.

 // Process a batch containing a single input element:
 Input testInput = ...;
 List<OutputT> testOutputs = fnTester.processBatch(testInput);
 Assert.assertThat(testOutputs, Matchers.hasItems(...));

 // Process a bigger batch:
 Assert.assertThat(fnTester.processBatch(i1, i2, ...), Matchers.hasItems(...));

Nested Class Summary

Nested Classes
Modifier and Type	Class and Description
`static class`	`DoFnTester.CloningBehavior` Whether or not a `DoFnTester` should clone the `DoFn` under test.
`static class`	`DoFnTester.OutputElementWithTimestamp<OutputT>` Holder for an OutputElement along with its associated timestamp.

Method Summary

All Methods Static Methods Instance Methods Concrete Methods
Modifier and Type	Method and Description
`void`	`clearOutputElements()` Clears the record of the elements output so far to the main output.
`<T> void`	`clearSideOutputElements(TupleTag<T> tag)` Clears the record of the elements output so far to the side output with the given tag.
`void`	`finishBundle()` Calls `DoFn.finishBundle(com.google.cloud.dataflow.sdk.transforms.DoFn<InputT, OutputT>.Context)` of the `DoFn` under test.
`<AggregateT> AggregateT`	`getAggregatorValue(Aggregator<?,AggregateT> agg)` Returns the value of the provided `Aggregator`.
`DoFnTester.CloningBehavior`	`getCloningBehavior()` Indicates whether this `DoFnTester` will clone the `DoFn` under test.
`static <InputT,OutputT> DoFnTester<InputT,OutputT>`	`of(DoFn<InputT,OutputT> fn)` Returns a `DoFnTester` supporting unit-testing of the given `DoFn`.
`static <InputT,OutputT> DoFnTester<InputT,OutputT>`	`of(DoFnWithContext<InputT,OutputT> fn)` Returns a `DoFnTester` supporting unit-testing of the given `DoFn`.
`List<OutputT>`	`peekOutputElements()` Returns the elements output so far to the main output.
`List<DoFnTester.OutputElementWithTimestamp<OutputT>>`	`peekOutputElementsWithTimestamp()` Returns the elements output so far to the main output with associated timestamps.
`<T> List<T>`	`peekSideOutputElements(TupleTag<T> tag)` Returns the elements output so far to the side output with the given tag.
`List<OutputT>`	`processBatch(InputT... inputElements)` A convenience method for testing `DoFns` with bundles of elements.
`List<OutputT>`	`processBatch(Iterable<? extends InputT> inputElements)` A convenience operation that first calls `startBundle()`, then calls `processElement(InputT)` on each of the input elements, then calls `finishBundle()`, then returns the result of `takeOutputElements()`.
`void`	`processElement(InputT element)` Calls `DoFn.processElement(com.google.cloud.dataflow.sdk.transforms.DoFn<InputT, OutputT>.ProcessContext)` on the `DoFn` under test, in a context where `DoFn.ProcessContext.element()` returns the given element.
`void`	`setCloningBehavior(DoFnTester.CloningBehavior newValue)` Instruct this `DoFnTester` whether or not to clone the `DoFn` under test.
`void`	`setSideInput(PCollectionView<?> sideInput, Iterable<com.google.cloud.dataflow.sdk.util.WindowedValue<?>> value)` Registers the values of a side input `PCollectionView` to pass to the `DoFn` under test.
`void`	`setSideInputInGlobalWindow(PCollectionView<?> sideInput, Iterable<?> value)` Registers the values for a side input `PCollectionView` to pass to the `DoFn` under test.
`void`	`setSideInputs(Map<PCollectionView<?>,Iterable<com.google.cloud.dataflow.sdk.util.WindowedValue<?>>> sideInputs)` Registers the tuple of values of the side input `PCollectionView`s to pass to the `DoFn` under test.
`void`	`setSideOutputTags(TupleTagList sideOutputTags)` Registers the list of `TupleTag`s that can be used by the `DoFn` under test to output to side output `PCollection`s.
`void`	`startBundle()` Calls `DoFn.startBundle(com.google.cloud.dataflow.sdk.transforms.DoFn<InputT, OutputT>.Context)` on the `DoFn` under test.
`List<OutputT>`	`takeOutputElements()` Returns the elements output so far to the main output.
`List<DoFnTester.OutputElementWithTimestamp<OutputT>>`	`takeOutputElementsWithTimestamp()` Returns the elements output so far to the main output with associated timestamps.
`<T> List<T>`	`takeSideOutputElements(TupleTag<T> tag)` Returns the elements output so far to the side output with the given tag.

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

- Method Detail
  - of
```
public static <InputT,OutputT> DoFnTester<InputT,OutputT> of(DoFn<InputT,OutputT> fn)
```
    Returns a DoFnTester supporting unit-testing of the given DoFn.
  - of
```
public static <InputT,OutputT> DoFnTester<InputT,OutputT> of(DoFnWithContext<InputT,OutputT> fn)
```
    Returns a DoFnTester supporting unit-testing of the given DoFn.
  - setSideInputs
```
public void setSideInputs(Map<PCollectionView<?>,Iterable<com.google.cloud.dataflow.sdk.util.WindowedValue<?>>> sideInputs)
```
    Registers the tuple of values of the side input PCollectionViews to pass to the DoFn under test.
    If needed, first creates a fresh instance of the DoFn under test.
    If this isn't called, DoFnTester assumes the DoFn takes no side inputs.
  - setSideInput
```
public void setSideInput(PCollectionView<?> sideInput,
                         Iterable<com.google.cloud.dataflow.sdk.util.WindowedValue<?>> value)
```
    Registers the values of a side input PCollectionView to pass to the DoFn under test.
    If needed, first creates a fresh instance of the DoFn under test.
    If this isn't called, DoFnTester assumes the DoFn takes no side inputs.
  - setSideInputInGlobalWindow
```
public void setSideInputInGlobalWindow(PCollectionView<?> sideInput,
                                       Iterable<?> value)
```
    Registers the values for a side input PCollectionView to pass to the DoFn under test. All values are placed in the global window.
  - setSideOutputTags
```
public void setSideOutputTags(TupleTagList sideOutputTags)
```
    Registers the list of TupleTags that can be used by the DoFn under test to output to side output PCollections.
    If needed, first creates a fresh instance of the DoFn under test.
    If this isn't called, DoFnTester assumes the DoFn doesn't emit to any side outputs.
  - setCloningBehavior
```
public void setCloningBehavior(DoFnTester.CloningBehavior newValue)
```
    Instruct this DoFnTester whether or not to clone the DoFn under test.
  - getCloningBehavior
```
public DoFnTester.CloningBehavior getCloningBehavior()
```
    Indicates whether this DoFnTester will clone the DoFn under test.
  - processBatch
```
public List<OutputT> processBatch(Iterable<? extends InputT> inputElements)
```
    A convenience operation that first calls startBundle(), then calls processElement(InputT) on each of the input elements, then calls finishBundle(), then returns the result of takeOutputElements().
  - processBatch
```
@SafeVarargs
public final List<OutputT> processBatch(InputT... inputElements)
```
    A convenience method for testing DoFns with bundles of elements. Logic proceeds as follows:
    1. Calls startBundle().
    2. Calls processElement(InputT) on each of the arguments.
    4. Calls finishBundle().
    5. Returns the result of takeOutputElements().
  - startBundle
```
public void startBundle()
```
    Calls DoFn.startBundle(com.google.cloud.dataflow.sdk.transforms.DoFn<InputT, OutputT>.Context) on the DoFn under test.
    If needed, first creates a fresh instance of the DoFn under test.
  - processElement
```
public void processElement(InputT element)
```
    Calls DoFn.processElement(com.google.cloud.dataflow.sdk.transforms.DoFn<InputT, OutputT>.ProcessContext) on the DoFn under test, in a context where DoFn.ProcessContext.element() returns the given element.
    Will call startBundle() automatically, if it hasn't already been called.
    
    Throws:
    
    IllegalStateException - if the DoFn under test has already been finished
  - finishBundle
```
public void finishBundle()
```
    Calls DoFn.finishBundle(com.google.cloud.dataflow.sdk.transforms.DoFn<InputT, OutputT>.Context) of the DoFn under test.
    Will call startBundle() automatically, if it hasn't already been called.
    
    Throws:
    
    IllegalStateException - if the DoFn under test has already been finished
  - peekOutputElements
```
public List<OutputT> peekOutputElements()
```
    Returns the elements output so far to the main output. Does not clear them, so subsequent calls will continue to include these elements.
    
    See Also:
    
    takeOutputElements(), clearOutputElements()
  - peekOutputElementsWithTimestamp
```
@Experimental
public List<DoFnTester.OutputElementWithTimestamp<OutputT>> peekOutputElementsWithTimestamp()
```
    Returns the elements output so far to the main output with associated timestamps. Does not clear them, so subsequent calls will continue to include these. elements.
    
    See Also:
    
    takeOutputElementsWithTimestamp(), clearOutputElements()
  - clearOutputElements
```
public void clearOutputElements()
```
    Clears the record of the elements output so far to the main output.
    
    See Also:
    
    peekOutputElements()
  - takeOutputElements
```
public List<OutputT> takeOutputElements()
```
    Returns the elements output so far to the main output. Clears the list so these elements don't appear in future calls.
    
    See Also:
    
    peekOutputElements()
  - takeOutputElementsWithTimestamp
```
@Experimental
public List<DoFnTester.OutputElementWithTimestamp<OutputT>> takeOutputElementsWithTimestamp()
```
    Returns the elements output so far to the main output with associated timestamps. Clears the list so these elements don't appear in future calls.
    
    See Also:
    
    peekOutputElementsWithTimestamp(), takeOutputElements(), clearOutputElements()
  - peekSideOutputElements
```
public <T> List<T> peekSideOutputElements(TupleTag<T> tag)
```
    Returns the elements output so far to the side output with the given tag. Does not clear them, so subsequent calls will continue to include these elements.
    
    See Also:
    
    takeSideOutputElements(com.google.cloud.dataflow.sdk.values.TupleTag<T>), clearSideOutputElements(com.google.cloud.dataflow.sdk.values.TupleTag<T>)
  - clearSideOutputElements
```
public <T> void clearSideOutputElements(TupleTag<T> tag)
```
    Clears the record of the elements output so far to the side output with the given tag.
    
    See Also:
    
    peekSideOutputElements(com.google.cloud.dataflow.sdk.values.TupleTag<T>)
  - takeSideOutputElements
```
public <T> List<T> takeSideOutputElements(TupleTag<T> tag)
```
    Returns the elements output so far to the side output with the given tag. Clears the list so these elements don't appear in future calls.
    
    See Also:
    
    peekSideOutputElements(com.google.cloud.dataflow.sdk.values.TupleTag<T>)
  - getAggregatorValue
```
public <AggregateT> AggregateT getAggregatorValue(Aggregator<?,AggregateT> agg)
```
    Returns the value of the provided Aggregator.

Was this page helpful? Let us know how we did:

Except as otherwise noted, the content of this page is licensed under the Creative Commons Attribution 4.0 License, and code samples are licensed under the Apache 2.0 License. For details, see our Site Policies. Java is a registered trademark of Oracle and/or its affiliates.

Last updated April 15, 2019.

DoFnTester (Google Cloud Dataflow SDK 1.9.1 API)

Google Cloud Dataflow SDK for Java, version 1.9.1

Class DoFnTester<InputT,OutputT>

Nested Class Summary

Method Summary

Methods inherited from class java.lang.Object

Method Detail

of

of

setSideInputs

setSideInput

setSideInputInGlobalWindow

setSideOutputTags

setCloningBehavior

getCloningBehavior

processBatch

processBatch

startBundle

processElement

finishBundle

peekOutputElements

peekOutputElementsWithTimestamp

clearOutputElements

takeOutputElements

takeOutputElementsWithTimestamp

peekSideOutputElements

clearSideOutputElements

takeSideOutputElements

getAggregatorValue

Send feedback about...