Pre-Written Transforms in the Cloud Dataflow SDK

Warning: Dataflow SDK 1.x for Java is unsupported as of October 16, 2018. See Migrating from Dataflow SDK 1.x for Java for migration guidance.

The documentation on this page applies only to the Dataflow SDK 1.x for Java.

The Dataflow SDK 2.x for Java and the Dataflow SDK for Python are based on Apache Beam. See the documentation for those SDKs.

The Dataflow SDKs provide a library of pre-written transforms that represent common and useful data processing operations. These transforms are either core transforms with generic processing functions already written for you, or composite transforms that combine simple pre-written transforms to perform useful processing functions.

Java

In the Dataflow SDK for Java, you can find these transforms in the package com.google.cloud.dataflow.sdk.transforms.

You can use any of the pre-written transforms in the Dataflow SDKs as-is in your own pipelines. These transforms are generic and convenient operations that can perform common data processing steps, such as counting elements in a collection, dividing a collection into quantiles, finding the top (or bottom) N elements in a collection, and performing basic mathematical combinations on numerical data.

Many of the pre-written transforms in the Dataflow SDKs are genericized composite transforms that can take different data types. They are made up of nested core transforms like ParDo, GroupByKey, and Combine.

Java

The Dataflow SDK for Java can represent most common data processing operations using core transforms. The pre-written transforms provided in the SDKs are essentially pre-built wrappers for generic ParDo, Combine, etc. transforms organized in such a way that they count elements or do basic mathematical combinations. Sum.integersGlobally, for example, wraps the Combine.Globally core transform for Integer types, and provides a pre-written CombineFn that computes the sum of all input elements. Rather than writing your own version of Combine.Globally with a sum CombineFn, you can use the pre-built provided in the SDK.

If the transforms included in the Dataflow SDKs don't fit your pipeline's use case exactly, you can create your own generic, reusable composite transforms. The source code for the included transforms can serve as a model for creating your own composite transforms using ParDo, Combine, and other core transforms. See Creating Composite Transforms for more information.

Common Processing Patterns

The transforms included in the Dataflow SDKs provide convenient mechanisms to perform common data processing operations in your pipelines. The source code for these transforms illustrate how the core transforms like ParDo can be used (or re-used) for various operations.

Simple ParDo Wrappers

Some of the simplest transforms provided in the Dataflow SDKs are utility transforms for dealing with key/value pairs. Given a PCollection of key/value pairs, the Keys transform returns a PCollection containing only the keys; the Values transform returns a PCollection containing only the values. The KvSwap transform swaps the key element and the value element for each key/value pair, and returns a PCollection of the reversed pairs.

Java

Keys, Values, KvSwap, MapElements, FlatMapElements, Filter, and Partition are simple transforms composed of a single ParDo. In each case, the ParDo invokes a relatively simple DoFn to produce the elements of the output PCollection.

Here's the apply method for the Keys transform, which accepts a generic PCollection of KV<K, V> elements, and returns a PCollection<K> of only the keys from the key/value pairs:

  @Override
  public PCollection<K> apply(PCollection<? extends KV<K, ?>> in) {
    return
        in.apply(ParDo.named("Keys")
                 .of(new DoFn<KV<K, ?>, K>() {
                     @Override
                     public void processElement(ProcessContext c) {
                       c.output(c.element().getKey());
                     }
                    }));
  }

In the example, the apply method applies a ParDo transform to the input collection (in). That ParDo invokes a simple DoFn to output the key part of the key/value pair. The DoFn is trivial, and can be defined as an anonymous inner class instance.

Patterns That Combine Elements

The Dataflow SDKs contain a number of convenient transforms that perform common statistical and mathematical combinations on elements. For example, there are transforms that accept a PCollection of numerical data (such as integers) and perform a mathematical combination: finding the sum of all elements, the mean average of all elements, or the largest/smallest of all elements in the collection. Examples of transforms of this kind are Sum and Mean.

Other transforms perform basic statistical analysis on a collection: finding the top N elements, for example, or returning a random sample of every N elements in a given PCollection. Examples of transforms of this kind include Top and Sample.

Java

These transforms are based on the Combine core transform. They include variants that work on PCollections of individual values (using Combine.globally) and PCollections of key/value pairs (using Combine.perKey).

See the source and API for Java reference documentation for the Top transform for an example of a combining transform with both global and per-key variants.

Map/Shuffle/Reduce-Style Processing

Some of the transforms included in the Dataflow SDKs perform processing similar to a Map/Shuffle/Reduce-style algorithm. These transforms include Count, which accepts a potentially non-unique collection of elements and returns a reduced collection of just the unique elements paired with an occurrence count for each. The RemoveDuplicates transform likewise reduces a non-unique collection to just the unique elements, but does not provide an occurrence count.

Java

These transforms make use of the core ParDo and Combine.perKey transforms. Combine.perKey is itself a composite operation that performs a GroupByKey and combines the resulting value stream for each key into a single value. The ParDo represents the Map phase of Map/Shuffle/Reduce; Combine.perKey represents the Shuffle and Reduce phases.

Here's the apply method for the Count transform, showing the processing logic in the nested ParDo and Combine.perKey transforms:

  @Override
  public PCollection<KV<T, Long>> apply(PCollection<T> in) {
    return
        in
        .apply(ParDo.named("Init")
               .of(new DoFn<T, KV<T, Long>>() {
                   @Override
                   public void processElement(ProcessContext c) {
                     c.output(KV.of(c.element(), 1L));
                   }
                 }))

        .apply(Combine.<T, Long>perKey(
                 new SerializableFunction<Iterable<Long>, Long>() {
                   @Override
                   public Long apply(Iterable<Long> values) {
                     long sum = 0;
                     for (Long value : values) {
                       sum += value;
                     }
                     return sum;
                   }
                 }));
  }

In the example, the apply method uses a ParDo transform to attach an occurrence count to each element in the input PCollection, creating a key/value pair for each element—this is the Map phase of Map/Shuffle/Reduce. Count then applies a Combine.perKey transform to perform the Shuffle and Reduce logic, and produce a PCollection of unique elements with a combined count of occurrences.