Transforms

Warning: Dataflow SDK 1.x for Java is unsupported as of October 16, 2018. After August 12, 2020, Dataflow will not run jobs using Dataflow 1.x and below. See Migrating from Dataflow SDK 1.x for Java for migration guidance.

The documentation on this page applies only to the Dataflow SDK 1.x for Java.

The Dataflow SDK 2.x for Java and the Dataflow SDK for Python are based on Apache Beam. See the documentation for those SDKs.

In a Dataflow pipeline, a transform represents a step, or a processing operation that transforms data. A transform can perform nearly any kind of processing operation, including performing mathematical computations on data, converting data from one format to another, grouping data together, reading and writing data, filtering data to output only the elements you want, or combining data elements into single values.

Transforms in the Dataflow model can be nested—that is, transforms can contain and invoke other transforms, thus forming composite transforms.

How Transforms Work

Transforms represent your pipeline's processing logic. Each transform accepts one (or multiple) PCollections as input, performs an operation on the elements in the input PCollection(s), and produces one (or multiple) new PCollections as output.

Java

To use a transform, you apply the transform to the input PCollection that you want to process by calling the apply method on the input PCollection. When you call PCollection.apply, you pass the transform you want to use as an argument. The output PCollection is the return value from PCollection.apply.

For example, the following code sample shows how to apply a user-defined transform called ComputeWordLengths to a PCollection<String>. ComputeWordLengths returns a new PCollection<Integer> containing the length of each String in the input collection:

  // The input PCollection of word strings.
  PCollection<String> words = ...;

  // The ComputeWordLengths transform, which takes a PCollection of Strings as input and
  // returns a PCollection of Integers as output.
  static class ComputeWordLengths
      extends PTransform<PCollection<String>, PCollection<Integer>> { ... }

  // Apply ComputeWordLengths, capturing the results as the PCollection wordLengths.
  PCollection<Integer> wordLengths = words.apply(new ComputeWordLengths());

When you build a pipeline with a Dataflow program, the transforms you include might not be executed precisely in the order you specify them. The Cloud Dataflow managed service, for example, performs optimized execution. In optimized execution, the Dataflow service orders transforms in dependency order, inferring the exact sequence from the inputs and outputs defined in your pipeline. Certain transforms may be merged or executed in a different order to provide the most efficient execution.

Types of Transforms in the Dataflow SDKs

Core Transforms

The Dataflow SDK contains a small group of core transforms that are the foundation of the Cloud Dataflow parallel processing model. Core transforms form the basic building blocks of pipeline processing. Each core transform provides a generic processing framework for applying business logic that you provide to the elements of a PCollection.

When you use a core transform, you provide the processing logic as a function object. The function you provide gets applied to the elements of the input PCollection(s). Instances of the function may be executed in parallel across multiple Google Compute Engine instances, given a large enough data set, and pending optimizations performed by the pipeline runner service. The worker code function produces the output elements, if any, that are added to the output PCollection(s).

Requirements for User-Provided Function Objects

The function objects you provide for a transform might have many copies executing in parallel across multiple Compute Engine instances in your Cloud Platform project. As such, you should consider a few factors when creating such a function:

• Your function object must be serializable.
• Your function object must be thread-compatible, and be aware that the Dataflow SDKs are not thread-safe.
• We recommend making your function object idempotent.

These requirements apply to subclasses of DoFn (used with the ParDo core transform), CombineFn (used with the Combine core transform), and WindowFn (used with the Window transform).

Serializability

The function object you provide to a core transform must be fully serializable. The base classes for user code, such as DoFn, CombineFn, and WindowFn, already implement Serializable. However, your subclass must not add any non-serializable members.

Some other serializability factors for which you must account:

• Transient fields in your function object are not carried down to worker instances in your Cloud Platform project, because they are not automatically serialized.
• Avoid loading large amounts of data into a field before serialization.
• Individual instances of function objects cannot share data.
• Mutating a function object after it gets applied has no effect.
• Take care when you declare your function object inline by using an anonymous inner class instance. In a non-static context, your inner class instance will implicitly contain a pointer to the enclosing class and its state. That enclosing class will also be serialized, and thus the same considerations that apply to the function object itself also apply to this outer class.

Thread-Compatibility

Your function object should be thread-compatible. Each instance of your function object is accessed by a single thread on a worker instance, unless you explicitly create your own threads. Note, however, that the Dataflow SDKs are not thread-safe. If you create your own threads in your function object, you must provide your own synchronization. Note that static members are not passed to worker instances and that multiple instances of your function may be accessed from different threads.

Idempotency

We recommend making your function object idempotent— that is, for any given input, your function always provides the same output. Idempotency is not required, but making your functions idempotent makes your output deterministic and can make debugging and troubleshooting your transforms easier.

Types of Core Transforms

You will often use the core transforms directly in your pipeline. In addition, many of the other transforms provided in the Dataflow SDKs are implemented in terms of the core transforms.

The Dataflow SDKs define the following core transforms:

• ParDo for generic parallel processing
• GroupByKey for Key-Grouping Key/Value pairs
• Combine for combining collections or grouped values
• Flatten for merging collections

Composite Transforms

The Dataflow SDKs support composite transforms, which are transforms built from multiple sub-transforms. The model of transforms in the Dataflow SDKs is modular, in that you can build a transform that is implemented in terms of other transforms. You can think of a composite transform as a complex step in your pipeline that contains several nested steps.

Composite transforms are useful when you want to create a repeatable operation that involves multiple steps. Many of the built-in transforms included in the Dataflow SDKs, such as Count and Top, are this sort of composite transform. They are used in exactly the same manner as any other transform.

See Creating Composite Transforms for more information.

Pre-Written Transforms in the Dataflow SDKs

The Dataflow SDKs provide a number of pre-written transforms, which are both core and composite transforms where the processing logic is already written for you. These are more complex transforms for combining, splitting, manipulating, and performing statistical analysis on data.

Java

You can find these transforms in the com.google.cloud.dataflow.sdk.transforms package and its subpackages.

For a discussion on using the transforms provided in the Dataflow SDKs, see Transforms Included in the SDKs.

Root Transforms for Reading and Writing Data

The Dataflow SDKs provide specialized transforms, called root transforms, for getting data into and out of your pipeline. These transforms can be used at any time in your pipeline, but most often serve as your pipeline's root and endpoints. They include read transforms, write transforms, and create transforms.

Read transforms, which can serve as the root of your pipeline to create an initial PCollection, are used to create pipeline data from various sources. These sources can include text files in Google Cloud Storage, data stored in BigQuery or Pub/Sub, and other cloud storage sources. The Dataflow SDKs also provide an extensible API for working with your own custom data sources.

Write transforms can serve as pipeline endpoints to write PCollections containing processed output data to external storage. External data storage sinks can include text files in Google Cloud Storage, BigQuery tables, Pub/Sub, or other cloud storage mechanisms.

Create transforms are useful for creating a PCollection from in-memory data. See Creating a PCollection for more information.

For more information on read and write transforms, see Pipeline I/O.

Transforms with Multiple Inputs and Outputs

Some transforms accept multiple PCollection inputs, or specialized side inputs. A transform can also produce multiple PCollection outputs and side outputs. The Dataflow SDKs provide a tagging API to help you track and pass multiple inputs and outputs of different types.

To learn about transforms with multiple inputs and outputs and the details of the tagging system, see Handling Multiple PCollections.