Stopping a running pipeline

If you need to stop a running Cloud Dataflow job, you can do so by issuing a command using either the Cloud Dataflow Monitoring Interface or the Cloud Dataflow Command-line Interface. There are two possible commands you can issue to stop your job: Cancel and Drain.

Note: The Drain command is currently only supported for streaming pipelines that use the Apache Beam SDK for Java.

Stopping a job using the Cloud Dataflow monitoring UI

To stop a job, select the job from the jobs list in the Cloud Dataflow Monitoring Interface. On the information card for your job, click Stop Job.

Figure 1: An information card for a Cloud Dataflow job, with the Stop Job button.

The Stop Job dialog appears with your options for how to stop your job:

Figure 2: The Stop Job dialog with options for Cancel and Drain.

Select Cancel or Drain option as appropriate and click the Stop Job button.

Cancel

Using the Cancel option to stop your job tells the Cloud Dataflow service to cancel your job immediately. The service will halt all data ingestion and processing as soon as possible and immediately begin cleaning up the Google Cloud Platform (GCP) resources attached to your job. These resources may include shutting down Compute Engine worker instances and closing active connections to I/O sources or sinks.

Because Cancel immediately halts processing, you may lose any "in-flight" data. "In-flight" data refers to data that has been read but is still being processed by your pipeline. Data written from your pipeline to an output sink before you issued the Cancel command may still be accessible on your output sink.

If data loss is not a concern, use the Cancel option to stop your to ensure the GCP resources associated with your job are shut down as soon as possible.

Drain

This section applies only to streaming jobs that use the Apache Beam SDK for Java. Currently, draining an existing pipeline has the following limitations:

• Draining a streaming pipeline that uses the Apache Beam SDK for Python is not supported.
• Draining batch pipelines is not supported.

Using the Drain option to stop your job tells the Cloud Dataflow service to finish your job in its current state. Your job will immediately stop ingesting new data from input sources. However, the Cloud Dataflow service will preserve any existing resources, such as worker instances, to finish processing and writing any buffered data in your pipeline. When all pending processing and write operations are complete, the Cloud Dataflow service will clean up the GCP resources associated with your job.

Note: Your pipeline will continue to incur the cost of maintaining any associated GCP resources until all processing and writing has completed.

Use the Drain option to stop your job if you want to prevent data loss as you bring down your pipeline.

Effects of draining a job

When you issue the Drain command, Cloud Dataflow immediately closes any in-process windows and fires all triggers. The system does not wait for any outstanding time-based windows to finish. For example, if your pipeline is ten minutes into a two-hour window when you issue the Drain command, Cloud Dataflow won't wait for the remainder of the window to finish. It will close the window immediately with partial results.

Cloud Dataflow causes open windows to close by advancing the system watermark to infinity. This functionality also works with custom data sources. When Draining a pipeline that uses a custom data source class, Cloud Dataflow will stop issuing requests for new data, advance the system watermark to infinity, and call your source's finalize() method on the last checkpoint.

In the detailed view of your pipeline's transforms, you can see the effects of an in-process Drain command:

Figure 3: A step view with Drain in progress; notice the watermark has advanced to the maximum value.

Note: It is possible to Update a pipeline that is currently being Drained.

As a Drain can take a significant amount of time to complete, such as when your pipeline has a large amount of buffered data, you may still Cancel a job that is currently Draining.