Dataflow service options

Service options are a type of pipeline option that let you specify additional job modes and configurations for a Dataflow job. Set these options by setting the Dataflow service options pipeline option.

Java

--dataflowServiceOptions=SERVICE_OPTION

Replace SERVICE_OPTION with the service option that you want to use.

Python

--dataflow_service_options=SERVICE_OPTION

Replace SERVICE_OPTION with the service option that you want to use.

Go

--dataflow_service_options=SERVICE_OPTION

Replace SERVICE_OPTION with the service option that you want to use.

gcloud

Use the gcloud dataflow jobs run command with the additional-experiments option. If you're using Flex Templates, use the gcloud dataflow flex-template run command.

--additional-experiments=SERVICE_OPTION

For example:

gcloud dataflow jobs run JOB_NAME --additional-experiments=SERVICE_OPTION

Replace the following values:

  • JOB_NAME: the name of your Dataflow job
  • SERVICE_OPTION: the service option that you want to use

REST

Use the additionalExperiments field in the RuntimeEnvironment object. If you're using Flex Templates, use the additionalExperiments field in the FlexTemplateRuntimeEnvironment object.

{
  additionalExperiments : ["SERVICE_OPTION"]
  ...
}

Replace SERVICE_OPTION with the service option that you want to use.

For more information, see Set Dataflow pipeline options.

Dataflow supports the following service options.

Option Description
automatically_use_created_reservation Use Compute Engine reservations for the Dataflow workers. For more information, see Use Compute Engine reservations with Dataflow
block_project_ssh_keys Prevents VMs from accepting SSH keys that are stored in project metadata. For more information, see Restrict SSH keys from VMs.
enable_confidential_compute

Enables Confidential VM with AMD Secure Encryption Virtualization (SEV) on Dataflow worker VMs. For more information, see Confidential Computing concepts. This service option is not compatible with Dataflow Prime or worker accelerators. You must specify a supported machine type. When this option is enabled, the job incurs additional flat per-vCPU and per-GB costs. For more information, see Dataflow pricing.

enable_dynamic_thread_scaling

Enable dynamic thread scaling on Dataflow worker VMs. For more information, see Dynamic thread scaling.

enable_google_cloud_heap_sampling Enable heap profiling. For more information, see Monitoring pipeline performance using Cloud Profile.
enable_google_cloud_profiler Enable performance profiling. For more information, see Monitoring pipeline performance using Cloud Profile.
enable_preflight_validation When you run your pipeline on Dataflow, before the job launches, Dataflow performs validation checks on the pipeline. This option is enabled by default. To disable pipeline validation, set this option to false. For more information, see Pipeline validation.
enable_prime Enable Dataflow Prime for this job. For more information, see Use Dataflow Prime.
enable_streaming_engine_resource_based_billing Enable resource-based billing for this job. For more information, see Pricing in "Use Streaming Engine for streaming jobs."
graph_validate_only Runs a job graph validation check to verify whether a replacement job is valid. For more information, see Validate a replacement job.
max_workflow_runtime_walltime_seconds

The maximum number of seconds the job can run. If the job exceeds this limit, Dataflow cancels the job. This service option is supported for batch jobs only. Batch jobs can't run for more than 10 days. After 10 days, the job is cancelled.

Specify the number of seconds as a parameter to the flag. For example:

--dataflowServiceOptions=max_workflow_runtime_walltime_seconds=300

min_num_workers The minimum number of workers the job uses. Guarantees that a Dataflow job always has at least the number of workers specified. Horizontal autoscaling doesn't scale below the number of workers specified.
sdf_checkpoint_after_duration

The maximum duration each worker buffers splittable DoFn (SDF) outputs before checkpointing for further processing. Set this duration when you want low-latency processing on pipelines that have low throughput per worker, such as when reading change streams from Spanner. The worker checkpoints when either the duration limit or the bytes limit is triggered, so you can use this service option with sdf_checkpoint_after_output_bytes or by itself.

This service option is supported for Streaming Engine jobs that use Runner v2.

Specify the duration as a parameter. For example, to change the default from 5 seconds to 500 milliseconds, use the following syntax:

--dataflowServiceOptions=sdf_checkpoint_after_duration=500ms

sdf_checkpoint_after_output_bytes

The maximum splittable DoFn (SDF) output bytes each worker produces and buffers before checkpointing for further processing. Set this value when you want low-latency processing on pipelines that have low throughput per worker, such as when reading change streams from Spanner. The worker checkpoints when either the duration limit or the bytes limit is triggered, so you can use this service option with sdf_checkpoint_after_duration or by itself.

This service option is supported for Streaming Engine jobs that use Runner v2.

Specify the number of bytes as a parameter. For example, to change the default from 5 MiB to 512 KiB, use the following syntax:

--dataflowServiceOptions=sdf_checkpoint_after_output_bytes=524288

streaming_mode_at_least_once Enables at-least-once streaming mode. For more information, see Set the pipeline streaming mode.
streaming_enable_pubsub_direct_output Enables at-least-once streaming mode for Pub/Sub output. This option does not affect the streaming mode for the rest of the pipeline. For more information, see Set the pipeline streaming mode.
worker_accelerator

Enable GPUs for this job. If you use right fitting, don't use this service option.

Specify the type and number of GPUs to attach to Dataflow workers as parameters to the flag. For a list of GPU types that are supported with Dataflow, see Dataflow support for GPUs. For example:

--dataflow_service_options "worker_accelerator=type:GPU_TYPE;count:GPU_COUNT;install-nvidia-driver"

If you're using NVIDIA Multi-Process Service (MPS), append the use_nvidia_mps parameter to the end of the list of parameters. For example:

"worker_accelerator=type:GPU_TYPE;count:GPU_COUNT;install-nvidia-driver;use_nvidia_mps"

For more information about using GPUs, see GPUs with Dataflow.

worker_utilization_hint Specifies the target CPU utilization. For more information, see Set the worker utilization hint.