Dataflow service options

Service options are a type of pipeline option that allows you to specify additional job modes and configurations for a Dataflow job. Set these options by setting the dataflowServiceOptions pipeline option. For more information, see Set Dataflow pipeline options.

Dataflow supports the following service options.

Option Description
automatically_use_created_reservation Use Compute Engine reservations for the Dataflow workers. For more information, see Use Compute Engine reservations with Dataflow
block_project_ssh_keys Prevents VMs from accepting SSH keys that are stored in project metadata. For more information, see Restrict SSH keys from VMs.
enable_confidential_compute

Enables Confidential VM on Dataflow worker VMs. For more information, see Confidential Computing concepts. This service option is not compatible with Dataflow Prime or worker accelerators. You must specify a supported machine type. When this option is enabled, the job incurs additional flat per-vCPU and per-GB costs. For more information, see Dataflow pricing.

enable_dynamic_thread_scaling

Enable dynamic thread scaling on Dataflow worker VMs. For more information, see Dynamic thread scaling.

enable_google_cloud_heap_sampling Enable heap profiling. For more information, see Monitoring pipeline performance using Cloud Profile.
enable_google_cloud_profiler Enable performance profiling. For more information, see Monitoring pipeline performance using Cloud Profile.
enable_prime Enable Dataflow Prime for this job. For more information, see Use Dataflow Prime.
max_workflow_runtime_walltime_seconds

The maximum number of seconds the job can run. If the job exceeds this limit, Dataflow cancels the job. This service option is currently supported for batch jobs only.

Specify the number of seconds as a parameter to the flag. For example:

--dataflowServiceOptions=max_workflow_runtime_walltime_seconds=300

worker_accelerator

Enable GPUs for this job.

Specify the type and number of GPUs to attach to Dataflow workers as parameters to the flag. For a list of GPU types that are supported with Dataflow, see Dataflow support for GPUs. For example:

--dataflow_service_options "worker_accelerator=type:GPU_TYPE;count:GPU_COUNT;install-nvidia-driver"

If you're using NVIDIA Multi-Process Service (MPS), append the use_nvidia_mps parameter to the end of the list of parameters. For example:

"worker_accelerator=type:GPU_TYPE;count:GPU_COUNT;install-nvidia-driver;use_nvidia_mps"

For more information about using GPUs, see GPUs with Dataflow.