Dataflow service options

Service options are a type of pipeline option that allows you to specify additional job modes and configurations for a Dataflow job. Set these options by setting the dataflowServiceOptions pipeline option. For more information, see Set Dataflow pipeline options.

Dataflow supports the following service options.

Option	Description
automatically_use_created_reservation	Use Compute Engine reservations for the Dataflow workers. For more information, see Use Compute Engine reservations with Dataflow
block_project_ssh_keys	Prevents VMs from accepting SSH keys that are stored in project metadata. For more information, see Restrict SSH keys from VMs.
enable_confidential_compute	Enables Confidential VM on Dataflow worker VMs. For more information, see Confidential Computing concepts. This service option is not compatible with Dataflow Prime or worker accelerators. You must specify a supported machine type. When this option is enabled, the job incurs additional flat per-vCPU and per-GB costs. For more information, see Dataflow pricing.
enable_dynamic_thread_scaling	Enable dynamic thread scaling on Dataflow worker VMs. For more information, see Dynamic thread scaling.
enable_google_cloud_heap_sampling	Enable heap profiling. For more information, see Monitoring pipeline performance using Cloud Profile.
enable_google_cloud_profiler	Enable performance profiling. For more information, see Monitoring pipeline performance using Cloud Profile.
enable_prime	Enable Dataflow Prime for this job. For more information, see Use Dataflow Prime.
max_workflow_runtime_walltime_seconds	The maximum number of seconds the job can run. If the job exceeds this limit, Dataflow cancels the job. This service option is currently supported for batch jobs only. Specify the number of seconds as a parameter to the flag. For example: --dataflowServiceOptions=max_workflow_runtime_walltime_seconds=300
worker_accelerator	Enable GPUs for this job. Specify the type and number of GPUs to attach to Dataflow workers as parameters to the flag. For a list of GPU types that are supported with Dataflow, see Dataflow support for GPUs. For example: --dataflow_service_options "worker_accelerator=type:GPU_TYPE;count:GPU_COUNT;install-nvidia-driver" If you're using NVIDIA Multi-Process Service (MPS), append the use_nvidia_mps parameter to the end of the list of parameters. For example: "worker_accelerator=type:GPU_TYPE;count:GPU_COUNT;install-nvidia-driver;use_nvidia_mps" For more information about using GPUs, see GPUs with Dataflow.