Service options are a type of pipeline option that allows you to specify
additional job modes and configurations for a Dataflow job. Set
these options by setting the dataflowServiceOptions
pipeline option. For more
information, see
Set Dataflow pipeline options.
Dataflow supports the following service options.
Option | Description |
---|---|
automatically_use_created_reservation |
Use Compute Engine reservations for the Dataflow workers. For more information, see Use Compute Engine reservations with Dataflow |
block_project_ssh_keys |
Prevents VMs from accepting SSH keys that are stored in project metadata. For more information, see Restrict SSH keys from VMs. |
enable_confidential_compute |
Enables Confidential VM on Dataflow worker VMs. For more information, see Confidential Computing concepts. This service option is not compatible with Dataflow Prime or worker accelerators. You must specify a supported machine type. When this option is enabled, the job incurs additional flat per-vCPU and per-GB costs. For more information, see Dataflow pricing. |
enable_dynamic_thread_scaling |
Enable dynamic thread scaling on Dataflow worker VMs. For more information, see Dynamic thread scaling. |
enable_google_cloud_heap_sampling |
Enable heap profiling. For more information, see Monitoring pipeline performance using Cloud Profile. |
enable_google_cloud_profiler |
Enable performance profiling. For more information, see Monitoring pipeline performance using Cloud Profile. |
enable_prime |
Enable Dataflow Prime for this job. For more information, see Use Dataflow Prime. |
max_workflow_runtime_walltime_seconds |
The maximum number of seconds the job can run. If the job exceeds this limit, Dataflow cancels the job. This service option is currently supported for batch jobs only. Specify the number of seconds as a parameter to the flag. For example:
|
worker_accelerator |
Enable GPUs for this job. Specify the type and number of GPUs to attach to Dataflow workers as parameters to the flag. For a list of GPU types that are supported with Dataflow, see Dataflow support for GPUs. For example: --dataflow_service_options "worker_accelerator=type:GPU_TYPE;count:GPU_COUNT;install-nvidia-driver"
If you're using
NVIDIA Multi-Process Service (MPS),
append the
For more information about using GPUs, see GPUs with Dataflow. |