Dataflow service options

Stay organized with collections Save and categorize content based on your preferences.

Service options are a type of pipeline option that allows you to specify additional job modes and configurations for a Dataflow job. Set these options by setting the dataflowServiceOptions pipeline option. For more information, see Set Dataflow pipeline options.

Dataflow supports the following service options.

Option Description
automatically_use_created_reservation Use Compute Engine reservations for the Dataflow workers. For more information, see Use Compute Engine reservations with Dataflow
block-project-ssh-keys Prevents VMs from accepting SSH keys that are stored in project metadata. For more information, see Restrict SSH keys from VMs.
enable_google_cloud_heap_sampling Enable heap profiling. For more information, see Monitoring pipeline performance using Cloud Profile.
enable_google_cloud_profiler Enable performance profiling. For more information, see Monitoring pipeline performance using Cloud Profile.
enable_prime Enable Dataflow Prime for this job. For more information, see Use Dataflow Prime.
max_workflow_runtime_walltime_seconds

The maximum number of seconds the job can run. If the job exceeds this limit, Dataflow cancels the job. This service option is currently supported for batch jobs only.

Specify the number of seconds as a parameter to the flag. Example:

--dataflowServiceOptions=max_workflow_runtime_walltime_seconds=300