Class FlexTemplateRuntimeEnvironment (0.4.1)

FlexTemplateRuntimeEnvironment(
    mapping=None, *, ignore_unknown_fields=False, **kwargs
)

The environment values to be set at runtime for flex template.

Attributes

NameDescription
num_workers int
The initial number of Google Compute Engine instances for the job.
max_workers int
The maximum number of Google Compute Engine instances to be made available to your pipeline during execution, from 1 to 1000.
zone str
The Compute Engine `availability zone
service_account_email str
The email address of the service account to run the job as.
temp_location str
The Cloud Storage path to use for temporary files. Must be a valid Cloud Storage URL, beginning with gs://.
machine_type str
The machine type to use for the job. Defaults to the value from the template if not specified.
additional_experiments Sequence[str]
Additional experiment flags for the job.
network str
Network to which VMs will be assigned. If empty or unspecified, the service will use the network "default".
subnetwork str
Subnetwork to which VMs will be assigned, if desired. You can specify a subnetwork using either a complete URL or an abbreviated path. Expected to be of the form "https://www.googleapis.com/compute/v1/projects/HOST_PROJECT_ID/regions/REGION/subnetworks/SUBNETWORK" or "regions/REGION/subnetworks/SUBNETWORK". If the subnetwork is located in a Shared VPC network, you must use the complete URL.
additional_user_labels Sequence[google.cloud.dataflow_v1beta3.types.FlexTemplateRuntimeEnvironment.AdditionalUserLabelsEntry]
Additional user labels to be specified for the job. Keys and values must follow the restrictions specified in the `labeling restrictions
kms_key_name str
Name for the Cloud KMS key for the job. Key format is: projects/
ip_configuration google.cloud.dataflow_v1beta3.types.WorkerIPAddressConfiguration
Configuration for VM IPs.
worker_region str
The Compute Engine region (https://cloud.google.com/compute/docs/regions-zones/regions-zones) in which worker processing should occur, e.g. "us-west1". Mutually exclusive with worker_zone. If neither worker_region nor worker_zone is specified, default to the control plane's region.
worker_zone str
The Compute Engine zone (https://cloud.google.com/compute/docs/regions-zones/regions-zones) in which worker processing should occur, e.g. "us-west1-a". Mutually exclusive with worker_region. If neither worker_region nor worker_zone is specified, a zone in the control plane's region is chosen based on available capacity. If both worker_zone and zone are set, worker_zone takes precedence.
enable_streaming_engine bool
Whether to enable Streaming Engine for the job.
flexrs_goal google.cloud.dataflow_v1beta3.types.FlexResourceSchedulingGoal
Set FlexRS goal for the job. https://cloud.google.com/dataflow/docs/guides/flexrs
staging_location str
The Cloud Storage path for staging local files. Must be a valid Cloud Storage URL, beginning with gs://.
sdk_container_image str
Docker registry location of container image to use for the 'worker harness. Default is the container for the version of the SDK. Note this field is only valid for portable pipelines.
disk_size_gb int
Worker disk size, in gigabytes.
autoscaling_algorithm google.cloud.dataflow_v1beta3.types.AutoscalingAlgorithm
The algorithm to use for autoscaling
dump_heap_on_oom bool
If true, save a heap dump before killing a thread or process which is GC thrashing or out of memory. The location of the heap file will either be echoed back to the user, or the user will be given the opportunity to download the heap file.
save_heap_dumps_to_gcs_path str
Cloud Storage bucket (directory) to upload heap dumps to the given location. Enabling this implies that heap dumps should be generated on OOM (dump_heap_on_oom is set to true).
launcher_machine_type str
The machine type to use for launching the job. The default is n1-standard-1.

Classes

AdditionalUserLabelsEntry

AdditionalUserLabelsEntry(mapping=None, *, ignore_unknown_fields=False, **kwargs)

The abstract base class for a message.

Parameters
NameDescription
kwargs dict

Keys and values corresponding to the fields of the message.

mapping Union[dict, .Message]

A dictionary or message to be used to determine the values for this message.

ignore_unknown_fields Optional(bool)

If True, do not raise errors for unknown fields. Only applied if mapping is a mapping type or there are keyword parameters.