public sealed class WorkerPool : IMessage<WorkerPool>, IEquatable<WorkerPool>, IDeepCloneable<WorkerPool>, IBufferMessage, IMessageDescribes one particular pool of Cloud Dataflow workers to be instantiated by the Cloud Dataflow service in order to perform the computations required by a job. Note that a workflow job may use multiple pools, in order to match the various computational requirements of the various stages of the job.
Implements
IMessage<WorkerPool>, IEquatable<WorkerPool>, IDeepCloneable<WorkerPool>, IBufferMessage, IMessageNamespace
Google.Cloud.Dataflow.V1Beta3Assembly
Google.Cloud.Dataflow.V1Beta3.dll
Constructors
WorkerPool()
public WorkerPool()WorkerPool(WorkerPool)
public WorkerPool(WorkerPool other)| Parameter | |
|---|---|
| Name | Description |
other | WorkerPool |
Properties
AutoscalingSettings
public AutoscalingSettings AutoscalingSettings { get; set; }Settings for autoscaling of this WorkerPool.
| Property Value | |
|---|---|
| Type | Description |
AutoscalingSettings | |
DataDisks
public RepeatedField<Disk> DataDisks { get; }Data disks that are used by a VM in this workflow.
| Property Value | |
|---|---|
| Type | Description |
RepeatedField<Disk> | |
DefaultPackageSet
public DefaultPackageSet DefaultPackageSet { get; set; }The default package set to install. This allows the service to select a default set of packages which are useful to worker harnesses written in a particular language.
| Property Value | |
|---|---|
| Type | Description |
DefaultPackageSet | |
DiskSizeGb
public int DiskSizeGb { get; set; }Size of root disk for VMs, in GB. If zero or unspecified, the service will attempt to choose a reasonable default.
| Property Value | |
|---|---|
| Type | Description |
Int32 | |
DiskSourceImage
public string DiskSourceImage { get; set; }Fully qualified source image for disks.
| Property Value | |
|---|---|
| Type | Description |
String | |
DiskType
public string DiskType { get; set; }Type of root disk for VMs. If empty or unspecified, the service will attempt to choose a reasonable default.
| Property Value | |
|---|---|
| Type | Description |
String | |
IpConfiguration
public WorkerIPAddressConfiguration IpConfiguration { get; set; }Configuration for VM IPs.
| Property Value | |
|---|---|
| Type | Description |
WorkerIPAddressConfiguration | |
Kind
public string Kind { get; set; }The kind of the worker pool; currently only harness and shuffle
are supported.
| Property Value | |
|---|---|
| Type | Description |
String | |
MachineType
public string MachineType { get; set; }Machine type (e.g. "n1-standard-1"). If empty or unspecified, the service will attempt to choose a reasonable default.
| Property Value | |
|---|---|
| Type | Description |
String | |
Metadata
public MapField<string, string> Metadata { get; }Metadata to set on the Google Compute Engine VMs.
| Property Value | |
|---|---|
| Type | Description |
MapField<String, String> | |
Network
public string Network { get; set; }Network to which VMs will be assigned. If empty or unspecified, the service will use the network "default".
| Property Value | |
|---|---|
| Type | Description |
String | |
NumThreadsPerWorker
public int NumThreadsPerWorker { get; set; }The number of threads per worker harness. If empty or unspecified, the service will choose a number of threads (according to the number of cores on the selected machine type for batch, or 1 by convention for streaming).
| Property Value | |
|---|---|
| Type | Description |
Int32 | |
NumWorkers
public int NumWorkers { get; set; }Number of Google Compute Engine workers in this pool needed to execute the job. If zero or unspecified, the service will attempt to choose a reasonable default.
| Property Value | |
|---|---|
| Type | Description |
Int32 | |
OnHostMaintenance
public string OnHostMaintenance { get; set; }The action to take on host maintenance, as defined by the Google Compute Engine API.
| Property Value | |
|---|---|
| Type | Description |
String | |
Packages
public RepeatedField<Package> Packages { get; }Packages to be installed on workers.
| Property Value | |
|---|---|
| Type | Description |
RepeatedField<Package> | |
PoolArgs
public Any PoolArgs { get; set; }Extra arguments for this worker pool.
| Property Value | |
|---|---|
| Type | Description |
Any | |
SdkHarnessContainerImages
public RepeatedField<SdkHarnessContainerImage> SdkHarnessContainerImages { get; }Set of SDK harness containers needed to execute this pipeline. This will only be set in the Fn API path. For non-cross-language pipelines this should have only one entry. Cross-language pipelines will have two or more entries.
| Property Value | |
|---|---|
| Type | Description |
RepeatedField<SdkHarnessContainerImage> | |
Subnetwork
public string Subnetwork { get; set; }Subnetwork to which VMs will be assigned, if desired. Expected to be of the form "regions/REGION/subnetworks/SUBNETWORK".
| Property Value | |
|---|---|
| Type | Description |
String | |
TaskrunnerSettings
public TaskRunnerSettings TaskrunnerSettings { get; set; }Settings passed through to Google Compute Engine workers when using the standard Dataflow task runner. Users should ignore this field.
| Property Value | |
|---|---|
| Type | Description |
TaskRunnerSettings | |
TeardownPolicy
public TeardownPolicy TeardownPolicy { get; set; }Sets the policy for determining when to turndown worker pool.
Allowed values are: TEARDOWN_ALWAYS, TEARDOWN_ON_SUCCESS, and
TEARDOWN_NEVER.
TEARDOWN_ALWAYS means workers are always torn down regardless of whether
the job succeeds. TEARDOWN_ON_SUCCESS means workers are torn down
if the job succeeds. TEARDOWN_NEVER means the workers are never torn
down.
If the workers are not torn down by the service, they will
continue to run and use Google Compute Engine VM resources in the
user's project until they are explicitly terminated by the user.
Because of this, Google recommends using the TEARDOWN_ALWAYS
policy except for small, manually supervised test jobs.
If unknown or unspecified, the service will attempt to choose a reasonable default.
| Property Value | |
|---|---|
| Type | Description |
TeardownPolicy | |
WorkerHarnessContainerImage
public string WorkerHarnessContainerImage { get; set; }Required. Docker container image that executes the Cloud Dataflow worker harness, residing in Google Container Registry.
Deprecated for the Fn API path. Use sdk_harness_container_images instead.
| Property Value | |
|---|---|
| Type | Description |
String | |
Zone
public string Zone { get; set; }Zone to run the worker pools in. If empty or unspecified, the service will attempt to choose a reasonable default.
| Property Value | |
|---|---|
| Type | Description |
String | |