Google Cloud Dataflow SDK for Java, version 1.9.1
Interface DataflowPipelineWorkerPoolOptions
-
- All Superinterfaces:
- HasDisplayData, PipelineOptions
- All Known Subinterfaces:
- BlockingDataflowPipelineOptions, DataflowPipelineOptions, DataflowWorkerHarnessOptions, TestDataflowPipelineOptions
public interface DataflowPipelineWorkerPoolOptions extends PipelineOptions
Options that are used to configure the Dataflow pipeline worker pool.
-
-
Nested Class Summary
Nested Classes Modifier and Type Interface and Description static class
DataflowPipelineWorkerPoolOptions.AutoscalingAlgorithmType
Type of autoscaling algorithm to use.static class
DataflowPipelineWorkerPoolOptions.TeardownPolicy
Deprecated.Dataflow Service will only support TEARDOWN_ALWAYS policy in the future.static class
DataflowPipelineWorkerPoolOptions.WorkerHarnessContainerImageFactory
Returns the default Docker container image that executes Dataflow worker harness, residing in Google Container Registry.-
Nested classes/interfaces inherited from interface com.google.cloud.dataflow.sdk.options.PipelineOptions
PipelineOptions.AtomicLongFactory, PipelineOptions.CheckEnabled
-
-
Method Summary
All Methods Instance Methods Abstract Methods Modifier and Type Method and Description DataflowPipelineWorkerPoolOptions.AutoscalingAlgorithmType
getAutoscalingAlgorithm()
The autoscaling algorithm to use for the workerpool.int
getDiskSizeGb()
Remote worker disk size, in gigabytes, or 0 to use the default size.List<String>
getFilesToStage()
List of local files to make available to workers.int
getMaxNumWorkers()
The maximum number of workers to use for the workerpool.String
getNetwork()
Compute Engine network for launching workers.int
getNumWorkers()
Number of workers to use when executing the Dataflow job.String
getSubnetwork()
Compute Engine subnetwork for launching workers.DataflowPipelineWorkerPoolOptions.TeardownPolicy
getTeardownPolicy()
The teardown policy for the VMs.Boolean
getUsePublicIps()
Specifies whether worker pools should be started with public IP addresses.String
getWorkerDiskType()
Specifies what type of persistent disk should be used.String
getWorkerHarnessContainerImage()
Docker container image that executes Dataflow worker harness, residing in Google Container Registry.String
getWorkerMachineType()
Machine type to create Dataflow worker VMs as.String
getZone()
Compute Engine availability zone for launching workers.void
setAutoscalingAlgorithm(DataflowPipelineWorkerPoolOptions.AutoscalingAlgorithmType value)
void
setDiskSizeGb(int value)
void
setFilesToStage(List<String> value)
void
setMaxNumWorkers(int value)
void
setNetwork(String value)
void
setNumWorkers(int value)
void
setSubnetwork(String value)
void
setTeardownPolicy(DataflowPipelineWorkerPoolOptions.TeardownPolicy value)
void
setUsePublicIps(Boolean value)
void
setWorkerDiskType(String value)
void
setWorkerHarnessContainerImage(String value)
void
setWorkerMachineType(String value)
void
setZone(String value)
-
Methods inherited from interface com.google.cloud.dataflow.sdk.options.PipelineOptions
as, cloneAs, getOptionsId, getRunner, getStableUniqueNames, getTempLocation, outputRuntimeOptions, setOptionsId, setRunner, setStableUniqueNames, setTempLocation
-
Methods inherited from interface com.google.cloud.dataflow.sdk.transforms.display.HasDisplayData
populateDisplayData
-
-
-
-
Method Detail
-
getNumWorkers
int getNumWorkers()
Number of workers to use when executing the Dataflow job. Note that selection of an autoscaling algorithm other thenNONE
will affect the size of the worker pool. If left unspecified, the Dataflow service will determine the number of workers.
-
setNumWorkers
void setNumWorkers(int value)
-
getAutoscalingAlgorithm
DataflowPipelineWorkerPoolOptions.AutoscalingAlgorithmType getAutoscalingAlgorithm()
The autoscaling algorithm to use for the workerpool.- NONE: does not change the size of the worker pool.
- BASIC: autoscale the worker pool size up to maxNumWorkers until the job completes.
- THROUGHPUT_BASED: autoscale the workerpool based on throughput (up to maxNumWorkers).
-
setAutoscalingAlgorithm
void setAutoscalingAlgorithm(DataflowPipelineWorkerPoolOptions.AutoscalingAlgorithmType value)
-
getMaxNumWorkers
int getMaxNumWorkers()
The maximum number of workers to use for the workerpool. This options limits the size of the workerpool for the lifetime of the job, including pipeline updates. If left unspecified, the Dataflow service will compute a ceiling.
-
setMaxNumWorkers
void setMaxNumWorkers(int value)
-
getDiskSizeGb
int getDiskSizeGb()
Remote worker disk size, in gigabytes, or 0 to use the default size.
-
setDiskSizeGb
void setDiskSizeGb(int value)
-
getWorkerHarnessContainerImage
@Default.InstanceFactory(value=DataflowPipelineWorkerPoolOptions.WorkerHarnessContainerImageFactory.class) @Hidden String getWorkerHarnessContainerImage()
Docker container image that executes Dataflow worker harness, residing in Google Container Registry.
-
setWorkerHarnessContainerImage
void setWorkerHarnessContainerImage(String value)
-
getNetwork
String getNetwork()
Compute Engine network for launching workers.Default is up to the Dataflow service.
-
setNetwork
void setNetwork(String value)
-
getSubnetwork
String getSubnetwork()
Compute Engine subnetwork for launching workers.Default is up to the Dataflow service. Expected format is regions/REGION/subnetworks/SUBNETWORK.
You may also need to specify network option.
-
setSubnetwork
void setSubnetwork(String value)
-
getZone
String getZone()
Compute Engine availability zone for launching workers.Default is up to the Dataflow service.
-
setZone
void setZone(String value)
-
getWorkerMachineType
String getWorkerMachineType()
Machine type to create Dataflow worker VMs as.See Compute Engine machine types for a list of valid options.
If unset, the Dataflow service will choose a reasonable default.
-
setWorkerMachineType
void setWorkerMachineType(String value)
-
getTeardownPolicy
DataflowPipelineWorkerPoolOptions.TeardownPolicy getTeardownPolicy()
The teardown policy for the VMs.If unset, the Dataflow service will choose a reasonable default.
-
setTeardownPolicy
void setTeardownPolicy(DataflowPipelineWorkerPoolOptions.TeardownPolicy value)
-
getFilesToStage
List<String> getFilesToStage()
List of local files to make available to workers.Files are placed on the worker's classpath.
The default value is the list of jars from the main program's classpath.
-
getWorkerDiskType
String getWorkerDiskType()
Specifies what type of persistent disk should be used. The value should be a full or partial URL of a disk type resource, e.g., zones/us-central1-f/disks/pd-standard. For more information, see the API reference documentation for DiskTypes.
-
setWorkerDiskType
void setWorkerDiskType(String value)
-
getUsePublicIps
@Experimental @Nullable Boolean getUsePublicIps()
Specifies whether worker pools should be started with public IP addresses.WARNING: This feature is experimental. You must be whitelisted to use it.
-
setUsePublicIps
void setUsePublicIps(@Nullable Boolean value)
-
-