DataflowPipelineWorkerPoolOptions (Google Cloud Dataflow SDK 1.9.1 API)

Google Cloud Dataflow SDK for Java, version 1.9.1

Warning: Dataflow SDK 1.x for Java is unsupported as of October 16, 2018. After August 12, 2020, you can use unsupported SDKs temporarily while you upgrade your pipelines. For more information, see Using an unsupported SDK.

For information on migrating from the 1.x SDK, see Migrating from Dataflow SDK 1.x for Java.

Note: If you are using Cloud Dataflow SDK version 2.x, see the Apache Beam API reference documentation.

com.google.cloud.dataflow.sdk.options

Interface DataflowPipelineWorkerPoolOptions

• All Superinterfaces:

  HasDisplayData, PipelineOptions

  All Known Subinterfaces:

  BlockingDataflowPipelineOptions, DataflowPipelineOptions, DataflowWorkerHarnessOptions, TestDataflowPipelineOptions

  ---

  
  

  public interface DataflowPipelineWorkerPoolOptions
  extends PipelineOptions

  Options that are used to configure the Dataflow pipeline worker pool.

• • Nested Class Summary

    Nested Classes 

    Modifier and Type	Interface and Description
    static class	DataflowPipelineWorkerPoolOptions.AutoscalingAlgorithmType Type of autoscaling algorithm to use.
    static class	DataflowPipelineWorkerPoolOptions.TeardownPolicy Deprecated.  Dataflow Service will only support TEARDOWN_ALWAYS policy in the future.
    static class	DataflowPipelineWorkerPoolOptions.WorkerHarnessContainerImageFactory Returns the default Docker container image that executes Dataflow worker harness, residing in Google Container Registry.

    • Nested classes/interfaces inherited from interface com.google.cloud.dataflow.sdk.options.PipelineOptions

      PipelineOptions.AtomicLongFactory, PipelineOptions.CheckEnabled

  • Method Summary

    All Methods Instance Methods Abstract Methods 

    Modifier and Type	Method and Description
    DataflowPipelineWorkerPoolOptions.AutoscalingAlgorithmType	getAutoscalingAlgorithm() The autoscaling algorithm to use for the workerpool.
    int	getDiskSizeGb() Remote worker disk size, in gigabytes, or 0 to use the default size.
    List<String>	getFilesToStage() List of local files to make available to workers.
    int	getMaxNumWorkers() The maximum number of workers to use for the workerpool.
    String	getNetwork() Compute Engine network for launching workers.
    int	getNumWorkers() Number of workers to use when executing the Dataflow job.
    String	getSubnetwork() Compute Engine subnetwork for launching workers.
    DataflowPipelineWorkerPoolOptions.TeardownPolicy	getTeardownPolicy() The teardown policy for the VMs.
    Boolean	getUsePublicIps() Specifies whether worker pools should be started with public IP addresses.
    String	getWorkerDiskType() Specifies what type of persistent disk should be used.
    String	getWorkerHarnessContainerImage() Docker container image that executes Dataflow worker harness, residing in Google Container Registry.
    String	getWorkerMachineType() Machine type to create Dataflow worker VMs as.
    String	getZone() Compute Engine availability zone for launching workers.
    void	setAutoscalingAlgorithm(DataflowPipelineWorkerPoolOptions.AutoscalingAlgorithmType value)
    void	setDiskSizeGb(int value)
    void	setFilesToStage(List<String> value)
    void	setMaxNumWorkers(int value)
    void	setNetwork(String value)
    void	setNumWorkers(int value)
    void	setSubnetwork(String value)
    void	setTeardownPolicy(DataflowPipelineWorkerPoolOptions.TeardownPolicy value)
    void	setUsePublicIps(Boolean value)
    void	setWorkerDiskType(String value)
    void	setWorkerHarnessContainerImage(String value)
    void	setWorkerMachineType(String value)
    void	setZone(String value)

    • Methods inherited from interface com.google.cloud.dataflow.sdk.options.PipelineOptions

      as, cloneAs, getOptionsId, getRunner, getStableUniqueNames, getTempLocation, outputRuntimeOptions, setOptionsId, setRunner, setStableUniqueNames, setTempLocation

    • Methods inherited from interface com.google.cloud.dataflow.sdk.transforms.display.HasDisplayData

      populateDisplayData

• • Method Detail

    • getNumWorkers

      int getNumWorkers()

      Number of workers to use when executing the Dataflow job. Note that selection of an autoscaling algorithm other then NONE will affect the size of the worker pool. If left unspecified, the Dataflow service will determine the number of workers.

    • setNumWorkers

      void setNumWorkers(int value)

    • getAutoscalingAlgorithm

      DataflowPipelineWorkerPoolOptions.AutoscalingAlgorithmType getAutoscalingAlgorithm()

      The autoscaling algorithm to use for the workerpool.

      • NONE: does not change the size of the worker pool.
      • BASIC: autoscale the worker pool size up to maxNumWorkers until the job completes.
      • THROUGHPUT_BASED: autoscale the workerpool based on throughput (up to maxNumWorkers).

    • setAutoscalingAlgorithm

      void setAutoscalingAlgorithm(DataflowPipelineWorkerPoolOptions.AutoscalingAlgorithmType value)

    • getMaxNumWorkers

      int getMaxNumWorkers()

      The maximum number of workers to use for the workerpool. This options limits the size of the workerpool for the lifetime of the job, including pipeline updates. If left unspecified, the Dataflow service will compute a ceiling.

    • setMaxNumWorkers

      void setMaxNumWorkers(int value)

    • getDiskSizeGb

      int getDiskSizeGb()

      Remote worker disk size, in gigabytes, or 0 to use the default size.

    • setDiskSizeGb

      void setDiskSizeGb(int value)

    • getWorkerHarnessContainerImage

      @Default.InstanceFactory(value=DataflowPipelineWorkerPoolOptions.WorkerHarnessContainerImageFactory.class)
       @Hidden
      String getWorkerHarnessContainerImage()

      Docker container image that executes Dataflow worker harness, residing in Google Container Registry.

    • setWorkerHarnessContainerImage

      void setWorkerHarnessContainerImage(String value)

    • getNetwork

      String getNetwork()

      Compute Engine network for launching workers.

      Default is up to the Dataflow service.

    • setNetwork

      void setNetwork(String value)

    • getSubnetwork

      String getSubnetwork()

      Compute Engine subnetwork for launching workers.

      Default is up to the Dataflow service. Expected format is regions/REGION/subnetworks/SUBNETWORK.

      You may also need to specify network option.

    • setSubnetwork

      void setSubnetwork(String value)

    • getZone

      String getZone()

      Compute Engine availability zone for launching workers.

      Default is up to the Dataflow service.

    • setZone

      void setZone(String value)

    • getWorkerMachineType

      String getWorkerMachineType()

      Machine type to create Dataflow worker VMs as.

      See Compute Engine machine types for a list of valid options.

      If unset, the Dataflow service will choose a reasonable default.

    • setWorkerMachineType

      void setWorkerMachineType(String value)

    • getTeardownPolicy

      DataflowPipelineWorkerPoolOptions.TeardownPolicy getTeardownPolicy()

      The teardown policy for the VMs.

      If unset, the Dataflow service will choose a reasonable default.

    • setTeardownPolicy

      void setTeardownPolicy(DataflowPipelineWorkerPoolOptions.TeardownPolicy value)

    • getFilesToStage

      List<String> getFilesToStage()

      List of local files to make available to workers.

      Files are placed on the worker's classpath.

      The default value is the list of jars from the main program's classpath.

    • setFilesToStage

      void setFilesToStage(List<String> value)

    • getWorkerDiskType

      String getWorkerDiskType()

      Specifies what type of persistent disk should be used. The value should be a full or partial URL of a disk type resource, e.g., zones/us-central1-f/disks/pd-standard. For more information, see the API reference documentation for DiskTypes.

    • setWorkerDiskType

      void setWorkerDiskType(String value)

    • getUsePublicIps

      @Experimental
       @Nullable
      Boolean getUsePublicIps()

      Specifies whether worker pools should be started with public IP addresses.

      WARNING: This feature is experimental. You must be whitelisted to use it.

    • setUsePublicIps

      void setUsePublicIps(@Nullable
                           Boolean value)