DataflowPipelineDebugOptions (Google Cloud Dataflow SDK 1.9.1 API)

Google Cloud Dataflow SDK for Java, version 1.9.1

Warning: Dataflow SDK 1.x for Java is unsupported as of October 16, 2018. After August 12, 2020, you can use unsupported SDKs temporarily while you upgrade your pipelines. For more information, see Using an unsupported SDK.

For information on migrating from the 1.x SDK, see Migrating from Dataflow SDK 1.x for Java.

Note: If you are using Cloud Dataflow SDK version 2.x, see the Apache Beam API reference documentation.

com.google.cloud.dataflow.sdk.options

Interface DataflowPipelineDebugOptions

• All Superinterfaces:

  HasDisplayData, PipelineOptions

  All Known Subinterfaces:

  BlockingDataflowPipelineOptions, DataflowPipelineOptions, DataflowWorkerHarnessOptions, TestDataflowPipelineOptions

  ---

  
  

  @Hidden
  public interface DataflowPipelineDebugOptions
  extends PipelineOptions

  Internal. Options used to control execution of the Dataflow SDK for debugging and testing purposes.

• • Nested Class Summary

    Nested Classes 

    Modifier and Type	Interface and Description
    static class	DataflowPipelineDebugOptions.DataflowClientFactory Returns the default Dataflow client built from the passed in PipelineOptions.
    static class	DataflowPipelineDebugOptions.PathValidatorFactory Creates a PathValidator object using the class specified in getPathValidatorClass().
    static class	DataflowPipelineDebugOptions.StagerFactory Creates a Stager object using the class specified in getStagerClass().

    • Nested classes/interfaces inherited from interface com.google.cloud.dataflow.sdk.options.PipelineOptions

      PipelineOptions.AtomicLongFactory, PipelineOptions.CheckEnabled

  • Method Summary

    All Methods Instance Methods Abstract Methods Deprecated Methods 

    Modifier and Type	Method and Description
    String	getApiRootUrl() The root URL for the Dataflow API.
    com.google.api.services.dataflow.Dataflow	getDataflowClient() An instance of the Dataflow client.
    String	getDataflowEndpoint() Dataflow endpoint to use.
    String	getDataflowJobFile() The path to write the translated Dataflow job specification out to at job submission time.
    boolean	getDumpHeapOnOOM() If true, save a heap dump before killing a thread or process which is GC thrashing or out of memory.
    List<String>	getExperiments() The list of backend experiments to enable.
    int	getNumberOfWorkerHarnessThreads() Number of threads to use on the Dataflow worker harness.
    String	getOverrideWindmillBinary() Custom windmill_main binary to use with the streaming runner.
    com.google.cloud.dataflow.sdk.util.PathValidator	getPathValidator() The path validator instance that should be used to validate paths.
    Class<? extends com.google.cloud.dataflow.sdk.util.PathValidator>	getPathValidatorClass() The class of the validator that should be created and used to validate paths.
    String	getPubsubRootUrl() Root URL for use with the Pubsub API.
    com.google.cloud.dataflow.sdk.util.Stager	getStager() The resource stager instance that should be used to stage resources.
    Class<? extends com.google.cloud.dataflow.sdk.util.Stager>	getStagerClass() The class responsible for staging resources to be accessible by workers during job execution.
    Map<String,String>	getTransformNameMapping() Mapping of old PTranform names to new ones, specified as JSON {"oldName":"newName",...}.
    boolean	getUpdate() Deprecated.  This property is replaced by DataflowPipelineOptions.getUpdate()
    String	getWindmillServiceEndpoint() Custom windmill service endpoint.
    int	getWindmillServicePort()
    void	setApiRootUrl(String value)
    void	setDataflowClient(com.google.api.services.dataflow.Dataflow value)
    void	setDataflowEndpoint(String value)
    void	setDataflowJobFile(String value)
    void	setDumpHeapOnOOM(boolean dumpHeapBeforeExit)
    void	setExperiments(List<String> value)
    void	setNumberOfWorkerHarnessThreads(int value)
    void	setOverrideWindmillBinary(String value)
    void	setPathValidator(com.google.cloud.dataflow.sdk.util.PathValidator validator)
    void	setPathValidatorClass(Class<? extends com.google.cloud.dataflow.sdk.util.PathValidator> validatorClass)
    void	setPubsubRootUrl(String value)
    void	setStager(com.google.cloud.dataflow.sdk.util.Stager stager)
    void	setStagerClass(Class<? extends com.google.cloud.dataflow.sdk.util.Stager> stagerClass)
    void	setTransformNameMapping(Map<String,String> value)
    void	setUpdate(boolean value) Deprecated.
    void	setWindmillServiceEndpoint(String value)
    void	setWindmillServicePort(int value)

    • Methods inherited from interface com.google.cloud.dataflow.sdk.options.PipelineOptions

      as, cloneAs, getOptionsId, getRunner, getStableUniqueNames, getTempLocation, outputRuntimeOptions, setOptionsId, setRunner, setStableUniqueNames, setTempLocation

    • Methods inherited from interface com.google.cloud.dataflow.sdk.transforms.display.HasDisplayData

      populateDisplayData

• • Method Detail

    • getExperiments

      @Experimental
      List<String> getExperiments()

      The list of backend experiments to enable.

      Dataflow provides a number of experimental features that can be enabled with this flag.

      Please sync with the Dataflow team before enabling any experiments.

    • setExperiments

      void setExperiments(List<String> value)

    • getApiRootUrl

      @Default.String(value="https://dataflow.googleapis.com/")
      String getApiRootUrl()

      The root URL for the Dataflow API. dataflowEndpoint can override this value if it contains an absolute URL, otherwise apiRootUrl will be combined with dataflowEndpoint to generate the full URL to communicate with the Dataflow API.

    • setApiRootUrl

      void setApiRootUrl(String value)

    • getDataflowEndpoint

      @Default.String(value="")
      String getDataflowEndpoint()

      Dataflow endpoint to use.

      Defaults to the current version of the Google Cloud Dataflow API, at the time the current SDK version was released.

      If the string contains "://", then this is treated as a URL, otherwise getApiRootUrl() is used as the root URL.

    • setDataflowEndpoint

      void setDataflowEndpoint(String value)

    • getDataflowJobFile

      String getDataflowJobFile()

      The path to write the translated Dataflow job specification out to at job submission time. The Dataflow job specification will be represented in JSON format.

    • setDataflowJobFile

      void setDataflowJobFile(String value)

    • getPathValidatorClass

      @Default.Class(value=com.google.cloud.dataflow.sdk.util.DataflowPathValidator.class)
      Class<? extends com.google.cloud.dataflow.sdk.util.PathValidator> getPathValidatorClass()

      The class of the validator that should be created and used to validate paths. If pathValidator has not been set explicitly, an instance of this class will be constructed and used as the path validator.

    • setPathValidatorClass

      void setPathValidatorClass(Class<? extends com.google.cloud.dataflow.sdk.util.PathValidator> validatorClass)

    • getPathValidator

      @Default.InstanceFactory(value=DataflowPipelineDebugOptions.PathValidatorFactory.class)
      com.google.cloud.dataflow.sdk.util.PathValidator getPathValidator()

      The path validator instance that should be used to validate paths. If no path validator has been set explicitly, the default is to use the instance factory that constructs a path validator based upon the currently set pathValidatorClass.

    • setPathValidator

      void setPathValidator(com.google.cloud.dataflow.sdk.util.PathValidator validator)

    • getStagerClass

      @Default.Class(value=com.google.cloud.dataflow.sdk.util.GcsStager.class)
      Class<? extends com.google.cloud.dataflow.sdk.util.Stager> getStagerClass()

      The class responsible for staging resources to be accessible by workers during job execution. If stager has not been set explicitly, an instance of this class will be created and used as the resource stager.

    • setStagerClass

      void setStagerClass(Class<? extends com.google.cloud.dataflow.sdk.util.Stager> stagerClass)

    • getStager

      @Default.InstanceFactory(value=DataflowPipelineDebugOptions.StagerFactory.class)
      com.google.cloud.dataflow.sdk.util.Stager getStager()

      The resource stager instance that should be used to stage resources. If no stager has been set explicitly, the default is to use the instance factory that constructs a resource stager based upon the currently set stagerClass.

    • setStager

      void setStager(com.google.cloud.dataflow.sdk.util.Stager stager)

    • getDataflowClient

      @Default.InstanceFactory(value=DataflowPipelineDebugOptions.DataflowClientFactory.class)
      com.google.api.services.dataflow.Dataflow getDataflowClient()

      An instance of the Dataflow client. Defaults to creating a Dataflow client using the current set of options.

    • setDataflowClient

      void setDataflowClient(com.google.api.services.dataflow.Dataflow value)

    • getPubsubRootUrl

      @Default.String(value="https://pubsub.googleapis.com")
      String getPubsubRootUrl()

      Root URL for use with the Pubsub API.

    • setPubsubRootUrl

      void setPubsubRootUrl(String value)

    • getUpdate

      @Deprecated
      boolean getUpdate()

      Deprecated. This property is replaced by DataflowPipelineOptions.getUpdate()
      Whether to update the currently running pipeline with the same name as this one.

    • setUpdate

      @Deprecated
      void setUpdate(boolean value)

      Deprecated. 

    • getTransformNameMapping

      Map<String,String> getTransformNameMapping()

      Mapping of old PTranform names to new ones, specified as JSON {"oldName":"newName",...}. To mark a transform as deleted, make newName the empty string.

    • setTransformNameMapping

      void setTransformNameMapping(Map<String,String> value)

    • getOverrideWindmillBinary

      String getOverrideWindmillBinary()

      Custom windmill_main binary to use with the streaming runner.

    • setOverrideWindmillBinary

      void setOverrideWindmillBinary(String value)

    • getWindmillServiceEndpoint

      String getWindmillServiceEndpoint()

      Custom windmill service endpoint.

    • setWindmillServiceEndpoint

      void setWindmillServiceEndpoint(String value)

    • getWindmillServicePort

      @Default.Integer(value=443)
      int getWindmillServicePort()

    • setWindmillServicePort

      void setWindmillServicePort(int value)

    • getNumberOfWorkerHarnessThreads

      int getNumberOfWorkerHarnessThreads()

      Number of threads to use on the Dataflow worker harness. If left unspecified, the Dataflow service will compute an appropriate number of threads to use.

    • setNumberOfWorkerHarnessThreads

      void setNumberOfWorkerHarnessThreads(int value)

    • getDumpHeapOnOOM

      boolean getDumpHeapOnOOM()

      If true, save a heap dump before killing a thread or process which is GC thrashing or out of memory. The location of the heap file will either be echoed back to the user, or the user will be given the opportunity to download the heap file.

      CAUTION: Heap dumps can of comparable size to the default boot disk. Consider increasing the boot disk size before setting this flag to true.

    • setDumpHeapOnOOM

      void setDumpHeapOnOOM(boolean dumpHeapBeforeExit)