Google Cloud Dataflow SDK for Java, version 1.9.1
Interface DataflowPipelineDebugOptions
-
- All Superinterfaces:
- HasDisplayData, PipelineOptions
- All Known Subinterfaces:
- BlockingDataflowPipelineOptions, DataflowPipelineOptions, DataflowWorkerHarnessOptions, TestDataflowPipelineOptions
@Hidden public interface DataflowPipelineDebugOptions extends PipelineOptions
Internal. Options used to control execution of the Dataflow SDK for debugging and testing purposes.
-
-
Nested Class Summary
Nested Classes Modifier and Type Interface and Description static class
DataflowPipelineDebugOptions.DataflowClientFactory
Returns the default Dataflow client built from the passed in PipelineOptions.static class
DataflowPipelineDebugOptions.PathValidatorFactory
Creates aPathValidator
object using the class specified ingetPathValidatorClass()
.static class
DataflowPipelineDebugOptions.StagerFactory
Creates aStager
object using the class specified ingetStagerClass()
.-
Nested classes/interfaces inherited from interface com.google.cloud.dataflow.sdk.options.PipelineOptions
PipelineOptions.AtomicLongFactory, PipelineOptions.CheckEnabled
-
-
Method Summary
All Methods Instance Methods Abstract Methods Deprecated Methods Modifier and Type Method and Description String
getApiRootUrl()
The root URL for the Dataflow API.com.google.api.services.dataflow.Dataflow
getDataflowClient()
An instance of the Dataflow client.String
getDataflowEndpoint()
Dataflow endpoint to use.String
getDataflowJobFile()
The path to write the translated Dataflow job specification out to at job submission time.boolean
getDumpHeapOnOOM()
If true, save a heap dump before killing a thread or process which is GC thrashing or out of memory.List<String>
getExperiments()
The list of backend experiments to enable.int
getNumberOfWorkerHarnessThreads()
Number of threads to use on the Dataflow worker harness.String
getOverrideWindmillBinary()
Custom windmill_main binary to use with the streaming runner.com.google.cloud.dataflow.sdk.util.PathValidator
getPathValidator()
The path validator instance that should be used to validate paths.Class<? extends com.google.cloud.dataflow.sdk.util.PathValidator>
getPathValidatorClass()
The class of the validator that should be created and used to validate paths.String
getPubsubRootUrl()
Root URL for use with the Pubsub API.com.google.cloud.dataflow.sdk.util.Stager
getStager()
The resource stager instance that should be used to stage resources.Class<? extends com.google.cloud.dataflow.sdk.util.Stager>
getStagerClass()
The class responsible for staging resources to be accessible by workers during job execution.Map<String,String>
getTransformNameMapping()
Mapping of old PTranform names to new ones, specified as JSON{"oldName":"newName",...}
.boolean
getUpdate()
Deprecated.This property is replaced byDataflowPipelineOptions.getUpdate()
String
getWindmillServiceEndpoint()
Custom windmill service endpoint.int
getWindmillServicePort()
void
setApiRootUrl(String value)
void
setDataflowClient(com.google.api.services.dataflow.Dataflow value)
void
setDataflowEndpoint(String value)
void
setDataflowJobFile(String value)
void
setDumpHeapOnOOM(boolean dumpHeapBeforeExit)
void
setExperiments(List<String> value)
void
setNumberOfWorkerHarnessThreads(int value)
void
setOverrideWindmillBinary(String value)
void
setPathValidator(com.google.cloud.dataflow.sdk.util.PathValidator validator)
void
setPathValidatorClass(Class<? extends com.google.cloud.dataflow.sdk.util.PathValidator> validatorClass)
void
setPubsubRootUrl(String value)
void
setStager(com.google.cloud.dataflow.sdk.util.Stager stager)
void
setStagerClass(Class<? extends com.google.cloud.dataflow.sdk.util.Stager> stagerClass)
void
setTransformNameMapping(Map<String,String> value)
void
setUpdate(boolean value)
Deprecated.void
setWindmillServiceEndpoint(String value)
void
setWindmillServicePort(int value)
-
Methods inherited from interface com.google.cloud.dataflow.sdk.options.PipelineOptions
as, cloneAs, getOptionsId, getRunner, getStableUniqueNames, getTempLocation, outputRuntimeOptions, setOptionsId, setRunner, setStableUniqueNames, setTempLocation
-
Methods inherited from interface com.google.cloud.dataflow.sdk.transforms.display.HasDisplayData
populateDisplayData
-
-
-
-
Method Detail
-
getExperiments
@Experimental List<String> getExperiments()
The list of backend experiments to enable.Dataflow provides a number of experimental features that can be enabled with this flag.
Please sync with the Dataflow team before enabling any experiments.
-
getApiRootUrl
@Default.String(value="https://dataflow.googleapis.com/") String getApiRootUrl()
The root URL for the Dataflow API.dataflowEndpoint
can override this value if it contains an absolute URL, otherwiseapiRootUrl
will be combined withdataflowEndpoint
to generate the full URL to communicate with the Dataflow API.
-
setApiRootUrl
void setApiRootUrl(String value)
-
getDataflowEndpoint
@Default.String(value="") String getDataflowEndpoint()
Dataflow endpoint to use.Defaults to the current version of the Google Cloud Dataflow API, at the time the current SDK version was released.
If the string contains "://", then this is treated as a URL, otherwise
getApiRootUrl()
is used as the root URL.
-
setDataflowEndpoint
void setDataflowEndpoint(String value)
-
getDataflowJobFile
String getDataflowJobFile()
The path to write the translated Dataflow job specification out to at job submission time. The Dataflow job specification will be represented in JSON format.
-
setDataflowJobFile
void setDataflowJobFile(String value)
-
getPathValidatorClass
@Default.Class(value=com.google.cloud.dataflow.sdk.util.DataflowPathValidator.class) Class<? extends com.google.cloud.dataflow.sdk.util.PathValidator> getPathValidatorClass()
The class of the validator that should be created and used to validate paths. If pathValidator has not been set explicitly, an instance of this class will be constructed and used as the path validator.
-
setPathValidatorClass
void setPathValidatorClass(Class<? extends com.google.cloud.dataflow.sdk.util.PathValidator> validatorClass)
-
getPathValidator
@Default.InstanceFactory(value=DataflowPipelineDebugOptions.PathValidatorFactory.class) com.google.cloud.dataflow.sdk.util.PathValidator getPathValidator()
The path validator instance that should be used to validate paths. If no path validator has been set explicitly, the default is to use the instance factory that constructs a path validator based upon the currently set pathValidatorClass.
-
setPathValidator
void setPathValidator(com.google.cloud.dataflow.sdk.util.PathValidator validator)
-
getStagerClass
@Default.Class(value=com.google.cloud.dataflow.sdk.util.GcsStager.class) Class<? extends com.google.cloud.dataflow.sdk.util.Stager> getStagerClass()
The class responsible for staging resources to be accessible by workers during job execution. If stager has not been set explicitly, an instance of this class will be created and used as the resource stager.
-
setStagerClass
void setStagerClass(Class<? extends com.google.cloud.dataflow.sdk.util.Stager> stagerClass)
-
getStager
@Default.InstanceFactory(value=DataflowPipelineDebugOptions.StagerFactory.class) com.google.cloud.dataflow.sdk.util.Stager getStager()
The resource stager instance that should be used to stage resources. If no stager has been set explicitly, the default is to use the instance factory that constructs a resource stager based upon the currently set stagerClass.
-
setStager
void setStager(com.google.cloud.dataflow.sdk.util.Stager stager)
-
getDataflowClient
@Default.InstanceFactory(value=DataflowPipelineDebugOptions.DataflowClientFactory.class) com.google.api.services.dataflow.Dataflow getDataflowClient()
An instance of the Dataflow client. Defaults to creating a Dataflow client using the current set of options.
-
setDataflowClient
void setDataflowClient(com.google.api.services.dataflow.Dataflow value)
-
getPubsubRootUrl
@Default.String(value="https://pubsub.googleapis.com") String getPubsubRootUrl()
Root URL for use with the Pubsub API.
-
setPubsubRootUrl
void setPubsubRootUrl(String value)
-
getUpdate
@Deprecated boolean getUpdate()
Deprecated. This property is replaced byDataflowPipelineOptions.getUpdate()
Whether to update the currently running pipeline with the same name as this one.
-
setUpdate
@Deprecated void setUpdate(boolean value)
Deprecated.
-
getTransformNameMapping
Map<String,String> getTransformNameMapping()
Mapping of old PTranform names to new ones, specified as JSON{"oldName":"newName",...}
. To mark a transform as deleted, make newName the empty string.
-
getOverrideWindmillBinary
String getOverrideWindmillBinary()
Custom windmill_main binary to use with the streaming runner.
-
setOverrideWindmillBinary
void setOverrideWindmillBinary(String value)
-
getWindmillServiceEndpoint
String getWindmillServiceEndpoint()
Custom windmill service endpoint.
-
setWindmillServiceEndpoint
void setWindmillServiceEndpoint(String value)
-
getWindmillServicePort
@Default.Integer(value=443) int getWindmillServicePort()
-
setWindmillServicePort
void setWindmillServicePort(int value)
-
getNumberOfWorkerHarnessThreads
int getNumberOfWorkerHarnessThreads()
Number of threads to use on the Dataflow worker harness. If left unspecified, the Dataflow service will compute an appropriate number of threads to use.
-
setNumberOfWorkerHarnessThreads
void setNumberOfWorkerHarnessThreads(int value)
-
getDumpHeapOnOOM
boolean getDumpHeapOnOOM()
If true, save a heap dump before killing a thread or process which is GC thrashing or out of memory. The location of the heap file will either be echoed back to the user, or the user will be given the opportunity to download the heap file.CAUTION: Heap dumps can of comparable size to the default boot disk. Consider increasing the boot disk size before setting this flag to true.
-
setDumpHeapOnOOM
void setDumpHeapOnOOM(boolean dumpHeapBeforeExit)
-
-