Google Cloud Dataflow SDK for Java, version 1.9.1
com.google.cloud.dataflow.sdk.options
Interface DataflowPipelineOptions
-
- All Superinterfaces:
- ApplicationNameOptions, BigQueryOptions, CloudDebuggerOptions, DataflowPipelineDebugOptions, DataflowPipelineWorkerPoolOptions, DataflowProfilingOptions, DataflowWorkerLoggingOptions, GcpOptions, GcsOptions, GoogleApiDebugOptions, HasDisplayData, PipelineOptions, StreamingOptions
- All Known Subinterfaces:
- BlockingDataflowPipelineOptions, DataflowWorkerHarnessOptions, TestDataflowPipelineOptions
public interface DataflowPipelineOptions extends PipelineOptions, GcpOptions, ApplicationNameOptions, DataflowPipelineDebugOptions, DataflowPipelineWorkerPoolOptions, BigQueryOptions, GcsOptions, StreamingOptions, CloudDebuggerOptions, DataflowWorkerLoggingOptions, DataflowProfilingOptions
Options that can be used to configure theDataflowPipeline
.
-
-
Nested Class Summary
Nested Classes Modifier and Type Interface and Description static class
DataflowPipelineOptions.JobNameFactory
Returns a normalized job name constructed fromApplicationNameOptions.getAppName()
, the local system user name (if available), and the current time.-
Nested classes/interfaces inherited from interface com.google.cloud.dataflow.sdk.options.DataflowPipelineDebugOptions
DataflowPipelineDebugOptions.DataflowClientFactory, DataflowPipelineDebugOptions.PathValidatorFactory, DataflowPipelineDebugOptions.StagerFactory
-
Nested classes/interfaces inherited from interface com.google.cloud.dataflow.sdk.options.DataflowPipelineWorkerPoolOptions
DataflowPipelineWorkerPoolOptions.AutoscalingAlgorithmType, DataflowPipelineWorkerPoolOptions.TeardownPolicy, DataflowPipelineWorkerPoolOptions.WorkerHarnessContainerImageFactory
-
Nested classes/interfaces inherited from interface com.google.cloud.dataflow.sdk.options.GcpOptions
GcpOptions.CredentialDirFactory, GcpOptions.DefaultProjectFactory, GcpOptions.GcpUserCredentialsFactory
-
Nested classes/interfaces inherited from interface com.google.cloud.dataflow.sdk.options.GoogleApiDebugOptions
GoogleApiDebugOptions.GoogleApiTracer
-
Nested classes/interfaces inherited from interface com.google.cloud.dataflow.sdk.options.PipelineOptions
PipelineOptions.AtomicLongFactory, PipelineOptions.CheckEnabled
-
Nested classes/interfaces inherited from interface com.google.cloud.dataflow.sdk.options.GcsOptions
GcsOptions.ExecutorServiceFactory
-
Nested classes/interfaces inherited from interface com.google.cloud.dataflow.sdk.options.DataflowWorkerLoggingOptions
DataflowWorkerLoggingOptions.Level, DataflowWorkerLoggingOptions.WorkerLogLevelOverrides
-
Nested classes/interfaces inherited from interface com.google.cloud.dataflow.sdk.options.DataflowProfilingOptions
DataflowProfilingOptions.DataflowProfilingAgentConfiguration
-
-
Method Summary
All Methods Instance Methods Abstract Methods Modifier and Type Method and Description String
getJobName()
The Dataflow job name is used as an idempotence key within the Dataflow service.Map<String,String>
getLabels()
Labels that will be applied to the billing records for this job.String
getProject()
Project id to use when launching jobs.String
getServiceAccount()
Run the job as a specific service account, instead of the default GCE robot.String
getStagingLocation()
GCS path for staging local files, e.g.boolean
getUpdate()
Whether to update the currently running pipeline with the same name as this one.void
setJobName(String value)
void
setLabels(Map<String,String> labels)
void
setProject(String value)
void
setServiceAccount(String value)
void
setStagingLocation(String value)
void
setUpdate(boolean value)
-
Methods inherited from interface com.google.cloud.dataflow.sdk.options.DataflowPipelineDebugOptions
getApiRootUrl, getDataflowClient, getDataflowEndpoint, getDataflowJobFile, getDumpHeapOnOOM, getExperiments, getNumberOfWorkerHarnessThreads, getOverrideWindmillBinary, getPathValidator, getPathValidatorClass, getPubsubRootUrl, getStager, getStagerClass, getTransformNameMapping, getWindmillServiceEndpoint, getWindmillServicePort, setApiRootUrl, setDataflowClient, setDataflowEndpoint, setDataflowJobFile, setDumpHeapOnOOM, setExperiments, setNumberOfWorkerHarnessThreads, setOverrideWindmillBinary, setPathValidator, setPathValidatorClass, setPubsubRootUrl, setStager, setStagerClass, setTransformNameMapping, setWindmillServiceEndpoint, setWindmillServicePort
-
Methods inherited from interface com.google.cloud.dataflow.sdk.options.DataflowPipelineWorkerPoolOptions
getAutoscalingAlgorithm, getDiskSizeGb, getFilesToStage, getMaxNumWorkers, getNetwork, getNumWorkers, getSubnetwork, getTeardownPolicy, getUsePublicIps, getWorkerDiskType, getWorkerHarnessContainerImage, getWorkerMachineType, getZone, setAutoscalingAlgorithm, setDiskSizeGb, setFilesToStage, setMaxNumWorkers, setNetwork, setNumWorkers, setSubnetwork, setTeardownPolicy, setUsePublicIps, setWorkerDiskType, setWorkerHarnessContainerImage, setWorkerMachineType, setZone
-
Methods inherited from interface com.google.cloud.dataflow.sdk.options.BigQueryOptions
getTempDatasetId, setTempDatasetId
-
Methods inherited from interface com.google.cloud.dataflow.sdk.options.StreamingOptions
isStreaming, setStreaming
-
Methods inherited from interface com.google.cloud.dataflow.sdk.options.ApplicationNameOptions
getAppName, setAppName
-
Methods inherited from interface com.google.cloud.dataflow.sdk.options.GcpOptions
getAuthorizationServerEncodedUrl, getCredentialDir, getCredentialFactoryClass, getCredentialId, getGcpCredential, getSecretsFile, getServiceAccountKeyfile, getServiceAccountName, getTokenServerUrl, setAuthorizationServerEncodedUrl, setCredentialDir, setCredentialFactoryClass, setCredentialId, setGcpCredential, setSecretsFile, setServiceAccountKeyfile, setServiceAccountName, setTokenServerUrl
-
Methods inherited from interface com.google.cloud.dataflow.sdk.options.GoogleApiDebugOptions
getGoogleApiTrace, setGoogleApiTrace
-
Methods inherited from interface com.google.cloud.dataflow.sdk.options.PipelineOptions
as, cloneAs, getOptionsId, getRunner, getStableUniqueNames, getTempLocation, outputRuntimeOptions, setOptionsId, setRunner, setStableUniqueNames, setTempLocation
-
Methods inherited from interface com.google.cloud.dataflow.sdk.transforms.display.HasDisplayData
populateDisplayData
-
Methods inherited from interface com.google.cloud.dataflow.sdk.options.GcsOptions
getExecutorService, getGcsEndpoint, getGcsUploadBufferSizeBytes, getGcsUtil, setExecutorService, setGcsEndpoint, setGcsUploadBufferSizeBytes, setGcsUtil
-
Methods inherited from interface com.google.cloud.dataflow.sdk.options.CloudDebuggerOptions
getDebuggee, getEnableCloudDebugger, getMaxConditionCost, setDebuggee, setEnableCloudDebugger, setMaxConditionCost
-
Methods inherited from interface com.google.cloud.dataflow.sdk.options.DataflowWorkerLoggingOptions
getDefaultWorkerLogLevel, getWorkerLogLevelOverrides, setDefaultWorkerLogLevel, setWorkerLogLevelOverrides
-
Methods inherited from interface com.google.cloud.dataflow.sdk.options.DataflowProfilingOptions
getEnableProfilingAgent, getProfilingAgentConfiguration, getSaveProfilesToGcs, setEnableProfilingAgent, setProfilingAgentConfiguration, setSaveProfilesToGcs
-
-
-
-
Method Detail
-
getProject
@Validation.Required @Default.InstanceFactory(value=GcpOptions.DefaultProjectFactory.class) String getProject()
Description copied from interface:GcpOptions
Project id to use when launching jobs.- Specified by:
getProject
in interfaceGcpOptions
-
setProject
void setProject(String value)
- Specified by:
setProject
in interfaceGcpOptions
-
getStagingLocation
String getStagingLocation()
GCS path for staging local files, e.g. gs://bucket/objectMust be a valid Cloud Storage URL, beginning with the prefix "gs://"
At least one of
PipelineOptions.getTempLocation()
orgetStagingLocation()
must be set. IfPipelineOptions.getTempLocation()
is not set, then the Dataflow pipeline defaults to usinggetStagingLocation()
.
-
setStagingLocation
void setStagingLocation(String value)
-
getJobName
@Default.InstanceFactory(value=DataflowPipelineOptions.JobNameFactory.class) String getJobName()
The Dataflow job name is used as an idempotence key within the Dataflow service. If there is an existing job that is currently active, another active job with the same name will not be able to be created. Defaults to using the ApplicationName-UserName-Date.
-
setJobName
void setJobName(String value)
-
getUpdate
boolean getUpdate()
Whether to update the currently running pipeline with the same name as this one.- Specified by:
getUpdate
in interfaceDataflowPipelineDebugOptions
-
setUpdate
void setUpdate(boolean value)
- Specified by:
setUpdate
in interfaceDataflowPipelineDebugOptions
-
getServiceAccount
@Hidden @Experimental String getServiceAccount()
Run the job as a specific service account, instead of the default GCE robot.
-
setServiceAccount
void setServiceAccount(String value)
-
getLabels
Map<String,String> getLabels()
Labels that will be applied to the billing records for this job.
-
-